Tasks filtering by start date and by due date, and their ordering

[shompoe]

When the task app allows a start date and a due date, our natural inclination is surely to set the start date to the time we need to become concerned about the task, in other words, the time that the task should come into view. In this way we may schedule many tasks into the future but maintain our immediate visibility only on those tasks which we need to be focussed on in the present. And the due date, logically, is the time at which we plan to complete the task.

However, the present version of the widget filters tasks by due date. This means that if I have a task which is already scheduled to start tomorrow, but which I have allocated two weeks from start to completion, and I have a widget which filters all but today's tasks and events, my task will not appear in the widget untill the day it is due to be completed. So for the two weeks that have now transpired the widget has given me no warning that I needed to be working on that task.

[shompoe]

I believe the ideal behaviour would be to have tasks filtered by start date if set or due date if not set. And to have them displayed under the day header for start date if set.... and move up the list as time progresses. Then to stay in Past and Due header until they are completed or rescheduled. If both start date and due date are set then they could be ordered in Past and Due by due date and highlighted once due date expires.

[yvolk]

Thank you @shompoe for raising this. Your suggestion continues our discussion with @ekalin here and below: #308 (comment) where we didn't define / agree on proper tasks ordering / showing / filtering...

Reading your above posts I see the problem and I also see that different cases still need more details/clarification. Otherwise we will fix one case and break another...

In particular, it seems that we need to filter tasks by both start and due dates...

[shompoe]

Hi again,

As far as I can see, if you do it how I suggest (use due date if start date not set, use start date if due date not set, but if both set then filter by start date and show under the day header from start date but sort by due date under Past and Due header) then it ought to work as the user expects regardless of whether the app they use has both start and due date or just one or the other???

I can't think of a scenario where this would cause unexpected behaviour.

[yvolk]

@shompoe No :-)
Suppose this case:

1. I have set a filter to "Show past events ended today"
2. I have a task starting last week and ending today.

Following your description "filter by start date" the task should be filtered out i.e. hidden.
?!

[shompoe]

No. It should stay under the Past and Due header regardless of start and due date until the task is marked complete. So actually, only future tasks filtered by start date but past tasks should remain unfiltered regardless of settings. I believe this is actually how the widget works at the moment. Ie. the display past events setting only affects events, but not tasks.

This should be considered expected behaviour because an event (say an appointment) expires at a certain time, regardless of wether you attend it or not. A task however, cannot expire until you have actioned it and so it becomes overdue.

I just did a quick test and this is what the widget is doing now, albeit filtering on due rather than start.

Does that make sense?

[yvolk]

OK, so we extend our case:

1. For past events:
   I set a filter to "Show past events ended today"
2. For current and future events:
   Date range: One month

I have these tasks in the Task app:

1. task1 starting last week and ending later today (incomplete).
2. task2 starting last week and ending in 40 days (incomplete).
3. task3 without start date and ending later today (incomplete).
4. task4 without start date and ending in 40 days (incomplete).
5. task5 starting last month and ending last week (incomplete).
6. task6 starting last month and ending tomorrow (complete).
7. task7 starting today and ending tomorrow (complete).
8. task8 starting next week and ending in 40 days (incomplete).

As we don't know, if tasks in the Task app have Start or Due date, we need:

1. Query the Task app:
   "Give me all INCOMPLETE tasks that don't have Start date OR have start date before (today + 1 month)

tasks 1 to 5 and task8 will be shown
task6 and task7 won't be shown at all...

Correct?

OR according to your words task4 shouldn't be shown also ?!

For the reference, current filtering logic is in this file:
https://github.com/andstatus/todoagenda/blob/master/app/src/main/java/org/andstatus/todoagenda/task/dmfs/DmfsOpenTasksProvider.java

below this line:

private String getWhereClause() {

(currently it's at line 76)

[shompoe]

tasks 1 to 5 and task8 will be shown task6 and task7 won't be shown at all...

Yes, this is what should happen. Show everything that is incomplete and either started in the past, will start within 1 month, or has no specified start date. Tasks that are complete must never be shown. And tasks that start after 1 month must not be shown.

You make an interesting point about task 4. My original reasoning was that some apps maybe don't have both start and due dates, some only start and some only due. I don't know if this is the case. If it is then it complicates things.

However, if the app supports both start and due then failure to set a start date could have different meanings for different people (I read the thread mentioned above)

For me, when I don't set start date it means something I should be starting at my earliest convenience. So I don't want to loose visibility of those tasks regardles of wether I have given them a due date.

So I think my original logic is only relevant for apps that don't have start date. Are there any? I don't know.

Going back to task 4. It would be useful for it to display the due date in the widget. But perhaps that's for another thread.

What do you think?
I'd be happy to do some testing of a hyperthetical update to the widget if that helps.

[yvolk]

To all!
For discussion below I wrote the same tasks, but made it easier to understand, what is shown and where (if not obvious...).
I added some more tasks for better coverage.
Moreover, I sorted them in a way they will be sorted in the widget (assuming initially they are sorted according to their numbers)

For which events/tasks	Filter settings
For past events	"Show past events ended today" is on
For current and future events	Date range: One month

New option introduced:
"Task scheduling" (What date gets main attention for a task) with two values:
T1. Show tasks under date due (Due date gets main attention, and we are showing tasks at their due dates, where it makes sense).
T2. Show tasks under date started (Start date gets main attention, and we are showing tasks at their start dates, where it makes sense).

In any case both start and due dates are taken into account, but differently.

task9 and task10 below (as other tasks without start and due dates) need special treatment i.e. additional option:
"Tasks without start and due dates" with values:
.1 Show at the end of the whole list (default option)
.2 Show at the end of Today.
.3 Hide.
Below tables are shown with default option

Task visibility and sorting if option T1 (Due date...) is selected:

Task description	Show	Where to show
task11 starting last month (after task5) and due last week (at the same time as task5) (complete)	no
task5 starting last month and due last week (incomplete)	yes	Last week (or "Past and Due" if turned on. The same for others in the past)
task3 without start date and due later today at 9PM (incomplete)	yes	Today or at 9PM
task1 starting last week and due later today at 10PM (incomplete)	yes	Today at 10PM
task6 starting last month and due tomorrow at 8AM (complete)	no
task7 starting today at 8AM and due tomorrow at 9AM (complete)	no
task12 starting today at 7AM and due tomorrow at 10AM (incomplete)	yes	at due date: Tomorrow at 10AM
task17 starting in 10 days, without due date (incomplete)	yes	At start date
task16 without start date and due in 20 days	yes	At due date
		End of list header
task14 starting in 39 days, and without due date (incomplete)	no
task8 starting next week, due in 41 days (incomplete)	yes	End of list, closer to due date
task4 without start date and due in 42 days (incomplete)	no
task2 starting last week and due in 43 days (incomplete)	yes	End of list
task15 starting in 40 days, due in 45 days (incomplete)	no
task10 without start date and without due date (incomplete)	yes	(Position depends on "Tasks without start and due dates")
task9 without start date and without due date (complete)	no	(Position depends on "Tasks without start and due dates")

Task visibility and sorting if option T2 (Start date...) is selected:

Task description	Show	Where to show
task11 starting last month (after task5) and due last week (at the same time as task5) (complete)	no
task5 starting last month and due last week (incomplete)	yes	Last week (or "Past and Due" if turned on. The same for others in the past)
task3 without start date and due later today at 9PM (incomplete)	yes	Start of today
task1 starting last week and due later today at 10PM (incomplete)	yes	Start of Today
task6 starting last month and due tomorrow at 8AM (complete)	no
task16 without start date and due in 20 days	yes	Start of today
task4 without start date and due in 42 days (incomplete)	yes	Start of today
task2 starting last week and due in 43 days (incomplete)	yes	Start of Today
task12 starting today at 7AM and due tomorrow at 10AM (incomplete)	yes	Today at 7AM
task7 starting today at 8AM and due tomorrow at 9AM (complete)	no
task8 starting next week, due in 41 days (incomplete)	yes	Next week
task17 starting in 10 days, without due date (incomplete)	yes	At start date
		End of list header
task14 starting in 39 days, and without due date (incomplete)	no
task15 starting in 40 days, due in 45 days (incomplete)	no
task10 without start date and without due date (incomplete)	yes	(Position depends on "Tasks without start and due dates")
task9 without start date and without due date (complete)	no	(Position depends on "Tasks without start and due dates")

[shompoe]

Q1: decision to show task4 (and task10 also?) depends on a User's understanding of an empty start date. It may mean "today" (see "Where to show" column) or "don't care" (add such an option?)

For me, it means today.

Q2. As we noted in #358 task10 may need special treatment. E.g.:
.1 Hide
.2 Show today
.3 Show only such tasks ???

For me, show today. But I go along with the principle that some users might like the ability to configure other behaviours.

Q3. Is sorting above correct? I see that sorting will depend on Q1 (how we treat no start day)

For me, the order is not correct. Start date just tells me if I'm working on a task or if I'm not yet concerned about it (ie. to display it in my date range or not) whereas due date tells me how long I have for each task and so should affect the priority (ie. tasks with shortest time to complete them at the top).

The decision to put tasks with no due date at the bottom is therefore correct.

[yvolk]

@shompoe @gtbuchanan Maybe we really can solve all above problems with sorting and filtering with only one additional option:

How do we treat a task without start date:

1. Don't care about such a task (and hide it if no end date also).
2. Show today, or at the end of list, if no end date.

?! (Maybe better wording is needed...)

[shompoe]

How do we treat a task without start date:

1. Don't care about such a task (and hide it if no end date also).

2. Show today, or at the end of list, if no end date.

Seems like this would work for my purposes and for #358.

Can I suggest that the default should be option 2 because this is closer to what already happens and because new users might get confused if they don't see their tasks.

As we are onto the subject of sorting. After sorting first by due date, tasks should then be sorted by priority?

[yvolk]

@shompoe I agree that option "2. Show today, or at the end of list, if no end date" should be default.

Regarding sorting, currently events (calendars events and tasks...) with the same date are sorted by the source, and then - by title. So I think that any other special sorting could be better done changing order of Calendars and Task lists. I.e. you can create different task lists for tasks with different priorities...

[shompoe]

Yes, I see now. As the widget displays two different types of data, compromises are necessary. So I go along with what you say.

[ekalin]

I think the proposal is quite reasonable. A few observations:

• Task 4 should not be shown. Once it's in range to be shown (in 12 days), it will be displayed, but in the "due date" date. To make this clear, I'd add a Task 13 to the table, with no start date, ending in 10 days - It's shown, in the 10th day listing.

• Task 10 should be shown, under Today. The reasoning I use is that a task with no start or end date is a reminder that doesn't need to be done by or at a certain date, but nevertheless I'd like to see it so that I can do it whenever possible.

• There definitely exists apps that don't support a start date. Google Tasks (integrated with Google Calendar and probably Gmail) doesn't, and it's probably used a lot. It believe that if there is no start date (because the app doesn't provide one or if it's not set), internally the start date could be set to the due date, and that would work as expected. But I haven't analyzed each possible case.

For coverage (in particular of the querying, not much of the ordering), one could add two more tasks:

• Task 14, starting in 40 days, no due date - Not shown
• Task 15, starting in 40 days, due in 45 days - Not shown

Also, I'm completely ignoring task times (again, which some apps don't support). I'm sure someone has an use case for them, and it might even be reasonable, but if you one really cares about the time, maybe they need a calendar event and not a task. But perhaps it could be used for sorting tasks in a day, but not for filtering.

[shompoe]

This makes complete sense to me. 😊

[yvolk]

@ekalin @shompoe
I added two more tasks to the list above, as @ekalin suggested, and split one list into two depending on the selected "How do we treat a task without start date" option.

Please review.

As I see task10 is NOT where @ekalin expected :-)

[shompoe]

Hi, I've read through the output for T2 a couple of times and I think that is how I'd like to see it.

A couple of thoughts for T1, though this is not how I use tasks. Task 12 is shown under tomorrow, because it is due tomorrow. So logically task 8 cannot be shown because the due date is beyond 1 month? And for task 10, should there be a separate option to show or not show it?

[gtbuchanan]

I agree with the current chart. Like @ekalin said, many people (including myself) are coming from Google Calendar where their "Reminders" only have a "due date" so there are no preconceived requirements about "start date". Nonetheless, I can see the usefulness of both sides being described.

I would like to mention that I personally don't care if "task 10" is hidden as long as it isn't displayed at the start of the widget (like "Today"). My main problem right now is my dozens of "task 10" situations push all my calendar events off-screen. I'd be okay if they were displayed at the end of the widget instead in T1. Though, I'm not certain if that is what you meant by "end of the list".

[yvolk]

@gtbuchanan Maybe we really need to show task10 even for T1 ("Don't care") case:

1. These tasks without any dates won't clutter your space (they will be shown after all other events in the widget), but you will still be able to see them. Otherwise, why you have such tasks at all
2. The "T1" option becomes even shorter to read and simpler to understand, without a special case:
   "Don't care about a task without start date"
   ?!

[ekalin]

I'm not sure I understand what your proposed option does, especially option T1.

I'm also not sure about the need for this option. It's one thing to add an option to hide tasks with no start and end date as in #358, but you seem to want to mix that feature with some other treatment. Moreover, in some cases it's not really the user's choice to set or not a start date, but a limitation on the task app.

Currently I have more issues with both your options than I had before :-) Hete's how I think it should look like. I've removed the completed tasks because they only make it harder to follow the chart, and I think everyone agrees that completed tasks are not to be shown:

Task description	Show	Where to show
task5 starting last month and due last week	yes	Last week (or "Past and Due" if turned on. The same for others in the past)
task3 without start date and due later today at 9PM	yes	Today
task1 starting last week and due later today at 10PM	yes	Today (after 3, since 1 is due later)
task12 starting today at 7AM and due tomorrow	yes	Today
task2 starting last week and due in 43 days	yes	Today
task10 without start date and without due date	Depends on setting of #358	If to be displayed, should be displayed today.
task8 starting next week, due in 41 days	yes	At start date
task16 without start date and due in 20 days	yes	At due date
task4 without start date and due in 42 days	no
task15 starting in 40 days, due in 45 days	no
task14 starting in 39 days, and without due date	no

Edit: Added task16

[gtbuchanan]

Yes, if T1 was added just for me then I agree with @ekalin chart for the new default behavior. Start dates always matter to me if there is one. For tasks without a start or due date I just use a separate task widget or my tasks app to set dates or complete when appropriate. However, if we still wanted to display them in this widget we can nix the original setting, use @ekalin chart, and add the single following option instead:

"Pin open-ended tasks to Today"

Default to true for current behavior. Setting to false would move the tasks with no start or end date to a new section at the bottom of the widget titled something like "Open-ended Tasks".

[yvolk]

@ekalin wrote

I'm also not sure about the need for this option. It's one thing to add an option to hide tasks with no start and end date as in #358, but you seem to want to mix that feature with some other treatment.

1. "some other treatment" is exactly the topic of this issue. Please read it from the very beginning.

2. Placing tasks without any date at the bottom of the list is logical ("correct" as @shompoe confirmed): these are the latest tasks you should care about.
   I see that actually this is the only point that doesn't fit your needs?!

in some cases it's not really the user's choice to set or not a start date, but a limitation on the task app

ToDo Agenda doesn't support such Task apps:
Samsung Calendar simply doesn't have start date, as implemented,
Open Tasks allows to leave start date empty.

4. Regarding "Depends on setting of Option to hide tasks with no dates #358" - I removed such reference in order to make the app simpler as what we have now satisfies @gtbuchanan (reporter of that issue).

[ekalin]

Samsung Calendar simply doesn't have start date, as implemented

That's exactly what I meant on limitation on the task app: since there's no way to set the start date, all tasks have start date empty, so whatever the user selects in that option will apply to all tasks.

[yvolk]

@gtbuchanan

Yes, if T1 was added just for me...

No, the two cases appeared as a result of THIS issue's discussion. Your discomfort with tasks without both dates just happen to fit into the new reasoning...

[yvolk]

Samsung Calendar simply doesn't have start date, as implemented

That's exactly what I meant on limitation on the task app: since there's no way to set the start date, all tasks have start date empty, so whatever the user selects in that option will apply to all tasks.

In previous implementation we set (absent) start date equal to present Due date. That was a workaround, because we didn't have logic that we are developing in this issue.
I mean we can (or maybe even should) remove that workaround and leave Start date empty (as it wasn't actually set by a User!)

Edit:
... OR we can change the Samsung "Provider" behavior depending on the "How do we treat a task without start date" option value?!

[gtbuchanan]

@yvolk I see. Well, T1 does fit my need for #358 but it prevents me from ever using start date in a useful way as this ticket suggests (which was the reason I entered a separate ticket). I said it is fine with me for now since it solves my immediate issue and you are trying to avoid adding multiple new settings. If it were my choice, I would prefer the new fixed behavior @ekalin suggested and a new setting for where the "no start or due date" tasks are displayed. Sorry for the confusion.

[yvolk]

@gtbuchanan As you see, this discussion is quite complex, it requires attention to details, to every line of the long post.
So please write which exactly line in my post's tables you would change and why.

Comparing different tables with one word won't make us closer to the better solution. I think :-)
Do you agree?

[yvolk]

Dear @shompoe @ekalin and @gtbuchanan

I made the first implementation of "Task scheduling" and "Tasks without start and due dates" options as described in this comment: #356 (comment)
Alpha version of the build is here: andstatus#3

This build allows you, in particular, to play with exactly those content (plus some calendars events...) that is now in T1 and T2 tables: just goto to Settings -> Feedback and Backup -> Restore settings from a backup -> and restore this file with settings and events (unzip it first manually!):
filter_tasks_308_no_filters.zip

During implementation I made some clarifications and some changes to the text table in above comment, please review them now!

As a result, our widget can now show T1 and T2 exactly as in the current version of the description. See screenshots below:
T1 (Show tasks under date due. "Tasks without start and due dates"= "Show at the end of the whole list", Filters turned off (new setting also...)):

T2 (Show tasks under date started. "Tasks without start and due dates"= "Show at the end of the whole list", Filters turned off (new setting also...))

[shompoe]

I just tried a simple test. Selecting show under start date option. My widget shows the next 7 days. My task started last week and is due in 2 weeks time. It is not shown in the widget.

[yvolk]

@shompoe It's like "task2" in above table, it's shown despite it is due in 43 days and we have "Data range" set to "one month"
?!

Another tip: switch "Filters troubleshooter" to "Debug filtering"! - This should solve the problem. I really see that there is a bug in the query (I didn't look there yet)

If the task isn't shown in "Debug filtering" mode either and you still cannot figure why... please "Share events and settings for debugging" and email me or attach here...
Note: Please turn off filters before sending to me. Otherwise I won't see that task either :-)

[yvolk]

In order to simplify completion of work on this issue I recreated its current state here: andstatus#4
Please follow up in that thread!

[shompoe]

Yes,  setting the filtering mode to "debug" and the event now appears as it should do. Thank you. Great work. -- Mark Desrousseaux 19 Jan 2020, 11:56 by notifications@github.com:

…

@shompoe <https://github.com/shompoe>> It's like "task2" in above table, it's shown despite it is due in 43 days and we have "Data range" set to "one month" ?! If you cannot figure out, why the task isn't shown... please "Share events and settings for debugging" and email me or attach here... — You are receiving this because you were mentioned. Reply to this email directly, > view it on GitHub <#356?email_source=notifications&email_token=AM6VTXYJU3QMAMHCDBJD5N3Q6Q5VFA5CNFSM4JR5WGRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKQKDY#issuecomment-575997199>> , or > unsubscribe <https://github.com/notifications/unsubscribe-auth/AM6VTXZD22VNIFBDPFR2C7LQ6Q5VFANCNFSM4JR5WGRA>> .