Doing Dailies early: Habit VS Todoist

[crookedneighbor]

I noticed a difference between how Habit handles dailies and how Todoist handles them.

In Habit, I can create a daily that is due on Monday and Thursday. If I do the daily on Wednesday, it's still going to be due on Thursday.

In Todoist, however, if I create a daily for Mon & Thurs, and mark it as completed on Wednesday, it won't be due again until Monday.

If this was for Todos, I wouldn't think it's a big deal, but since Habit calculates damage based on undone dailies, I'd hate to see someone take damage for doing their task early and not realizing that it was still due in Habit.

[cygnoir]

I've been taking damage from this difference in dailies for the past few days. I can help test out potential solutions if needed.

[cygnoir]

I completed all of my dailies yesterday but I took 6.8 HP of damage from them (and missed out on the "Perfect Day" buff). This has happened several times now. I'm not sure how to find out exactly which dailies I'm taking damage for other than "Write in journal" because that 18-day streak has now been broken. (This is an "every day" daily so I'm not sure why I took damage from it.)

I checked HabitRPG before my day reset (at midnight) and all dailies had been checked off. The only thing I can think of is that I took the damage from the dailies I completed earlier than the days they were due.

[crookedneighbor]

One of the devs made an awesome tool for getting extra data about your tasks. You should be able to see which tasks were checked that way.

[cygnoir]

Thanks for reminding me of that. The only thing I can see with an "x" for yesterday under Dailies History is the "Write in journal" daily which I checked off in Todoist way before midnight, so it would have had hours to sync.

However, I had a number of Dailies due yesterday that I had checked off early (weekly work tasks). They're not showing up with an "x" on the history table, though.

[cygnoir]

I'm perplexed. I completed all five of my dailies (double-checked this with the User Data Display) and I still took damage and lost the Perfect Day buff as if I had not completed all of my dailies.

Obviously this isn't a huge concern at the moment, but it will prevent me from participating in boss quests with my party if they will be taking damage as well.

[crookedneighbor]

@cygnoir It's quite possible that you're experiencing the dreaded cron bug, and not something to do with todoist-habitrpg at all. IE, cron is running twice for you.

When you get a chance, check out the linked issue and see if that matches your experience.

You don't happen to use two or more devices (IE, phone and computer) to access Habit that are set to separate time zones? That can cause cron to run twice like you're describing.

Are you on a quest? Is the quest info (number of items received or damage from boss/damage to boss) getting repeated?

[cygnoir]

Thanks for suggesting the linked issue. I've read it through but I'm not sure this is my issue. The timing on my issue seems to coincide with starting to use todoist-habitrpg and having dailies managed by external repeating tasks. I've not had this problem before then, haven't switched browsers, don't use the mobile site or mobile apps, and don't have a custom cron time set.

Last night my dailies reset properly and maintained their streaks but still dealt me damage. I believe I took damage from four "every Sunday" work-related dailies that I completed days ago in Todoist but had to manually check off in HabitRPG last night because they were still appearing as undone and I wanted to make sure I did not take damage for them. However, it's not like Todoist is unchecking them in HabitRPG -- before my cron ran, I made sure that nothing was showing as undone in either HabitRPG or Todoist. Everything was checked off.

I'm not on a quest yet but was planning to start one in the next few days with my party. However, I won't be joining it because I can't guarantee this won't kill them, and they find death pretty demotivating. Also, I'm the party healer, so I really shouldn't be killing them. :)

[crookedneighbor]

Hmmm... So you checked the data display tool and it showed your dailies as checked off, but you still got dealt damage? That still sounds like a cron bug to me...

I'd still like to see if cron is running twice, and the easiest way to do that is by doing a quest.

How about this, go to this guild and look for my user (Blade, just posted a message), private message me so I can reply back. In January (when my gem purchasing resets), I'll send you some gems to buy a quest. You can start a quest, immediately begin it before your party members have time to accept the invitation, and check out if cron is running twice or not.

It'll prevent your party from starting a quest while doing this experiment, so maybe let your party know what's happening, supply them with your user id (or if they are real life friends, just email them), leave the party temporarily, start a new party that's just you, and start the quest to test. You don't even have to finish the quest, you can just abort it and have the scroll returned to your inventory after we've determined what cron is doing, so just give it a day or two. Once the experiment is over, you can request to join the party again.

If you have gems or a quest to spare, we can do this experiment now, and I'll reimburse you the gems in January.

Does that sound all right?

The other troubleshooting step we can take later is to turn off Todoist syncing, have you manually check things both places, and see if you're still receiving damage.

In the meantime, I'll create a task that repeats every day and see what happens on my end.

[crookedneighbor]

@kusold for problems like this one, would it be worth abandoning having dailies show up on particular days and just make all Todoist generated dailies grey for every day? This will allow them to function as dailies, but not be in danger of damaging the user.

Obviously, it takes away the incentive to do the daily if it's grayed out, since it will never damage you legitimately, but I think it's better than the alternative of having it accidentally damage you.

[kusold]

I'm a little confused as to what the root issue is here. Is the root issue the fact that people expect tasks to be able to be completed in advance? Or is the root issue something going on with cron?

I've been pondering if this might be because Todoist resets dailies at midnight and HabitRPG can reset them later (according to when the cron is).

@cygnoir Is your HabitRPG cron set to Midnight local time or set to a custom time?

[crookedneighbor]

@kusold That is a great point about the difference between when Todoist resets and when Habit does.

Here's the info I got from @cygnoir via Habit:

Hi there. The first night of my lone quest, I appeared to trigger cron twice, taking double damage for dailies I had completed. Last night I only triggered it once, but I still took over 23 points of damage when I only left 1 daily incomplete. - 2 days ago

Hi there. Last night I left one daily incomplete and I took 13.8 HP of damage according to the quest. However, I lost 25 HP. I don't see how this is possible ... ? -14 hours ago

Very strange: it's before my cron, and I just took 3 points of damage for 3 Habits that were suddenly created with the following "names": 5301a694-0b3d-4483-bb96-8a698a46cde2, 28ab472b-b1ef-4296-b990-384b697f2c02, 363c4706-f325-4fb8-9388-e72d8fabb03a. Note these are Habits, not Dailies or To-Dos. -an hour ago
01/05/2015 17:39:31

@cygnoir, those habits might have been created when Todoist tried to sync a todo that had been cleared from Habit, so Habit created a habit with the name (ID) of the old todo. I'll investigate some more tonight or tomorrow.

[cygnoir]

@kusold My HabitRPG cron has always remained at midnight local time. Also, I do not want to complete tasks ahead of time; I just don't want to take damage for them when I have completed them and synced. The reason why I think this is a greyed-out task issue and not a completion issue is that I make sure before cron that the sync with Todoist has properly completed my dailies in HabitRPG. I still take damage.

@crookedneighbor Last night I left one daily incomplete and took 25.8 HP of damage from it. Here is the quest text: "cygnoir attacks Vice's Shade for 9.7 damage, Vice's Shade attacks party for 14.3 damage."

I checked the data display tool before cron, after confirming that only one daily was left incomplete, and it reported that the total damage I should be taking would be 3.5 HP. That leaves 22.3 HP unaccounted for.

[cygnoir]

@crookedneighbor I took 24 HP of damage from leaving the same daily incomplete. Quest text: "cygnoir attacks Vice's Shade for 7.8 damage, Vice's Shade attacks party for 14.0 damage."

[crookedneighbor]

@cygnoir How much damage did you take apart from the quest? IE, what does the data display tool say? The Dailies Incomplete link in particular. If you've already completed the task for today, could you uncheck it and use the data display tool again? I just want to get a ballpark figure of how much non-quest damage you'd get for not completing it.

Also, thanks for being a champ and helping us troubleshoot this.

[cygnoir]

@crookedneighbor I neglected to check the data display tool before cron, unfortunately, but today it says that if I leave that specific daily incomplete I will take a total of 3.7 HP damage for it (1.7 to party, 2 to me alone). Since the day before yesterday it said I would take 3.5 HP, maybe yesterday I would be taking 3.6? Does it increment like that?

However, in looking at the data display tool I found something interesting: I have three missing dailies! Meaning, I have three dailies that are not appearing in either Todoist or HabitRPG but are still somehow appearing in the data display tool in Dailies History, nowhere else, and with no red x's but still ... could they be giving me damage?

Also, whenever I load the data display tool these days, I get two of these errors: DataTables warning (table id = 'DataTables_Table_4'): Requested unknown parameter {mData function} from the data source for row 0

[crookedneighbor]

Mysterious Pop Up

Don't worry about the dialog that pops up. I get that too. Though, now that you mention it, I bet that is being caused by Todoist-HabitRPG... Hmmm... Seems likely now that I think about it. I'll have to look into it later. For now, don't worry about it.

Damage Estimator

I think the damage estimation on the Data display tool is incorrect. Vice Awakens is a tough boss, and it wouldn't be unheard of for it to damage the party a ton for a single red daily, especially if that daily is very very red.

I did some testing on a local install and got similar results for an unchecked daily during Vice Awakens.

It would be helpful to know the value of the daily causing the damage, that should give us some indication how much damage it should be giving you.

Would you mind going to the api page, inputting your user id and api key, and then going to GET /user/tasks. Click the try it button, and when it loads your tasks, do a search for the name of the daily that you've been leaving unchecked and record the value here? The value of history would be helpful as well. That'll give us it's value over time.

Mysterious Missing Dailies

Dailies should only hurt you if there's an x on them. the - means they are grayed out and not counted toward your damage.

Can you verify that no tag is enabled (Press the clear tags button even if you're sure you don't have any tags enabled)

Can you also verify that it's not just a grayed out daily? I'm sure you have already done that, but I'm saying everything I'm thinking on this matter, just in case.

[cygnoir]

Today's Results

Cron just ran and I happened to still be awake: 22 HP of damage for the single (same) unchecked daily.

Mysterious Pop-Up

I thought this might have something to do with the tool itself so I reported it on the wiki.

Damage Estimator

I wasn't sure exactly what you needed so here is all of it:
{
"text": "Write 500 words",
"challenge": {},
"attribute": "str",
"priority": 1.5,
"value": -6.418582324043852,
"tags": {},
"notes": "",
"dateCreated": "2015-01-02T12:24:21.000Z",
"id": "d9a04127-18cb-4e40-8bc6-885f1b1896d8",
"streak": 0,
"checklist": [],
"collapseChecklist": false,
"repeat": {
"su": true,
"s": true,
"f": true,
"th": true,
"w": true,
"t": true,
"m": true
},
"completed": false,
"history": [
{
"date": 1420283858049,
"value": -1
},
{
"date": 1420283858084,
"value": -1
},
{
"value": -2.025956704627065,
"date": 1420341642031
},
{
"value": -3.079244230667636,
"date": 1420454645732
},
{
"date": 1420536181232,
"value": -4.161348258391507
},
{
"date": 1420589985373,
"value": -5.273878399739519
},
{
"value": -6.418582324043852,
"date": 1420675284912
}
],
"type": "daily"
},

Mysterious Missing Dailies

No tags are enabled (I don't use HabitRPG tags, anyway, and have just hit the "clear" button several times). And I have checked that they aren't just greyed-out dailies. These three are really not showing up anywhere except the DDT.

[crookedneighbor]

I'm starting to worry about those mysterious missing dailies. A value of -6 shouldn't do that much damage.

For the sake of completeness, try checking off all dailies (or greying the ones you don't do) and see if you still get damage. If you do, I can walk you through deleting those dailies through the API.

[cygnoir]

I checked off all dailies and received 23.8 HP in damage.

[crookedneighbor]

Very very interesting!

If it's not too much trouble, I'd like to see the data of the mystery dailies. (feel free to remove any personal data from them, like the task name or notes) I suspect they're missing data about the days of the week they should be active. Maybe they were converted to regular todos at some point, but their data type wasn't changed from daily to todo? I believe that it is not possible to do that in the habit API, so if that is the case, it might be important on the todoist-habitrpg side to delete the habit version of the task and recreate it as a todo (or vice versa, todo becoming a daily).

Any easy way to find the ids of the dailies is to use the data display tool. Select the "Tasks Overview" link. Change the type to daily. Select the "Toggle Developer Data" on the right, just above the box displaying your dailies. The ids should become visible under the name of the dailies.

From there, you can go to the static/api page, put in your user id and api token, and then navigate to the GET /user/tasks/{id} section and put in the task's id to get the data from an individual task.

Alternatively, you could just skip the data display tool step altogether and use Get /user/tasks and navigate to the mysterious tasks by searching for their names. Whichever is easier for you.

If you don't feel comfortable doing that, and just want to delete the mystery tasks, use the DELETE /user/tasks/{id} option. Regardless, of whether or not you decide to post the daily data, you should delete those tasks and see if you still get damage.

And, again, you rock. Thanks for being such a good sport while we figure this out.

[cygnoir]

OK, so I found the culprits in the DDT and their magnitudes are off the charts. No wonder I am taking all this damage! My problem is that I am unable to delete these mystery tasks with the API page. I've entered my user ID and API token at the top of the page, navigated to the delete option, and entered the ID of the task I want to delete, but it doesn't do anything. I've checked the DDT and all of the tasks I am trying to delete are still there.

Help?

[cygnoir]

Sorry, I meant to add a screenshot of one of them. Here it is. As you can see, the date is "undefined" which means it doesn't show up at all in HabitRPG. Also, I still have not been successful in deleting any of these undefined dailies and soon it's going to cause daily death ... please help!

[crookedneighbor]

Okay, we might need to seek help from on high (a database admin) to remove those tasks.

I'm one of the site moderators for HabitRPG, so I can get in touch with the appropriate person. Can you post your user id here, along with the ids of the undefined dailies?

In addition, can you give me a little narrative of the history of this particular daily? It'd be helpful to know every step of the process from the creation of the daily in Todoist until now so we can try to replicate it and see at what point it became undefined and fix the module to prevent it from doing the same thing in the future.

Again, thank you for being so awesome, patient and accommodating while we figure this out.

[kusold]

@cygnoir Have you tried deleting it using this tool? https://habitrpg.com/static/api#!/user/user_tasks_id_DELETE

Make sure you enter your user id and api key at the very top of the page, and your id (ee--7b).

[crookedneighbor]

@kusold Yes, it looks like @cygnoir was unable to delete the task using the API.

@cygnoir, just to double check, could you create a dummy task, get its id and try to delete it?

You also may want to try deleting it on the mobile app (if it appears there).

[cygnoir]

@crookedneighbor @kusold Hi there. I did use exactly that to attempt to delete these rogue (no pun intended) dailies and it didn't work.

As @crookedneighbor suggested, I tested it with a dummy daily and was able to delete that with the API, no problem. So clearly I have broken the universe (or just my part of it) and I do apologise. ;)

So far as a narrative ... I wish I could. I'm pretty sure that, as you mentioned, I either turned these dailies into regular to-dos or deleted them and then lost track of what was going on there. I'm not used to futzing around with my to-do list so much and I don't have a good sense of what exactly I did.

My user ID:
a202659f-68c0-41c4-ace4-3bbc1272cf6d

The dailies of doom:
ee7497ec-6f6b-4719-a7e5-9c978202627b
825637b0-5f20-4f0e-8ca7-72797a097e9a
d40181a8-96e7-4f3d-857a-c356a14017a2
13949132-10a6-401e-8cae-c9e86e13edd7
5baf4c65-ba28-425f-9932-ded840b97665

[cygnoir]

As a side note, is there any way to delete a habit that has no ID? I created a daily less than a week ago to repeat every Monday. Somehow it got turned into a habit and now has an undefined ID. I can't delete it in HabitRPG as a result.

[kusold]

OK. We'll have to get a HabitRPG admin to delete that task and code in some logic to make sure this won't happen again. Unfortunately I won't be able to tackle this tonight (GO BUCKS!), but this is a high priority.

@crookedneighbor Can you get in touch with someone? Also, I think @lefnire might be interested in this issue.

[cygnoir]

It's no bother; I've aborted the test quest and checked into the tavern so I won't die from the doom dailies.

Thanks for all your help, @kusold and @crookedneighbor!

[Alys]

@kusold when changing a task from one type to another, it's really important to change the type attribute of the task. When HabitRPG is processing the task (in particular, when it does cron calculations for damage from Dailies), it looks at that attribute instead of which array the task is in.

@cygnoir I've deleted those "dailies of doom" you mentioned above. :) They weren't appearing in your Dailies list because they had been added to the To-Do list, but they were being treated as Dailies for cron.

@kusold Ping me if there's any other HabitRPG database problems you need to have sorted out, but if you could adjust the code for the type issue, that would be lovely. :)

[crookedneighbor]

@Alys I don't think changing the type works as you have described. I just copied the contents of a habit and used the PUT /user/tasks/{id} on the static/api page to update a daily with the data of a habit, and all it succeeded in doing was to create a daily with habit characteristics:

If I refresh, the direction options disappear, but this is what it looks like when I edit it (nothing below difficulty options):

[crookedneighbor]

I am also unable to delete it from the web app (even though I can see it) or the api. I did this testing on my local install, so I can edit the db directly.

[Alys]

@crookedneighbor I haven't looked into how the user interface handles it, and it's of less importance when it's obviously handling it wrongly because a user can see that there's something wrong and can report it. The effects on the cron code are more critical and are less visible. Cron definitely does daily-style processing for tasks that have the type set to daily. It's very important that a task's type matches the array that the task is in (and all other attributes should be consistent too of course).

[crookedneighbor]

Here's the task object. Interestingly enough, it retains some of it's daily attributes, even though its type is "habit".

{
    "text": "new habit",
    "challenge": {},
    "attribute": "str",
    "priority": 1,
    "value": 0,
    "tags": {},
    "notes": "",
    "dateCreated": "2015-01-06T21:50:55.606Z",
    "id": "13cbc6da-584c-47e3-a972-233a469bd574",
    "streak": 0,
    "checklist": [],
    "collapseChecklist": false,
    "repeat": {
      "su": true,
      "s": true,
      "f": true,
      "th": true,
      "w": true,
      "t": true,
      "m": true
    },
    "completed": false,
    "history": [
      {
        "date": 1420676875258,
        "value": -1
      },
      {
        "value": 0.025956704627064786,
        "date": 1420676984897
      },
      {
        "date": 1420738337202,
        "value": -0.9733783617683589
      },
      {
        "value": -1.9986354035070777,
        "date": 1420738389685
      },
      {
        "date": 1420818243308,
        "value": -3.0511857564916074
      },
      {
        "value": -4.132512016169044,
        "date": 1420898475716
      },
      {
        "date": 1420901518437,
        "value": -5.244220363156963
      },
      {
        "value": -5.160633315601223,
        "date": 1421017803219
      },
      {
        "date": 1421020561846,
        "value": -6.302020161904617
      },
      {
        "value": -7.477284061619849,
        "date": 1421089386309
      },
      {
        "date": 1421184619549,
        "value": -8.688481512325257
      }
    ],
    "type": "habit"
  }

[kusold]

@Alys Can you please expand upon this array you keep mentioning? I'm not sure what you are referencing. This tool only interacts with HabitRPG via the API.

[crookedneighbor]

@Alys Right, what I'm saying is that changing the type is not sufficient. If a user has created a daily that repeats on MWF in Todoist, and then decides to remove the repeating code and convert it to a todo, it's not a matter of just changing the type, because it remains in the dailies array, even though it has the type "todo".

So, if that happens, the module should first delete the daily and re-create the task as a new todo. That's the only way to remove it from the dailies column.

@Alys, does that sound correct to you?

[Alys]

@kusold In the HabitRPG database, each user has an object ("array" is a bit of a loose term) for dailies, another for todos, etc. Don't worry about that. Just please ensure that whenever a task is created for a certain type of task, all of its attributes exactly match that type of task. For example, if a todo is created by converting a daily into a todo, then the new todo must have all the correct attributes as if it was a todo created from scratch. Create a new task using HabitRPG itself, then carefully examine that tasks using data from the API. Do the same thing for all four tasks types (including rewards). All tasks created/modified by your tool need to match what you are seeing there.

@crookedneighbor I'm sorry, I really don't have time to look into these bugs in detail. No, of course changing the type is not sufficient. It's essential, but so is changing anything else that needs to change if tasks are being moved between types.

[lefnire]

FYI from @crookedneighbor and my convo:

Tasks are stored as lists: user.habits/dailys/todos/rewards. the only reason type is stored in the task object (since it should be able to figure it out given the list it's in) is convenience throughout html,
so changing type won't put it in the other list, it will just change type, but it'll still be in habits and will cause issues. So you can't change type via the API, it will just cause problems. However, if someone's willing to massage this untested gist https://gist.github.com/lefnire/afbcf83f6d1dac6617dd , it could be supported without much difficulty.

[cygnoir]

@Alys Thank you very much for deleting my doom dailies! I see two other dailies that are about to cause me problems as well (on Monday, when they are due, and they are not appearing in HabitRPG anymore but they show in the DDT). I'm not sure where I should ask for help on clearing them, though.

[Alys]

@cygnoir Can you tell me the text of the Dailies? If you'd prefer to keep it private, email it to lady_alys@oldgods.net. Alternatively, if you happen to know the unique ID numbers of the tasks (they'll look like eeb9b8f0-ae33-4032-ae19-acaf50180e82) paste them into this comment. Non-admin users won't be able to get any private data from those IDs.

[Alys]

For the record, @cygnoir has sent details of the Dailies to me and we'll be discussing it by email.

[kusold]

Thanks for your help @Alys, and thanks for sticking with us @cygnoir! It seems like we have some good stories/bugs to tackle from this thread. I'm going to close this issue out now that it seems to be resolved.