Changes in Date or Time on Deadlines or Scheduled Events Not Syncing Since Recent Changes

[alraban]

I've been using org-caldav successfully for several years and previously when I changed the date or time on a deadline or scheduled event org-caldav-sync would sync the changes to my nextcloud calendar. Today I updated to the version that's current in MELPA and changes to dates or times in the org file no longer sync to the nextcloud calendar. When I run org-caldav-sync it reports nothing to sync, no errors. It's as though nothing had changed in the source files.

To be clear, brand new deadlines or scheduled entries sync to the calendar correctly, as do deletions, but date changes on existing todo entries with deadlines or scheduled timestamps no longer sync. This is potentially very disruptive for folks who make extensive use of repeating entries and/or org-habits.

That said, I'm very pleased to see some development activity on this repo after a long hiatus, and thank you very much for your work on improving org-caldav!

Please let me know if I can provide any additional configuration or debugging information.

Edited to add: Events without a todo status and a bare timestamp (i.e. without a deadline or scheduled keyword) seem to still sync date/time changes correctly. This issue only seems to affect date or time changes for todo entries that have deadline dates or scheduled dates.

[jackkamm]

Could you please provide the following info:

• Emacs version
• Org-mode version
• Value of:
  • org-caldav-sync-todo
  • org-caldav-sync-changes-to-org
  • org-icalendar-use-scheduled
  • org-icalendar-include-todo
• An example of an Org entry where the timestamp isn't getting updated on nextcloud

For TODO sync, the SCHEDULED timestamp is mapped to icalendar's DTSTART, bare timestamp isn't mapped to any icalendar field (though it may be included in the DESCRIPTION). We actually inherit this behavior from upstream ox-icalendar.el (see org-icalendar--vtodo).

Also, TODO sync was only recently added to org-caldav (#218). Were you using the main release of org-caldav before, or an experimental branch with the VTODO support?

[alraban]

To answer your last question first, I was using the main release of org-caldav from this repo through MELPA. The old org-caldav supported syncing todos that had date stamps (such as scheduled or deadline date stamps) to calendars, but you're right that the old org-caldav did not support syncing todos without datestamps. The sync functionality for dated todos has been in place for at least 4 or 5 years I think (it's worked since I've been using the package). So this was a breaking change for a long established workflow, and I appreciate your help very much!

I did notice that the new version now syncs todos that didn't have any date stamps, which is a nice improvement to the old functionality.

My emacs version is 28.2
My org version is 9.6.1
org-caldav-sync-todo is nil (default)
org-caldav-sync-changes-to-org is title and timestamp (default)
org-icalendar-include-todo is set to t
org-icalendar-use-deadline is set to '(event-if-todo event-if-not-todo todo-due)
org-icalendar-use-scheduled is set to '(todo-start event-if-todo event-if-not-todo)

As a test, I tried enabling org-caldav-sync-todo, but that resulted in no change to the behavior I observed above (I still saw no timestamp changes getting synced).

Also, as a testing note: when I upgraded to the new version of org-caldav I noticed that all my non-dated todos got synced to my tasks, even though I had org-caldav-sync-todo set to nil, which is interesting. I suspect the fact that I had org-icalendar-include-todo already set may have caused that? In any case you may want to check and make sure that the org-caldav-sync-todo code-path is working as expected, as I would not have expected to get them all synced with that set to nil if I were coming to org-caldav as a new user, if you see what I mean?

I've included two example entries that exhibit my problem:

** TODO Find New Barber
DEADLINE: <2023-02-02 Thu>
:PROPERTIES:
:ID:       687f2e81-8bf4-4372-87a2-8e6146052412
:END:

** TODO Wind Clock
SCHEDULED: <2023-02-01 Wed .+2d/3d>
:PROPERTIES:
:LAST_REPEAT: [2023-01-30 Mon 09:53]
:STYLE: habit
:ID:       ed4d15ef-a895-46fc-b928-e5b95cc88048
:END:

To be clear both of those entries get synced to the calendar correctly when first created, and if I delete them they get removed correctly. The issue is if I change the datestamp in my org file no changes are propagated to the calendar (which used to work flawlessly). This is especially important for entries like the Wind Clock entry which gets updated every few days. But the issue exists with a simple deadline like my first example too.

Thank you very much for the speedy reply, I sincerely appreciate you taking the time to help figure it out!

Edited to Add: I can confirm that reverting to an older version makes everything work correctly, but I had to step back through several older versions, essentially going back to the last changes dengste made in 2020. Specifically, rolling back to this commit resulted in seeing time or datestamp changes getting picked up again. Interestingly, rolling back alone wasn't enough, I also had to remove the sync states to start fresh, but after that everything works normally. I am not an elisp expert, so unfortunately I can't seem to spot which specific changes might've caused it, but please let me know if I can provide any more information to help reproduce the issue.

[jackkamm]

I think the following setting will solve your issue, can you please try it:

(setq org-icalendar-include-todo nil)

Explanation: your org-icalendar settings export multiple icalendar entries for each org entry: org-icalendar-include-todo will create VTODO (that ends up in Tasks), while org-icalendar-use-scheduled/deadline will create VEVENT (that ends up in Calendar).

However, org-caldav does not handle these types of duplicates well. Therefore, you should unset org-icalendar-include-todo to prevent the VTODO creation.

The problem happens in org-caldav-update-eventdb-from-org, specifically

org-caldav/org-caldav.el

Line 840 in cc1b820

(org-caldav-event-set-md5 event md5)

The first time the event is encountered, it will notice the md5 changed, mark the event as changed-in-org, and update the md5 in database. The second time the event is encountered, the md5 sum will be the same as in database, so org-caldav thinks it's unchanged, and marks it as up-to-date.

The underlying issue was already present in older versions of org-caldav, for example if you try the following event:

* TODO Find New Barber
SCHEDULED: <2023-02-02 Thu> DEADLINE: <2023-02-02 Thu>

then the ox-icalendar export will create duplicate events (for Scheduled and Deadline timestamps), which will cause the same error with being unable to update the event.

I noticed that all my non-dated todos got synced to my tasks, even though I had org-caldav-sync-todo set to nil, which is interesting. I suspect the fact that I had org-icalendar-include-todo already set may have caused that?

Yes, we should perhaps bind org-icalendar-include-todo to be consistent with org-caldav-sync-todo, rather than relying on the user to set them in a consistent way.

Also, the duplicate entries can explain the different behavior with dated/non-dated todos. It appears the dated todos don't show up in Tasks because of the duplicate calendar entry.

Finally: for your config, rather than unsetting org-icalendar-include-todo, an alternative option would be to reset org-icalendar-use-scheduled/deadline to their default values, and set org-caldav-sync-todo. Then, all TODOs (including scheduled/deadline) will end up in Tasks rather than Calendar. But unfortunately recurring VTODO export is not yet implemented.

[alraban]

After testing that does seem to fix it, and it makes sense why that would work that way.

Am I correct in concluding based on your answer that there's no way to get datestamped todos to the calendar, but also have undated todos go to tasks? Basically if I understand you correctly one has to pick whether you want todos on the calendar or in tasks, but there's no way to get both? No worries if so, just making sure I'm understanding.

It's much more important for my workflow that I get the calendar entries and alarms for my timestamped todos, so I'll be sticking to calendar if I have to pick, but it would be nice to be able to have less urgent items on the task list too if that makes sense.

In any case, I greatly appreciate the help and troubleshooting! Thank you.

[jackkamm]

Am I correct in concluding based on your answer that there's no way to get datestamped todos to the calendar, but also have undated todos go to tasks? Basically if I understand you correctly one has to pick whether you want todos on the calendar or in tasks, but there's no way to get both? No worries if so, just making sure I'm understanding.

Yes that is my understanding of the current situation.

It's much more important for my workflow that I get the calendar entries and alarms for my timestamped todos, so I'll be sticking to calendar if I have to pick, but it would be nice to be able to have less urgent items on the task list too if that makes sense.

Understandable, especially since recurring tasks are not yet implemented. Still, I'd recommend to consider switching to tasks after we've added that -- it's more natural to export Org TODOs as iCalendar tasks, and Nextcloud shows upcoming tasks both on the main dashboard and in the calendar.