Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewriting the CalDAV plug from scratch #525

Merged
merged 118 commits into from
Jan 4, 2022

Conversation

jaesivsm
Copy link
Contributor

@jaesivsm jaesivsm commented Dec 9, 2020

Based on #419 but rewritten the entire backend. I tried to make smalls patches, mentioned them on #407 but finally started rewriting the entire thing.

CalDAV features support status in GTG :

  • registering task relation through RELATED-TO parameters
    • handle PERCENT-COMPLETE based on children statuses
    • set CalDAV status to IN-PROCESS based on children statuses
    • support PARENT / CHILDREN relationships
    • ensure compatibility with tasks.org (which only writes PARENT relationship)
  • Dates / DAV UTC date compatibility
    • add support for DTSTART CalDAV property
    • add support for DTSTAMP CalDAV property
  • add support for SEQUENCE CalDAV property
  • add support for CATEGORIES to dav
  • handle DAV collection switch (create + delete) on tag edition
  • handle GTG task creation while DAV is updating (no race condition)
  • support recurring events
  • push proper task content (proper parsing of GTG.core.task.Task.content)
  • Change original feature behavior :
    • make clear that tags with the prefix are CalDAV collections (plugin text ?)
    • no default calendar, only sync for task with at least one prefixed tag
  • add new dependencies to build instructions

General :

  • Add comments everywhere
  • Complete feature unittest
    • translators
    • do_periodic_imports
    • set_task
    • remove_task

Notes :

  • Rewriten part of the Date class
    • only one cached datetime object
    • accuracy handling
    • code clean
  • Altered Task class so that it uses same GTG.core.task.Task.tid and GTG.core.task.Task.uuid on task creation

@jaesivsm jaesivsm marked this pull request as draft December 9, 2020 17:59
@eladyn
Copy link
Contributor

eladyn commented Oct 2, 2021

Hi!

I just tried this PR today (again) and I am genuinely impressed, how well this works. While using it, some minor issues arose for me, but those should be relatively simple to fix:

  • tasks that are created while the synchronization is happening, does often delete them immediately as the creation has not yet happened in the backend
  • setting the start date later than the due date is perfectly valid in GTG (I use it quite frequently), but is apparently not supported in CalDAV (Sabre refuses with: <?xml version="1.0" encoding="utf-8"?>\n<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">\n <s:exception>Sabre\\DAV\\Exception\\UnsupportedMediaType</s:exception>\n <s:message>Validation error in iCalendar: DUE must occur after DTSTART</s:message>\n</d:error>\n)
  • dragging and dropping tasks to make them subtasks or change the parent doesn't affect the relationships in CalDAV
  • the flatpak manifest does not correctly build caldav, since the name is wrong (lxml) and it is missing the dependencies in the sources

For the last point, I got it to work by replacing the caldav section with:

    {
      "name": "python3-caldav",
      "buildsystem": "simple",
      "build-commands": [
        "pip3 install --use-deprecated=legacy-resolver --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} caldav --no-build-isolation"
      ],
      "sources": [
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/4c/c4/13b4776ea2d76c115c1d1b84579f3764ee6d57204f6be27119f13a61d0a9/python-dateutil-2.8.2.tar.gz",
            "sha256": "0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/da/ce/27c48c0e39cc69ffe7f6e3751734f6073539bf18a0cfe564e973a3709a52/vobject-0.9.6.1.tar.gz",
            "sha256": "96512aec74b90abb71f6b53898dd7fe47300cc940104c4f79148f0671f790101"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/80/be/3ee43b6c5757cabea19e75b8f46eaf05a2f5144107d7db48c7cf3a864f73/urllib3-1.26.7.tar.gz",
            "sha256": "4987c65554f7a2dbf30c18fd48778ef124af6fab771a377103da0585e2336ece"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/cb/38/4c4d00ddfa48abe616d7e572e02a04273603db446975ab46bbcd36552005/idna-3.2.tar.gz",
            "sha256": "467fbad99067910785144ce333826c71fb0e63a425657295239737f7ecd125f3"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/eb/7f/a6c278746ddbd7094b019b08d1b2187101b1f596f35f81dc27f57d8fcf7c/charset-normalizer-2.0.6.tar.gz",
            "sha256": "5ec46d183433dcbd0ab716f2d7f29d8dee50505b3fdb40c6b985c7c4f5a3591f"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/6d/78/f8db8d57f520a54f0b8a438319c342c61c22759d8f9a1cd2e2180b5e5ea9/certifi-2021.5.30.tar.gz",
            "sha256": "2bbf76fd432960138b3ef6dda3dde0544f27cbf8546c458e60baf371917ba9ee"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/e7/01/3569e0b535fb2e4a6c384bdbed00c55b9d78b5084e0fb7f4d0bf523d7670/requests-2.26.0.tar.gz",
            "sha256": "b8aa58f8cf793ffd8782d3d8cb19e66ef36f7aba4353eec859e74678b01b07a7"
        },
        {
	    "type": "file",
	    "url": "https://files.pythonhosted.org/packages/2c/4d/3ec1ea8512a7fbf57f02dee3035e2cce2d63d0e9c0ab8e4e376e01452597/lxml-4.5.2.tar.gz",
	    "sha256": "cdc13a1682b2a6241080745b1953719e7fe0850b40a5c71ca574f090a1391df6"
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/97/b9/7d66fcc73bcc186ae018ee919f20498da86f095a632e4e26b3cebce584a3/caldav-0.8.0.tar.gz",
            "sha256": "5b40dc2b39950e78989d515ce6a2c1131f20cc2c413ba28f8d5b582546b40a4c"
        }
      ]
    },

(note: the legacy-resolver is necessary because of dateutil)
Including modules with multiple dependencies is also described here.

Thanks for working on this!

@jaesivsm
Copy link
Contributor Author

@eladyn thanks for the comment and the appreciation ! Sorry for not responding earlier, I'm just back from vacation.

Lemme get back to you point per point:

tasks that are created while the synchronization is happening, does often delete them immediately as the creation has not yet happened in the backend

It's a known issue. I noticed it a long time ago and I remember trying things with locks around the main entrypoints of the plugin to fix that, but in the end it didn't work. I'm guessing it's more of a problem of the machinery that execute the plugin which but, I should be working on that in another PR at the very least.

setting the start date later than the due date is perfectly valid in GTG (I use it quite frequently), but is apparently not supported in CalDAV [...]

Noted, I'll fix it in the translation GTG => CalDAV by just protecting for such value to be uploaded. The more appropriate way would be to handle it in the main UI only when the plugin is active, but implication are deeper and it might take a while to be integrated.

dragging and dropping tasks to make them subtasks or change the parent doesn't affect the relationships in CalDAV

Hum. That's a problem. I'm guessing the attributes I'm accessing on the task are somehow not up to date when set_task is called with said task. Top of my head I'd say, it's a locking issue. I don't really see a fix for it though, but maybe the new core PR might change that. I'll try stuff. It's definitely noted.

the flatpak manifest does not correctly build caldav, since the name is wrong (lxml) and it is missing the dependencies in the sources

Spoiler : I don't know how to build flatpak x), I updated it manually just as a hint to maintainers for when this will be merged. But I'll definitely update the PR with yours ! Thanks a lot.

Again thanks a lot, especially for the feedback !

@diegogangl
Copy link
Contributor

Hi @jaesivsm o/
Is this one up to date with the Date class changes we merged?
Also do you guys know any free caldav providers I could use to test this? Google now requires HTTPS + Oauth2

@jaesivsm
Copy link
Contributor Author

jaesivsm commented Nov 8, 2021

@diegogangl Yup, merged master a while back into the branch (hence the no conflicts with the base branch thingy :) The diff with master in the GTG.core.dates module is, since the merge, very small (one line).

It's basically ready for a merge with master.

NTH thingies mentioned somewhere above:

  • a little bug about constrained dates that I should take the time to fix soon enough
  • recurring task that aren't supported yet

Both this things are not merge-blocking and could be solved in another PR.

This review could still benefit a review by a core maintainer. Notably:

  1. I added a context manager along side the task class and added a parameter to said class. The goal of the mechanism is to prevent signal spam when building / editing a task. Maybe it's a bad design, not sure.
  2. I didn't run the flatpak command to build package (didn't find time / documentation / howtos and I don't know how ^^)

@diegogangl
Copy link
Contributor

Awesome o/
I'll review this when I have some more free time, and see if I can test it somehow.
The new prop and class look ok. We mightl have to port them over to the new core later (haven't even touched backends there TBH). I'll check out the flatpak later too.

I can merge this, what GH won't let me do is rebase and merge, which is how we usually merge PRs (easier for @nekohayo to make changelogs afterwards)

Copy link
Contributor

@diegogangl diegogangl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM in general, I'll try to test soon.
Unit tests run ok as well

GTG/backends/__init__.py Show resolved Hide resolved

def get_gtg(self, task: Task, namespace: str = None):
"Extract value from GTG.core.task.Task according to specified getter"
return getattr(task, self.task_get_func_name)()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not call the getter straight from the task?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean. Could you give an example ?
The Field class is suppose to be a one attribute translator made destined to get and set attribute from task. Getters and setters are set up in task_get_func_name and task_set_func_name because most of the current Task API is made through getters and setters and no property. But that's a part I'm going to alter if the new_core branch gets merged before this one.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean why not use task.get_status() or task.get_due_date() instead of going through getattr?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just a way to have a declarative way to set field translator. Having a getter allows some overriding of the Field class.

@diegogangl
Copy link
Contributor

Got this error when trying with a (probably) incorrect server url:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib64/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/core/datastore.py", line 498, in __backend_startup
    backend.start_get_tasks()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/core/datastore.py", line 693, in start_get_tasks
    self.backend.start_get_tasks()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/core/interruptible.py", line 38, in new
    return fn(*args)
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/backends/periodic_import_backend.py", line 79, in start_get_tasks
    self._start_get_tasks()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/backends/periodic_import_backend.py", line 98, in _start_get_tasks
    self.do_periodic_import()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/core/interruptible.py", line 38, in new
    return fn(*args)
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/backends/backend_caldav.py", line 111, in do_periodic_import
    self._do_periodic_import()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/backends/backend_caldav.py", line 139, in _do_periodic_import
    self._refresh_calendar_list()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/backends/backend_caldav.py", line 218, in _refresh_calendar_list
    principal = self._dav_client.principal()
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/davclient.py", line 339, in principal
    self._principal = Principal(client=self, *largs, **kwargs)
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 388, in __init__
    cup = self.get_property(dav.CurrentUserPrincipal())
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 170, in get_property
    foo = self.get_properties([prop], **passthrough)
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 200, in get_properties
    properties = response.expand_simple_props(props)
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/davclient.py", line 257, in expand_simple_props
    self.find_objects_and_props()
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/davclient.py", line 182, in find_objects_and_props
    responses = self._strip_to_multistatus()
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/davclient.py", line 125, in _strip_to_multistatus
    if (tree.tag == 'xml' and tree[0].tag == dav.MultiStatus.tag):
AttributeError: 'NoneType' object has no attribute 'tag'

@diegogangl
Copy link
Contributor

Another one, when the server rejects you:

2021-11-12 19:44:01,740 - INFO - backend_caldav:_do_periodic_import:137 - Running periodic import
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/core/datastore.py", line 498, in __backend_startup
    backend.start_get_tasks()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/core/datastore.py", line 693, in start_get_tasks
    self.backend.start_get_tasks()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/core/interruptible.py", line 38, in new
    return fn(*args)
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/backends/periodic_import_backend.py", line 79, in start_get_tasks
    self._start_get_tasks()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/backends/periodic_import_backend.py", line 98, in _start_get_tasks
    self.do_periodic_import()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/core/interruptible.py", line 38, in new
    return fn(*args)
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/backends/backend_caldav.py", line 111, in do_periodic_import
    self._do_periodic_import()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/backends/backend_caldav.py", line 139, in _do_periodic_import
    self._refresh_calendar_list()
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/backends/backend_caldav.py", line 218, in _refresh_calendar_list
    principal = self._dav_client.principal()
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/davclient.py", line 339, in principal
    self._principal = Principal(client=self, *largs, **kwargs)
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 388, in __init__
    cup = self.get_property(dav.CurrentUserPrincipal())
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 170, in get_property
    foo = self.get_properties([prop], **passthrough)
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 193, in get_properties
    response = self._query_properties(props, depth)
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 137, in _query_properties
    return self._query(root, depth)
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 162, in _query
    raise error.exception_by_method[query_method](errmsg(ret))
caldav.lib.error.PropfindError: 405 Not Allowed

b'<html>\n<head><title>405 Not Allowed</title></head>\n<body bgcolor="white">\n<center><h1>405 Not Allowed</h1></center>\n<hr><center>nginx</center>\n</body>\n</html>\n'

@jaesivsm
Copy link
Contributor Author

All of those errors arose deep from the caldav / vobject libraries and seem to be related to either the Webdav servers response or of some transport issue. So in a way it's not related to the plugin (phew x)
In another way it hints toward the fact that there is no proper error handling / user warning made about these. This is clearly mandatory for the feature to hit master, sadly I don't know about the internals of GTG on pop ups and so. But I'll try to muster something (help is of course more than welcome :)

@diegogangl
Copy link
Contributor

There isn't much in the way of a standard error dialog TBH. We should look into it once we get to Gtk4 + libadwaita and the new toasts.

What we do have is a catch-all dialog that pops up when there's an uncaught exception. It seems like these errors aren't triggering it because they are coming from threads. Maybe you can expect these, and raise another exception in the plugin code?

Also could you send me the server url you used? I forgot to ask :)

@diegogangl
Copy link
Contributor

Seems like you need to tag tasks with DAV_[CALENDARNAME] to get them to sync with a calendar? This should be documented somewhere, specially in the help files.

Also, when I get this error when I set a due earlier than a start:

Traceback (most recent call last):
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 1464, in save
    self._create(self.data, self.id, path)
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 1374, in _create
    raise error.PutError(errmsg(r))
caldav.lib.error.PutError: 415 Unsupported Media Type

b'<?xml version="1.0" encoding="utf-8"?>\n<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">\n  <s:exception>Sabre\\DAV\\Exception\\UnsupportedMediaType</s:exception>\n  <s:message>Validation error in iCalendar: DUE must occur after DTSTART</s:message>\n</d:error>\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/januz/code/gtg/.local_build/install/lib/python3.9/site-packages/GTG/backends/backend_caldav.py", line 174, in _set_task
    todo.save()
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 1466, in save
    self._create(self.vobject_instance.serialize(), self.id, path)
  File "/home/januz/.local/lib/python3.9/site-packages/caldav/objects.py", line 1374, in _create
    raise error.PutError(errmsg(r))
caldav.lib.error.PutError: 415 Unsupported Media Type

b'<?xml version="1.0" encoding="utf-8"?>\n<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">\n  <s:exception>Sabre\\DAV\\Exception\\UnsupportedMediaType</s:exception>\n  <s:message>Validation error in iCalendar: DUE must occur after DTSTART</s:message>\n</d:error>\n'

Also, could you rebase on current master and squash/reword some of commits? We should keep history as clean as possible, otherwise making a changelog becomes challenging.

@jaesivsm
Copy link
Contributor Author

Seems like you need to tag tasks with DAV_[CALENDARNAME] to get them to sync with a calendar? This should be documented somewhere, specially in the help files.

Totally ! I'm not sure how though, could you point me toward the proper file to edit ?

<s:message>Validation error in iCalendar: DUE must occur after DTSTART</s:message>\n</d:error>

(Hopefully) just fixed that :)

Also, could you rebase on current master and squash/reword some of commits? We should keep history as clean as possible, otherwise making a changelog becomes challenging.

I try to keep up with master as often as I can. All the commit in there are worthless (and reaaaaaally entangled with the rest of the commits from master). It'd be simpler to merge squash in master.

@diegogangl
Copy link
Contributor

Def fixed, did you get a chance to look at the other errors? (from caldav)
As for help, this is more @nekohayo 's territory.

@diegogangl
Copy link
Contributor

o/ @jaesivsm
I think for adding a help page would be ok for now, until we have a better API. The help files are here:
https://github.com/getting-things-gnome/gtg/tree/master/docs/user_manual/C

We should have a new page for caldav sync and a link to it in index.page

@vansia43
Copy link
Contributor

vansia43 commented Dec 7, 2021

@jaesivsm feel free to tag me when you add the help page. I am starting to take a look at making any needed updates to the other areas of the user manual. There are a couple of sync pages that we had before. I am not sure if you want to use one as a starting point since it provides general sync services info and then you could get more specific with that: gtg-add-sync.page, gtg-sync.page, gtg-create-sync.page. They’re all in the location Diego noted above. I removed these pages from the manual last year since we no longer had sync services. We could then link any new ones in the Going Further or a new section in the index. We also need to remember to add any new pages to meson.build.

@nekohayo
Copy link
Member

I'd definitely be happier about Danielle taking care of the docs here indeed, as I am too stretched and unfocused right now to be able to commit to this, and I trust she would do a better & more attentive job at it than me!

@diegogangl diegogangl merged commit 64d535a into getting-things-gnome:master Jan 4, 2022
@jaesivsm jaesivsm deleted the caldav branch June 4, 2022 11:03
@nekohayo nekohayo added the maintainability Automated tests suite, tooling, refactoring, or anything that makes it easier for developers label Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement maintainability Automated tests suite, tooling, refactoring, or anything that makes it easier for developers plugins Plugins and extra backends
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants