Allow sync to single ics file (per calendar)

[slavkoja]

Please, add option to sync into one (single) ics file, which then can be used as foreign file for Orage (the XFCE's light calendar) - there is a lack of CalDAV clients in Linux, if one don't want to use Thunderbird or Evolution, then working with shared calendars can be problematic ;)

I think initialy about merging synced ics files into one and this seems to be trivial task, but split them back can be more trickly...

[untitaker]

Does orage use a single file as a storage format?

[untitaker]

It's rather unlikely that i will want to support this. The hack used for supporting single-file HTTP resources is awful enough.

[slavkoja]

Orage uses only one own calendar in one ics file, but it allows to add "remote" files (on local FS) and thus allows to use more calendars (and ics files as storage) at once. The remote files can be additionally marked as RO, but can be RW too. You can read more here http://www.kolumbus.fi/~w408237/orage/orage_doc_4_7.html#exchange-foreign-tab

IMO if you have one-file support, it can be simplest, in mean that there is not needing to start from scratch. I am not experienced Pythonist, but merging seems to be simple task - i did some initial playing with the icalendar module. There can be any VEVENT/VTODO/VJOURNAL simple added into one calendar object instead putting into separate file. If you have HTTP one file implemented, thus this must be the same, only by opposed direction.

The simplest way seems to be to store internally in vdir as usual and add some pre/posprocessing to achieve synchronization of the separate file as result and then reuse the most of existing code. But i cannot help, then it is your decision.

[untitaker]

The icalendar module is not as useful as it might seem, as simply extracting events, tasks and journal components is not enough -- timezones that appear outside of them have to be copied into each of them. Furthermore, starting to depend on icalendar would only solve the calendar stuff, addressbooks are obviously not parsable with icalendar, which means that we still have to maintain the custom hack.

I don't have a personal motivation to implement this, but everybody is welcome to try it themselves. A proper implementation would have its own module in vdirsyncer.storage, subclass vdirsyncer.storage.base.Storage and use split_collection to load the file and yet another hack to save the file back. I'd be delighted if somebody finds a way of not loading the whole file into memory, but i don't think anybody cares to try that hard.

[untitaker]

This is actually a lot easier than i thought since one can use icalendar.cal.Component to parse anything vobject-like, even if it's a vcf.

[untitaker]

Btw you could use radicale additionally to vdirsyncer for now. It is a caldav server which stores its stuff in a single ics file.

[slavkoja]

For now i have my own radicale on my own server, with ssh access, then i use rsync to sync from server to local machine (only one way) and this synced .ics as RO remote file in Orage. Do you mean to use two radicale servers with vdirsyncer between them?

[untitaker]

I wasn't aware that you already use radicale on the server side. In that case your approach works fine. You might want to check out Unison, which is like rsync but syncs in both directions. However, it can't handle the case where the file changes on both sides -- which is why a local radicale installation might still be helpful to let vdirsyncer figure out the conflict resolution. But for now i'd just swap rsync for unison, i am going to try to implement your feature request in the near future i think.

[slavkoja]

Thanks for willingness to implement this – that will be one small step for man; but one giant leap for XFCE's mankind :-)

I will learn more about unison and try differences, thanks to point to it

[untitaker]

Can you try the singlefilestorage branch?

pip install --user --upgrade --force-reinstall git://github.com/untitaker/vdirsyncer.git#singlefilestorage

Or

git clone git://github.com/untitaker/vdirsyncer.git
git checkout singlefilestorage
pip install --user --upgrade --force-reinstall .

[untitaker]

Eh, i should actually make this usable from the CLI...

[untitaker]

Okay, now. Usage:

[storage foo]
path = ~/path/to/file.ics

[slavkoja]

OK, tested it and it seems as working. I was success to modify the (one) VTODO and then sync it back to the server. I encouter one small error while testing (and trying), i can post more detailed report about configuration - there are more changes required in storage configuration (i did by try and fail).

I have only one suggestion now – it seems, that while syncing back to the server there was synced all calendar components, not only the one which was changed – is it intended behavior? The "LAST-MODIFIED" property was changed only in one item (as expected), but for any item there was a line:

Copying (updating) item 86359880-7eb1-4d71-a2fd-8c2de5bf9a4d to <CaldavStorage(**{'username': 'xxx', 'url': 'https://calend.slavino.sk/xxx/hlavny.ics/

[untitaker]

That is interesting... I'll try and see if i can reproduce this error.

[untitaker]

I don't seem to be able to reproduce this error. It might be that icalendar does some inappropriate reformatting which leads to this at the beginning, but there is little i can do about it.

[slavkoja]

It isn't related to Orage. I removed the file from orage and did some syncs with minimal delay (only hit up and enter) and always are synced all items. Once from server, send to server, then from server etc...

I am able to debug it, but please, can you tell me by by what you decide to update item? Or i can sync some times, copy both ics files (local and from radicale) and publish it somewhere, to you can see both. But tomorrow :)

[untitaker]

I hash the item and if the hash changes i decide that it needs to be uploaded to the server. Before hashing i strip off any whitespace, so newlines between items shouldn't matter.

[slavkoja]

If you re comparing the hash from server and hash from local, then these files differs from start. I was curious after the fist sync, then i did diff (more precise i used the meld for this) on them (because i have one synced via rsync, it is simple for me) and the meld provides a lot of differences. I didn't check these differences in the depth. But i can create one very simple calendar with only one item, to better see it. Or, if you want, i can create account for your testing on this server, but credentials must be send out of public channel ;-) I can use IRC (at freenode) or XMPP.

[untitaker]

I don't compare hash from server and from local. For the server, i use the etags the server provides. Initially, i try to match up items by their uid or by content, and save that information in the status file.

[untitaker]

I don't think i need a test account, i use radicale myself. I will try to recreate the exact setup you're using, with radicale and orage.

[slavkoja]

You don't need test with Orage - save your time, as I wrote, it happens without using the file in Orage (i removed it from settings and to make sure, i moved destination file for syncing too), then Orage has not impact on it.

It can come from etag, because i am proxying the radicale via nginx (nginx as reverse proxy for radicale) to i can use standard 80/443 ports, i will check etag provided by proxy and i will try to connect directly to compare results.

[slavkoja]

OK, i did first tests and upload related .ics files - it differs, here is my action steb by step:

0, delete all vdirsyncer stuf (.ics file and status), to start from zero
1, download ics file from radicale (scp) and name it as hlavny-1.ics (http://anfo.slavino.sk/hlavny-1.ics)
2, do first sync (local file created) and make copy synced ics file as test-1.ics (http://anfo.slavino.sk/test-1.ics)
3, do second sync (changes written to server), download radicale file again and name it hlavny-2.ics (http://anfo.slavino.sk/hlavny-2.ics)
4, do third sync (changes taken from server), copy local file as test-2.ics (http://anfo.slavino.sk/test-2.ics)

If you take look into files, you can see, that they differs in some VTODOs and others are identic. I hope, that there is not an error in ics file - items was created and modified by the Thunderbird's Lightning. Only one item was changed via Orage, but it seems as not different (but i forget which item exactly i was trying in change testing). Please, can you try this calendar i your own setup - nothing private (nor useful for others) in it.

Now i go to etag check...

[untitaker]

I changed singlefilestorage to remember the order. Please rerun your procedure after doing:

pip install --user --upgrade --force-reinstall git://github.com/untitaker/vdirsyncer.git#singlefilestorage

[slavkoja]

Dont worry about pip, i have own deb package builded from github sources, i hope that git pull ont the singlefilestorage branch will be enough.

BTW, the nginx (proxy) is not involved in this, i connect directly to radicale (0.8) and behavior is the same.

[untitaker]

I pushed some fixes, please repull or whatever if you already have.

[slavkoja]

I tried your changes:

On branch singlefilestorage
Your branch is up-to-date with 'origin/singlefilestorage'.

First create new sync file:

vdirsyncer sync
Syncing sla_calendar
Copying (uploading) item 86359880-7eb1-4d71-a2fd-8c2de5bf9a4d to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item 53b9f269-3eb0-4121-a2e8-a097d8d9bcec to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item 718cc169-44e2-400c-8d52-46679175b9cf to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item c04dc7d4-ee21-4214-a584-89cda51f1c91 to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item 088436fa-2185-4fa6-90b7-fe5069902e47 to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item a466129e-5167-4248-bda9-420b0e3c800a to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item 5de4f13f-bb89-4e04-a8ad-a5b71c681640 to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item c59fe0dd-0882-4db4-a4f0-9d30c47d9b88 to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item 9643fd80-e490-44d7-ad5e-c0ba79180f38 to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item 4508134e-f120-420c-9bfa-443d68479e05 to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item 9cbbc02c-a1a8-4262-8dd9-f5f90f0e10d3 to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item 2f3cd688-f084-4056-80c4-aa9b1fec84e4 to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item efdc8f16-4ef2-41b0-bfe5-fd2d7c39b587 to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item a41e8760-e99b-4aa5-9f1f-b9ea7db5819d to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item 647970a7-25e3-4cef-bf4d-6ab4a7e62e0b to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item d1ab96ff-d2d5-44e9-9a1f-9374c37cef16 to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>
Copying (uploading) item 2a4aed99-cbdc-42d0-85b6-5dcdd6dc8e65 to <SingleFileStorage(**{'path': '/home/slavko/testcal/test.ics'})>

Then sync again, with minimal delay:

vdirsyncer sync
Syncing sla_calendar
Copying (updating) item 86359880-7eb1-4d71-a2fd-8c2de5bf9a4d to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item 53b9f269-3eb0-4121-a2e8-a097d8d9bcec to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item 718cc169-44e2-400c-8d52-46679175b9cf to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item 088436fa-2185-4fa6-90b7-fe5069902e47 to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item a466129e-5167-4248-bda9-420b0e3c800a to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item 5de4f13f-bb89-4e04-a8ad-a5b71c681640 to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item c59fe0dd-0882-4db4-a4f0-9d30c47d9b88 to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item 4508134e-f120-420c-9bfa-443d68479e05 to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item c04dc7d4-ee21-4214-a584-89cda51f1c91 to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item 2a4aed99-cbdc-42d0-85b6-5dcdd6dc8e65 to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item 2f3cd688-f084-4056-80c4-aa9b1fec84e4 to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item efdc8f16-4ef2-41b0-bfe5-fd2d7c39b587 to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item 9643fd80-e490-44d7-ad5e-c0ba79180f38 to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item a41e8760-e99b-4aa5-9f1f-b9ea7db5819d to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item 647970a7-25e3-4cef-bf4d-6ab4a7e62e0b to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item d1ab96ff-d2d5-44e9-9a1f-9374c37cef16 to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>
Copying (updating) item 9cbbc02c-a1a8-4262-8dd9-f5f90f0e10d3 to <CaldavStorage(**{'username': 'xxx', 'url': 'http://raspi.skk:5232/xxx/hlavny.ics/'})>

Both files are really updated - the file timestamps was changed. I took copy of the files again and then do two synces. Now are both (local and remote) files identical after syncing.

[slavkoja]

I rearange manually ics file from remote and then did diff with local one and i can tell:

1, beside order, there are only two different lines, both directly under VCALENDAR:

PRODID:-//Radicale//NONSGML Radicale Server//EN
VERSION:2.0

2, All VTODO items are identical (include values order), only their order differs.
3, VTIMEZONE is identical, only values order differs
4, DAYLIGHT and STANDARD are identical, only values order differs

I don't know, if the order of values and/or subcomponents is meaningful.

But IMO the problem is coming in the local side, where the local file is always uploaded. You are wrote, that you are calculating the hash and compares the etag. Initialy i think, that you mean the HTTP etag, but it will be icalendar etag (true?) and this must to change when new (as identical) content is uploaded, then it is OK, that the local file is always updated, because the server side was updated by previous sync, even without changes.

I did some inspection via wireshark and i see, that the server send the same etags (e.g. <getetag>"320619659"</getetag>) for particular items, or more precise it provides identical XML repsonse to REPORT request.

[untitaker]

Hmm, i pushed another commit. I am not really sure what else there is to do.

[untitaker]

An issue i opened related to this:
collective/icalendar#133

[slavkoja]

IMO, order is only cosmetic issue, but can be nice to have it :)

Please, can you point me into code, where the etags are get from remote, how is etag (hash) calculated for for local and where is decision if upload or not particular item? I am not very experienced Pythonist, but i am patient debugger and print is my friend :) And finally, i am motivated in this. I try some investigation and perhaps i get some useful idea.

[slavkoja]

OK, i found where the decision is made, and IMO i fount the problem. After some syncs i add some prints in the get_actions function, where i see:

status_etag_a: ac41b83366949ae55a46be962ea3c8e0e3cecbbeaef72e16549971714478b322
a['etag']:     7b2c994b973e9a9e77ac47253a910ca7e8d53064bd4d6ced4183dcde49e69b30
status_etag_b: "142650559"
b['_etag']:    "142650559"
action_update a b

It is only for one item, but i get this for all. These two variables are then compared to make decision and the action is upload to remote. I don't know how and where the etag is created/calculated yet, but i am sure, that localfile was not modified by me. But the next is IMO more interested. In the next sync i see:

status_etag_a: 7b2c994b973e9a9e77ac47253a910ca7e8d53064bd4d6ced4183dcde49e69b30
a['etag']:     7b2c994b973e9a9e77ac47253a910ca7e8d53064bd4d6ced4183dcde49e69b30
status_etag_b: [u'/slavko/hlavny.ics/9cbbc02c-a1a8-4262-8dd9-f5f90f0e10d3.ics', u'"142650559"']
b['_etag']:    "142650559"
action_update b a

As you can see, the status_etag_b is not etag string but list (BTW, why with quotes?) and you can see, that the etags are identical, then there is mistake somewhere with variables.

Can you please try to fix it (it must be more simple for you than for me)?

[untitaker]

Local and remote storage are somewhat similar -- they both have their own etags (in caldav they're fetched from the server, in singlefile they're hashes of the item) which change when the item does. Take a look at https://github.com/untitaker/vdirsyncer/tree/master/vdirsyncer/storage/base.py for the interface documentation, and at the files singlefile.py and dav.py for the relevant subclasses.

The actual sync logic is in https://github.com/untitaker/vdirsyncer/blob/master/vdirsyncer/sync.py. Take a look at the top docstring for a very relevant blogpost.

[untitaker]

I see the problem, but i am currently on the go. I will try later. The faulty logic is in vdirsyncer.utils.vobject.hash_item

[slavkoja]

nice :-)

[untitaker]

Now, the pushed fix might or might not completely fix the problem, but it
definetly is a step closer to the solution.

[untitaker]

@slavkoja Can you try again?

[slavkoja]

Nothing changed with the etag values. At first sync:

status_etag_a: 92423d11790c94c87ccd3e852a562fbf8a0ec477273aa36d0d6bf4eefba52a5f
a['etag']:     06f39f787d14b1f892ebbe20e56fffad66fe272de38f33382caa64b8aab5a9bb
status_etag_b: "142650559"
b['etag']:     "142650559"
action_update 'a', b'

After this, second sync:

status_etag_a: 06f39f787d14b1f892ebbe20e56fffad66fe272de38f33382caa64b8aab5a9bb
a['etag']:     06f39f787d14b1f892ebbe20e56fffad66fe272de38f33382caa64b8aab5a9bb
status_etag_b: [u'/slavko/hlavny.ics/9cbbc02c-a1a8-4262-8dd9-f5f90f0e10d3.ics', u'"142650559"']
b['etag']:     "142650559"
action_update 'b', 'a'

And this alternate, once differs a's etag and second a's etag, then a's etag againt, etc.
IMO, there is needed to fix the second (action_update 'b', 'a') problem, where status_etag_b is not a etag, but list, then never happen b['etag' == status_etag_b. This is saved in appropriate status file as list, but on second sync it is stored as etag string again...

[untitaker]

Oh jesus christ, that was some bad reading comprehension on my part. I am so sorry, i hunted for a completely different problem than the one you held directly in front of my nose.

[untitaker]

I still can't reproduce this. Can you post the status?

[untitaker]

Eh, scratch that, i found it.

[untitaker]

Okay, this is one of the last shots... it really should be working now.

[slavkoja]

OK, the second problem solved! Thanks! Now are both b's etags always the same. Now the a's etag lefts. I tried three situations:

1, I tried from scratch (deleted all local things) and do first sync, immediate after this i did second sync and all local etag differs, then all items are uploaded to remote. All next syncs (without local or remote changes) are not uploaded, nor to remote nor to local.
2, Then i itry do remote change (via thunderbird). The situation repeats – remote changes is updated to local, next sync it is updated to remote and all next syncs no update happen.
3, At last i tried local change (via txt editor) – it is uploaded to remote and all next syncs no update heappens.

It seems, that there is small problem with initial creating of the etag, when item is synced to(created in) singlefile, but this is OK after syncing (upload) to remote. Latter i will try it with Orage...

[slavkoja]

I tried the local file with Orage, where i modified one item – all works as expected! I was worrying, if Oroge will not do more changes, but not – only one item was uploaded to server! I am happy!

Then only previous mentioned problem persists, when changes are downloaded from server, they are always first time uploaded back to server. It seems as harmlessly, but if the same item will be changed in remote 2 (or more) times between syncing, there can conflict happen... Eh, i am not able to describe it better in English, i hope that it is clear for you.

[untitaker]

There is no way for vdirsyncer to tell how often a item changed on one side since the last sync, so i can't imagine how changing the item twice would lead to problems... note that changing an item and then reverting the changes still counts as a change, and i can do nothing about it.

[untitaker]

Could you give me another step by step tutorial on how to reproduce this?

[slavkoja]

Sure, there are two situation, when this happens. First new storage:

1, setup new singlefile storage
2, do initial sync – all item are uploaded to local file
3, do next sync – all items are uploaded back to server
4, all next sync (without items change) are not uploaded by any direction

The same happens, when remote item is changed:

0, have synced singlefile storage
1, do some change in remote item
2, do first sync – the changed item is uploaded to local
3, do next sync – the same item is uploaded to remote
4, all next sync (without items change) are without upload by any direction

Because i have enabled the a['etag'] and status_etag_a printing, i see that they differs in both step 2, but they are identical in step 3 in both above described situations.

The problem can happen in this situation:

1, do change in remote storage
2, do sync – the change is uploaded to local
3, do another change in the same remote item
4, do sync – because above mentioned behavior, the vdirsyncer can think, that the change happens on both ends, in the remote (real) and in the local (above behavior) too, then there can be conflict.

[untitaker]

Now it should FINALLY WORK.

[slavkoja]

Yes, i can confirm. I tested all previous situations and there are not needed uploads. Thanks!

[untitaker]

Great!

[slavkoja]

Thanks for patient and useful collaboration!

[slavkoja]

please, add check and exception handling, if the path (directories) to local file doesn't exists.

[untitaker]

Should work now.

[untitaker]

I consider this case closed. From now on, please file new issues for everything.