Validation error in iCalendar: A calendar object on a CalDAV server MUST NOT have a METHOD property.

[flart]

Hello everyone,

I used vdirsyncer for a while now but unfortunatelly I recognized differences between the two calenders vdirsyncer shoud syncronize in the past few days. So I started the syncronization manually with -vdebug option and stumbled upon the above mentioned error.
When I tried to delete the errorness calender item, the error is caused by another calender item.
Furthermore I tried to sync both calender server against the filesystem. In this case the sync process runs without any errors.

I don't know if the METHOD:PUBLISH key in the item vdirsyncer tries to store on the SabreDAV server comes from the davmail server or the exchange server behind davmail but why it doesn't matter when syncing to filesystem?

Thanks in advance for any ideas how to solve this problem.

Yours sincerely
flart

• Your vdirsyncer version :
• 0.12.1
• If applicable, which server software (and which version) you're using:
• SabreDAV 3.2.0
• davmail 4.7.2
• Your Python version
• Python 3.5.2
• Your operating system
• ArchARM
• Your config file

[general]

# A folder where vdirsyncer can store some metadata about each pair.
status_path = ~/.vdirsyncer/status/

# CALDAV
[pair ralf_calendar]
a = ralf_calendar_local
b = ralf_calendar_remote


# Synchronize all collections available on "side B" (in this case the server).
# You need to run `vdirsyncer discover` if new calendars/addressbooks are added
# on the server.
# Omitting this parameter implies that the given path and URL in the
# corresponding `[storage <name>]` blocks are already directly pointing to a
# collection each.
collections = [["KalenderRalf", "KalenderRalf", "calendar"]]

# To resolve a conflict the following values are possible:
#   `null` - abort when collisions occur (default)
#   `"a wins"` - assume a's items to be more up-to-date
#   `"b wins"` - assume b's items to be more up-to-date
conflict_resolution = "b wins"

[storage ralf_calendar_local]
type = caldav
verify_fingerprint = "****"
verify=false
url = https://ratte/SabreDAV/groupwareserver.php/calendars/ralf/KalenderRalf/
username = ******
password = ******
auth = guess
useragent = "vdirsyncer"



[storage ralf_calendar_remote]
type = caldav
url = http://ratte:1080/users/******/calendar/
username = ******
password = ******

• Use vdirsyncer -vdebug for debug output.

[...]
Doing conflict resolution for item 040000008200E00074C5B7101A82E00800000000B0A0A55B5C66D0010000000000000000100000000256A7A71FA2B34FA3A251FF1B194388...
...ralf_calendar_remote/calendar wins.
Copying (updating) item 040000008200E00074C5B7101A82E00800000000B0A0A55B5C66D0010000000000000000100000000256A7A71FA2B34FA3A251FF1B194388 to ralf_calendar_local/KalenderRalf
debug: Already normalized: '/SabreDAV/groupwareserver.php/calendars/ralf/KalenderRalf/040000008200E00074C5B7101A82E00800000000B0A0A55B5C66D0010000000000000000100000000256A7A71FA2B34FA3A251FF1B194388.ics'
debug: PUT https://ratte/SabreDAV/groupwareserver.php/calendars/ralf/KalenderRalf/040000008200E00074C5B7101A82E00800000000B0A0A55B5C66D0010000000000000000100000000256A7A71FA2B34FA3A251FF1B194388.ics
debug: {'Content-Type': 'text/calendar', 'User-Agent': 'vdirsyncer', 'If-Match': '"95f58f7da9c2e4145bd07361bb8fe8da"'}
debug: b'BEGIN:VCALENDAR\nMETHOD:PUBLISH\nPRODID:Microsoft Exchange Server 2010\nVERSION:2.0\nBEGIN:VTIMEZONE\nTZID:Europe/Berlin\nBEGIN:STANDARD\nDTSTART:16010101T030000\nTZOFFSETFROM:+0200\nTZOFFSETTO:+0100\nRRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=-1SU;BYMONTH=10\nEND:STANDARD\nBEGIN:DAYLIGHT\nDTSTART:16010101T020000\nTZOFFSETFROM:+0100\nTZOFFSETTO:+0200\nRRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=-1SU;BYMONTH=3\nEND:DAYLIGHT\nEND:VTIMEZONE\nBEGIN:VEVENT\nORGANIZER;CN=

[...]

\nSUMMARY;LANGUAGE=de-DE:Besprechung Abschiedsgeschenk Bine\nDTSTART;TZID="Europe/Berlin":20150325T100000\nDTEND;TZID="Europe/Berlin":20150325T103000\nUID:040000008200E00074C5B7101A82E00800000000B0A0A55B5C66D00100000000000000001\n 00000000256A7A71FA2B34FA3A251FF1B194388\nCLASS:PUBLIC\nPRIORITY:5\nDTSTAMP:20150324T170005Z\nTRANSP:OPAQUE\nSTATUS:CONFIRMED\nSEQUENCE:0\nLOCATION;LANGUAGE=de-DE:302\nX-MICROSOFT-CDO-APPT-SEQUENCE:0\nX-MICROSOFT-CDO-OWNERAPPTID:0\nX-MICROSOFT-CDO-BUSYSTATUS:BUSY\nX-MICROSOFT-CDO-INTENDEDSTATUS:BUSY\nX-MICROSOFT-CDO-ALLDAYEVENT:FALSE\nX-MICROSOFT-CDO-IMPORTANCE:1\nX-MICROSOFT-CDO-INSTTYPE:0\nX-MICROSOFT-DISALLOW-COUNTER:FALSE\nATTENDEE;

[...]

\nLAST-MODIFIED:20160105T220724Z\nEND:VEVENT\nEND:VCALENDAR\n'
debug: Sending request...
debug: 415
debug: {'Date': 'Tue, 13 Sep 2016 19:47:04 GMT', 'Server': 'nginx/1.10.1', 'X-Powered-By': 'PHP/7.0.8', 'Connection': 'keep-alive', 'Content-Type': 'application/xml; charset=utf-8', 'X-Sabre-Version': '3.2.0', 'Transfer-Encoding': 'chunked'}
debug: b'<?xml version="1.0" encoding="utf-8"?>\n<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">\n  <s:sabredav-version>3.2.0</s:sabredav-version>\n  <s:exception>Sabre\\DAV\\Exception\\UnsupportedMediaType</s:exception>\n  <s:message>Validation error in iCalendar: A calendar object on a CalDAV server MUST NOT have a METHOD property.</s:message>\n</d:error>\n'
error: Unknown error occured for ralf_calendar/KalenderRalf: 415 Client Error: Unsupported Media Type for url: https://ratte/SabreDAV/groupwareserver.php/calendars/ralf/KalenderRalf/040000008200E00074C5B7101A82E00800000000B0A0A55B5C66D0010000000000000000100000000256A7A71FA2B34FA3A251FF1B194388.ics
error: Use `-vdebug` to see the full traceback.
debug:   File "/home/ralf/.local/venvs/vdirsyncer/lib/python3.5/site-packages/vdirsyncer/cli/tasks.py", line 66, in sync_collection
debug:     force_delete=force_delete
debug:   File "/home/ralf/.local/venvs/vdirsyncer/lib/python3.5/site-packages/vdirsyncer/sync.py", line 228, in sync
debug:     action(a_info, b_info, conflict_resolution)
debug:   File "/home/ralf/.local/venvs/vdirsyncer/lib/python3.5/site-packages/vdirsyncer/sync.py", line 338, in inner
debug:     _action_update(ident, b, a)(a, b, conflict_resolution)
debug:   File "/home/ralf/.local/venvs/vdirsyncer/lib/python3.5/site-packages/vdirsyncer/sync.py", line 275, in inner
debug:     dest_meta['etag'])
debug:   File "/home/ralf/.local/venvs/vdirsyncer/lib/python3.5/site-packages/vdirsyncer/storage/base.py", line 17, in inner
debug:     return f(self, *args, **kwargs)
debug:   File "/home/ralf/.local/venvs/vdirsyncer/lib/python3.5/site-packages/vdirsyncer/storage/base.py", line 17, in inner
debug:     return f(self, *args, **kwargs)
debug:   File "/home/ralf/.local/venvs/vdirsyncer/lib/python3.5/site-packages/vdirsyncer/storage/dav.py", line 498, in update
debug:     href, etag = self._put(self._normalize_href(href), item, etag)
debug:   File "/home/ralf/.local/venvs/vdirsyncer/lib/python3.5/site-packages/vdirsyncer/storage/dav.py", line 474, in _put
debug:     headers=headers
debug:   File "/home/ralf/.local/venvs/vdirsyncer/lib/python3.5/site-packages/vdirsyncer/storage/dav.py", line 333, in request
debug:     return utils.http.request(method, url, session=self._session, **more)
debug:   File "/home/ralf/.local/venvs/vdirsyncer/lib/python3.5/site-packages/vdirsyncer/utils/http.py", line 74, in request
debug:     r.raise_for_status()
debug:   File "/home/ralf/.local/venvs/vdirsyncer/lib/python3.5/site-packages/requests/models.py", line 862, in raise_for_status
debug:     raise HTTPError(http_error_msg, response=self)
error: 2 out of 4 tasks failed.

[untitaker]

I don't know if the METHOD:PUBLISH key in the item vdirsyncer tries to store on the SabreDAV server comes from the davmail server or the exchange server behind davmail but why it doesn't matter when syncing to filesystem?

SabreDAV is extremely strict with regards to those methods, filesystem storage accepts almost everything.

@evert can you help us out here? There's a iCalendar exposed by DavMail that contains METHOD:PUBLISH, and @flart configured vdirsyncer to synchronize the entire calendar 1:1 to SabreDAV. Is there a possibility SabreDAV could just ignore this property? I've seen it very often from DavMail.

[evert]

Hi @untitaker ,

It is fixable, BUT: I think it's better if vdirsyncer tries to ensure to it only sends valid data to a server. It is wrong to set the METHOD: property as it has a specific meaning in iCalendar.

If you don't see a way to fix this client-side, another option might be to report the error to DavMail. I'm very much a proponent of trying to fix bugs where they originate, instead of trying to create workaround for other people's mistakes ;) If you don't see a way to fix this easily in vdirsyncer, and DavMail is not fixing or acknowledging this bug, we can add a 'fixer' on the sabre/vobject side.

Some more details about this topic here:

http://sabre.io/blog/2016/validation-changes/

Basically, by default sabre/dav runs in a mode where it will try to automatically fix broken data. We don't have fixers for everything, such as this, but adding one would be fairly easy.

After a fixer is added, sabre/dav will automatically repair the file before storing it, and not send back an etag.

[untitaker]

If you don't see a way to fix this client-side, another option might be to report the error to DavMail

I've tried that once and they didn't respond at all: https://sourceforge.net/p/davmail/bugs/598/

A fixer sounds good. Maybe I should add something like that to vdirsyncer as well.

[evert]

yea i really think that, even if we can do something on our side, the correct thing for you to do would actually be to reject invalid iCalendar or also fix it yourself. Being strict is good for everyone. I'm asked to create a workaround because you're not strict because davmail is not strict ;)

[evert]

Of course I could also not be strict, but then the next client who comes along might be strict and break based on what the server sends it :P

[flart]

OK, I understand your point of view and advocate being strict to.
But how can the problem be solved for the user?

Would it be possible for vdirsyncer to ignore such a wrong property or treat that issue as a warning instead of an error and to continue the syncronisation process with the other calendar entries?

I also would like to report that issue to davmail but unfortunatelly I don't know how to point out the problem correcly.

[untitaker]

Here's something that I'd like you to try: if you sync DavMail to filesystem
storage and then delete the lines that start with METHOD, does this really sync
back the changes, or will DavMail always reinsert the METHOD property?

Based on that we could add it to the repair command instead of doing the
fixup on-the-fly (which is much riskier in my eyes).

On Wed, Sep 21, 2016 at 01:25:03PM -0700, flart wrote:

OK, I understand your point of view and advocate being strict to.
But how can the problem be solved for the user?

Would it be possible for vdirsyncer to ignore such a wrong property or treat that issue as a warning instead of an error and to continue the syncronisation process with the other calendar entries?

I also would like to report that issue to davmail but unfortunatelly I don't know how to point out the problem correcly.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#502 (comment)

[flart]

I tried this for one element and after deleting the METHOD property on the local file storage vdirsyncer reported
Copying (updating) item 040000008200E00074C5B7101A82E00800000000D0CD710C34DCD1010000000000000000100000007FF631E4167B6D4B94B755EA83A28DF6 to ralf_calendar_remote/calendar without an error. I did a second sync and there was no update to the element on the local file storage. It seems to me like DavMail accepts this change.

[untitaker]

Does the event appear broken or not at all in other Exchange clients (web interface, no idea what's out there)

On 22 September 2016 22:38:42 CEST, flart notifications@github.com wrote:

I tried this for one element and after deleting the METHOD property on
the local file storage vdirsyncer reported
Copying (updating) item 040000008200E00074C5B7101A82E00800000000D0CD710C34DCD1010000000000000000100000007FF631E4167B6D4B94B755EA83A28DF6 to ralf_calendar_remote/calendar without an error. I did a second
sync and there was no update to the element on the local file storage.
It seems to me like DavMail accepts this change.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#502 (comment)

Sent from my Android device with K-9 Mail. Please excuse my brevity.

[flart]

That modified event is still ok in OWA and Outlook. A little bit too ok perhaps.
I think there is no change on that event at the exchange server at all.
When I did a sync to a different local directory and opened the above mentioned ics-file it has the method property again.
Therefor I'm not sure if davmail really syncs that event back to the exchange server or simply ignores the update from vdirsyncer.

[untitaker]

If you delete the items file for the first directory, I suspect that vdirsyncer will say that there is indeed a difference between those two items, and will report a sync conflict accordingly (provided that you didn't set conflict_resolution).

When I grep the sourcecode of DavMail, it appears that this property may have been added by Exchange, but DavMail has extra code to insert that property if it is missing: https://sourceforge.net/p/davmail/code/HEAD/tree/trunk/src/java/davmail/exchange/VCalendar.java#l155 I seriously wonder what is going on here. Is this code there to adapt to the requirements imposed by Exchange?

Anyway I've filed https://sourceforge.net/p/davmail/bugs/628/.

[untitaker]

By now I am starting doubt that adding a fixer in vdirsyncer would actually solve the problem since it's likely that this upload failure will pop up in other setups that don't involve vdirsyncer. I'd much rather see a fixer in SabreDAV (or sabre-vobject) for this, assuming DavMail won't fix this bug. That doesn't mean I want specifically you to do the work @evert, I totally understand why you don't want a bug filed against SabreDAV.

[untitaker]

@flart For now the best workaround I can offer you is, instead of syncing DavMail <-> SabreDAV, to sync DavMail <-> filesystem <-> SabreDAV and writing a shell script that removes all lines from your .ics files that start with METHOD:. cd into a collection/calendar directory and execute this:

sed -i 's/^METHOD\:.*//g' *.ics