Discovery fails on Nextcloud Caldav/Cardav server

[GideonWolfe]

• vdirsyncer version: 0.16.8-3
• Nextcloud 18.0.4
• Python version: 3.8.5
• operating system: Arch Linux (Kernel 5.x)
• config file

[general]
status_path = "~/.config/vdirsyncer/status/"

[pair my_contacts]
a = "my_contacts_local"
b = "my_contacts_remote"
collections = ["from a", "from b"]
conflict_resolution = "b wins"

[storage my_contacts_local]
type = "filesystem"
path = "~/Documents/Contacts/"
fileext = ".vcf"

[storage my_contacts_remote]
type = "carddav"
url = "https://<nextcloud_domain>.com/"
username = "<my_user>"
password = "<secure_password>"

[pair my_calendar]
a = "my_calendar_local"
b = "my_calendar_remote"
collections = ["from a", "from b"]
conflict_resolution = "b wins"

[storage my_calendar_local]
type = "filesystem"
path = "~/Documents/Calendar/"
fileext = ".ics"

[storage my_calendar_remote]
type = "caldav"
url = "https://<nextcloud_domain>.com/"
username = "<my_user>"
password = "<secure_password>"

I had this setup working pretty much perfectly on my computer, then I built a new computer and set up everything to be exactly the same, same programs, OS, and vdirsyncer config.

When I run vdirsyncer -vdebug discover my_calendar, I get the following output

debug: Using 1 maximal workers.
Discovering collections for pair my_calendar
my_calendar_local:
debug: PROPFIND https://<nextcloud_domain>.com/
debug: {'User-Agent': 'vdirsyncer/0.16.9.dev0+gb5dd092.d20200713', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'\n    <d:propfind xmlns:d="DAV:">\n        <d:prop>\n            <d:resourcetype />\n        </d:prop>\n    </d:propfind>\n    '
debug: Sending request...
debug: 405
debug: {'Cache-Control': 'no-store, no-cache, must-revalidate', 'Content-Length': '0', 'Content-Security-Policy': "default-src 'self'; script-src 'self' 'nonce-<redacted>'; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';", 'Content-Type': 'text/html; charset=UTF-8', 'Date': 'Sat, 15 Aug 2020 19:35:08 GMT', 'Expires': 'Thu, 19 Nov 1981 08:52:00 GMT', 'Pragma': 'no-cache', 'Referrer-Policy': 'no-referrer', 'Server': 'Apache/2.4.38 (Debian)', 'Set-Cookie': 'oc9e2k9kuiv7=d80be3978cff8ba40bf5437dbfaa6180; path=/; secure; HttpOnly, oc_sessionPassphrase=<redacted> path=/; secure; HttpOnly, __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax, __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict, oc9e2k9kuiv7=<redacted>; path=/; secure; HttpOnly', 'X-Content-Type-Options': 'nosniff', 'X-Download-Options': 'noopen', 'X-Frame-Options': 'SAMEORIGIN', 'X-Permitted-Cross-Domain-Policies': 'none', 'X-Powered-By': 'PHP/7.3.17', 'X-Robots-Tag': 'none', 'X-Xss-Protection': '1; mode=block'}
debug: b''
debug: Given URL is not a homeset URL
debug: PROPFIND https://<nextcloud_domain>.com/
debug: {'User-Agent': 'vdirsyncer/0.16.9.dev0+gb5dd092.d20200713', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '0'}
debug: b'\n        <d:propfind xmlns:d="DAV:">\n            <d:prop>\n                <d:current-user-principal />\n            </d:prop>\n        </d:propfind>\n        '
debug: Sending request...
debug: 405
debug: {'Cache-Control': 'no-store, no-cache, must-revalidate', 'Content-Length': '0', 'Content-Security-Policy': "default-src 'self'; script-src 'self' 'nonce-<redacted>='; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';", 'Content-Type': 'text/html; charset=UTF-8', 'Date': 'Sat, 15 Aug 2020 19:35:09 GMT', 'Expires': 'Thu, 19 Nov 1981 08:52:00 GMT', 'Pragma': 'no-cache', 'Referrer-Policy': 'no-referrer', 'Server': 'Apache/2.4.38 (Debian)', 'X-Content-Type-Options': 'nosniff', 'X-Download-Options': 'noopen', 'X-Frame-Options': 'SAMEORIGIN', 'X-Permitted-Cross-Domain-Policies': 'none', 'X-Powered-By': 'PHP/7.3.17', 'X-Robots-Tag': 'none', 'X-Xss-Protection': '1; mode=block'}
debug: b''
debug: Trying out well-known URI
debug: PROPFIND https:/<nextcloud_domain>/.well-known/caldav
debug: {'User-Agent': 'vdirsyncer/0.16.9.dev0+gb5dd092.d20200713', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '0'}
debug: b'\n        <d:propfind xmlns:d="DAV:">\n            <d:prop>\n                <d:current-user-principal />\n            </d:prop>\n        </d:propfind>\n        '
debug: Sending request...
debug: Rewriting status code from 301 to 307
debug: 404
debug: {'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'Date': 'Sat, 15 Aug 2020 19:35:09 GMT', 'Content-Length': '19'}
debug: b'404 page not found\n'
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/discover.py", line 202, in _print_collections
debug:     discovered = get_discovered()
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/discover.py", line 131, in get_self
debug:     return self._discovered
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/utils.py", line 168, in __get__
debug:     obj.__dict__[self.__name__] = result = self.fget(obj)
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/discover.py", line 140, in _discovered
debug:     return handle_storage_init_error(self._cls, self._config)
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/discover.py", line 136, in _discovered
debug:     discovered = list(self._cls.discover(**self._config))
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 265, in discover
debug:     for c in self.find_collections():
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 225, in find_collections
debug:     return self._find_collections_impl(self.find_home())
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 201, in find_home
debug:     url = self.find_principal()
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 172, in find_principal
debug:     return self._find_principal_impl(self._well_known_uri)
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 185, in _find_principal_impl
debug:     response = self.session.request('PROPFIND', url, headers=headers,
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 388, in request
debug:     return http.request(method, url, session=self._session, **more)
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/http.py", line 160, in request
debug:     raise exceptions.NotFoundError(r.reason)
warning: Failed to discover collections for my_calendar_remote, use `-vdebug` to see the full traceback.
debug: PROPFIND https://<nextcloud_domain>.com/
debug: {'User-Agent': 'vdirsyncer/0.16.9.dev0+gb5dd092.d20200713', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'\n    <d:propfind xmlns:d="DAV:">\n        <d:prop>\n            <d:resourcetype />\n        </d:prop>\n    </d:propfind>\n    '
debug: Sending request...
debug: 405
debug: {'Cache-Control': 'no-store, no-cache, must-revalidate', 'Content-Length': '0', 'Content-Security-Policy': "default-src 'self'; script-src 'self' 'nonce-<redacted>='; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';", 'Content-Type': 'text/html; charset=UTF-8', 'Date': 'Sat, 15 Aug 2020 19:35:09 GMT', 'Expires': 'Thu, 19 Nov 1981 08:52:00 GMT', 'Pragma': 'no-cache', 'Referrer-Policy': 'no-referrer', 'Server': 'Apache/2.4.38 (Debian)', 'Set-Cookie': '<redacted>; path=/; secure; HttpOnly, oc_sessionPassphrase=<redacted>; path=/; secure; HttpOnly, __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax, __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict, oc9e2k9kuiv7=74c4b4162336a340137501a3fe07d4b5; path=/; secure; HttpOnly', 'X-Content-Type-Options': 'nosniff', 'X-Download-Options': 'noopen', 'X-Frame-Options': 'SAMEORIGIN', 'X-Permitted-Cross-Domain-Policies': 'none', 'X-Powered-By': 'PHP/7.3.17', 'X-Robots-Tag': 'none', 'X-Xss-Protection': '1; mode=block'}
debug: b''
debug: Given URL is not a homeset URL
debug: PROPFIND https://<nextcloud_domain>.com/
debug: {'User-Agent': 'vdirsyncer/0.16.9.dev0+gb5dd092.d20200713', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '0'}
debug: b'\n        <d:propfind xmlns:d="DAV:">\n            <d:prop>\n                <d:current-user-principal />\n            </d:prop>\n        </d:propfind>\n        '
debug: Sending request...
debug: 405
debug: {'Cache-Control': 'no-store, no-cache, must-revalidate', 'Content-Length': '0', 'Content-Security-Policy': "default-src 'self'; script-src 'self' 'nonce-<redacted>='; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';", 'Content-Type': 'text/html; charset=UTF-8', 'Date': 'Sat, 15 Aug 2020 19:35:09 GMT', 'Expires': 'Thu, 19 Nov 1981 08:52:00 GMT', 'Pragma': 'no-cache', 'Referrer-Policy': 'no-referrer', 'Server': 'Apache/2.4.38 (Debian)', 'X-Content-Type-Options': 'nosniff', 'X-Download-Options': 'noopen', 'X-Frame-Options': 'SAMEORIGIN', 'X-Permitted-Cross-Domain-Policies': 'none', 'X-Powered-By': 'PHP/7.3.17', 'X-Robots-Tag': 'none', 'X-Xss-Protection': '1; mode=block'}
debug: b''
debug: Trying out well-known URI
debug: PROPFIND https://<nextcloud_domain>/.well-known/caldav
debug: {'User-Agent': 'vdirsyncer/0.16.9.dev0+gb5dd092.d20200713', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '0'}
debug: b'\n        <d:propfind xmlns:d="DAV:">\n            <d:prop>\n                <d:current-user-principal />\n            </d:prop>\n        </d:propfind>\n        '
debug: Sending request...
debug: Rewriting status code from 301 to 307
debug: 404
debug: {'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'Date': 'Sat, 15 Aug 2020 19:35:10 GMT', 'Content-Length': '19'}
debug: b'404 page not found\n'
error: Unknown error occurred: Not Found
error: Use `-vdebug` to see the full traceback.
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/utils.py", line 339, in _worker
debug:     func(wq=self)
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/tasks.py", line 80, in discover_collections
debug:     rv = collections_for_pair(pair=pair, **kwargs)
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/discover.py", line 73, in collections_for_pair
debug:     rv = list(expand_collections(
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/discover.py", line 161, in expand_collections
debug:     collections = get_b_discovered()
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/discover.py", line 131, in get_self
debug:     return self._discovered
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/utils.py", line 168, in __get__
debug:     obj.__dict__[self.__name__] = result = self.fget(obj)
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/discover.py", line 140, in _discovered
debug:     return handle_storage_init_error(self._cls, self._config)
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/discover.py", line 136, in _discovered
debug:     discovered = list(self._cls.discover(**self._config))
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 265, in discover
debug:     for c in self.find_collections():
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 225, in find_collections
debug:     return self._find_collections_impl(self.find_home())
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 201, in find_home
debug:     url = self.find_principal()
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 172, in find_principal
debug:     return self._find_principal_impl(self._well_known_uri)
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 185, in _find_principal_impl
debug:     response = self.session.request('PROPFIND', url, headers=headers,
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 388, in request
debug:     return http.request(method, url, session=self._session, **more)
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/http.py", line 160, in request
debug:     raise exceptions.NotFoundError(r.reason)
error: 1 out of 1 tasks failed.

One thing I noticed: In that output, if I highlight the domain here: PROPFIND https:/<nextcloud_domain>/.well-known/caldav and visit it, I get an http page, which results in a 404.

However if I manually change the URL to https, I get the page I expect, which says

This is the WebDAV interface. It can only be accessed by WebDAV clients such as the Nextcloud desktop sync client.

So that might be the issue? My nextcloud instance itself handles http -> https redirects perfectly, so I'm not sure where the issue lies. I would be happy to assist with this issue in any way possible as vdirsyncer is the the only way I am able to sync my contacts+calendar across all devices and use them in the programs I like.

[bordapapa]

Hey, bit late for the party, but I was struggling with the same-ish issue, and based on your description, we have a similar setup.
Resolution of well known DAV addresses and http-https redirects is handled in an Nginx reverse proxy for me.
What solved my issue is putting the webdav URL for address book into the config file in this format:
url = "https://my.domain:https port/remote.php/dav/addressbooks/users/my user/address book name/"
for calendar:
url = "https://my.domain:https port/remote.php/dav/calendars/my user/calendar name/"

[dilyanpalauzov]

Does the Discovery work with Evolution or DAVx⁵?

The log above is not clear what docs curl -uusername:password -XPROPFIND https:/<nextcloud_domain>/.well-known/caldav return. If it is 404, then the server is not configured properly. If it is 301/307, then vdirsyncer does not follow the redirection.

What means

debug: Rewriting status code from 301 to 307
debug: 404
debug: {'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'Date': 'Sat, 15 Aug 2020 19:35:09 GMT', 'Content-Length': '19'}
debug: b'404 page not found\n'

Is the code 301, 307 or 404? If the inital responsce is 301, vdirsyncer follows the redirection and the new URL returns 404, thenthe server is not configured properly.

For performance reasons, I recommend to set in the configuration file the Principal-URL of the user. This skips on each synchronizations several lookups to find the principal-url (or set to the calendar-home-set - URL, if there is only one calendar-home set): I do not remember actually anymore, if vdirsyncer calls the principal-URL, I see in the code however <current-user-principal/> snippet.)

[ModProg]

For performance reasons, I recommend to set in the configuration file the Principal-URL of the user.

I don't know if I really had the same problem, got a 401, but this solved it