Easy connect

[chripo]

all changes are related to the PROPFIND request / response.
now it is possible to connect and retrieve your collections by

1. request your current-user-principal or principal-URL from any URL / path
2. request your calendar/addressbook-home-set from the principal URL
3. perform a PROPFIND with depth = 1 on the "*-home-set" URL to get all your collections

the specs taken from rfc4791 and rfc5397.

this fixes are required to get the firefox-os calendar app working.

[rngtng]

Nice! So whats the server path i have to configure my carddav Client with? Just the radicale host returns 'user not authenticated'

[chripo]

this depends on your client, which collection discovery method is used.
the stock calendar app from firefox-os, does the discovery like the described way.
in this way the inital server url / path dosen't matter because of retrieving the starting url from the current-user-principal value (kind of user home) .

radicale organizes collections and items like the following way:
/base_prefix/user-principal/collection/item

using '/base_prefix/user-principal/' as server path should work in most cases.
the base_prefix depends on your configuration default is set to '/'.

the authentication issues should be fixed in the second commit 66e38bd

[rngtng]

Thanks, well ok, that's exactly what I did - tested with latest apple clients, on iOS and Mavericks using DB as radicale backend - auth issues still remain :(

[liZe]

Thanks a lot for your work, the code is clean, that's really nice!

A couple of questions before merging:

• Is there any obvious security problem with always allowing PROPFIND requests when the user is connected?
• The "principal" URLs are following the RFC now, but this behaviour may break the compatibility with other clients. Did you try any clients except from firefox-os?

Thanks again!

[chripo]

@rngtng
did you your test with this branch or the master

[rngtng]

With this branch. With master auth works, but addressbook/contacts
obviously don't pick up the right user-collection...

[chripo]

@liZe
i'll try to answer your questions as well as i can

the security problem:
i checked it again and i don't think there is a security problem, because only PROPFIND requests are passed for authenticated users and the response contains only user related data. plus all requests whithout (so called) items will return meta- or service related data. so it's schouldn't be an issue.
furthermore binding propfind requests on "collection or item" like the master branch does will break valid requests.

about the compatibility:
of course it may break some clients, but those are broken anyway. i did some tests with agendav and mozilla thunderbirds calendar extrension, and with a self written android sync client (called radicalesync, i'll release them soon). all of them working, but they use an other or none collection discovery method. therefore they're almost meaningless.
you welcome to test your clients, and give some feedback.
a radicale instance is running at https://pim.christoph-polcin.com, user: test, password: test, with a cacert certificate;

by the way, there is an other PROPFIND tag which is not RFC compliant

• calendar-user-address-set

[chripo]

@rngtng
could you provide some radicale log output? (remove the auth header)

[rngtng]

@chripo thx for ur test server - strange this worked for me.. What auth method and Backend do you use?

Sure will provide logs soonish... Thanks!!!

[chripo]

acal on android is able to discover the principal and all calendars. the path dosen't matter.
discovery is done like the descibed way above.

1. OPTIONS
2. PROPFIND with props: resourcetype, current-user-principal, principal-collection-set, owner
3. PROPFIND & DEPTH=0 on current-user-principal with props: calendar-home-set, addressbook-home-set
4. PROPFIND & DEPTH=1 on calendar-home-set with props: displayname, resourcetype, supported-report-set, current-user-privilege-set, sync-token, supported-calendar-component-set, calendar-timezone, calendar-color
5. for each collection REPORT & DEPTH=1 with getetag , calendar-data ...

[chripo]

@rngtng
awesome, -i catched your logs too ;) this helps to improve radicale.-
backend is file, and auth is htaccess with owner_only

[rngtng]

Interesting ... So maybe DB backend is the problem. Will check asap I'm
back on machine..

[chripo]

@liZe
i reconsidered the compatibility issues.
i think (+did some curl tests) the master is broken somehow, therefore it's hard (one case) to get a half auto collection discovery to work, because of

• requests to /, /user/, /otheruser/, /foobar/ are not permitted
• PROPFIND (with DEPTH=0) on a valid collection /test/test/ will return

  <response>
    <href>/test/test/</href>
    <propstat>
      <prop>
        <current-user-principal>
          <href>/test/calendar.ics/</href>
        </current-user-principal>
        <principal-URL>
          <href>/test/test/</href>
        </principal-URL>
        <displayname>test</displayname>
        <C:calendar-home-set>
          <href>/test/test/</href>
        </C:calendar-home-set>
        <principal-collection-set>
          <href>/test/test/</href>
        </principal-collection-set>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>

using current-user-principal or principal-URL as further request path to discover the calendar-home-set (PROPFIND + DEPTH=0) will result in getting the wrong path for the next collection discovery request (PROPFIND + DEPTH=1) which will return items as collections which will lead to client processing errors.

skipping the principal discovery request and start over with a PROPFIND + DEPTH=1 on the calendar-home-set path will lead to the same problems. with a DEPTH=0 on the (wrong) principal path or using directly the calendar-home-set from the first request which is not the common / rfc way, will work but will return only one of many collections.

for this reason this patchset will get some broken clients to discover your collections fully. plus it will give rfc complaint clients to discover and access all your collections.

use this curl query and play with the path to reproduce the collection discovery:

curl -i -X PROPFIND -d '<?xml version="1.0" encoding="UTF-8"?><N0:propfind xmlns:N0="DAV:" xmlns:N1="urn:ietf:params:xml:ns:caldav"><N0:prop><N0:current-user-principal /><N0:principal-URL /><N0:displayname /><N1:calendar-home-set /><N0:principal-collection-set/></N0:prop></N0:propfind>' -u 'USER:PASSWORD' RADICALE_SERVICE_URL

[rngtng]

@chripo tested with Mac OS X contacts client, seems to work as well :) Next, I'll check my DB setup and check there..

[rngtng]

@chripo wanted to test with carddavmate, but your server returns headers:

access-control-allow-origin: *

I guess its case sensitive, so can u please update this to Access-Control-Allow-Origin: and I'll continue testing..

[rngtng]

@chripo btw.: tested with Mac OS X calendar - worked nice as well. Only Reminders (Tasks) failed when moving task from one Group to another, but I guess this is a feature not supported by Radicale (yet)

[chripo]

@rngtng from RFC2616, pg. 30. "Field names are case-insensitive." so it shouldn't be an issue. anyway i changed it to case sensitive. pls give it a new try.

i think the connect issue is related to the wildcard in Access-Control-Allow-Origin and the requiered credentials.

added to my .htaccess

<IfModule mod_headers.c>
  SetEnvIf Origin "^(.+)$" ALLOW_ORIGIN=$1
  SetEnvIf Origin "^$" ALLOW_ORIGIN=*
  Header always set Access-Control-Allow-Origin "%{ALLOW_ORIGIN}e" env=ALLOW_ORIGIN
</IfModule>

[rngtng]

@chripo hm true, wasn't the problem - carddavemate still doesn't work, See JS output here, but I guess its a config or browser (chrome) issue. Not sure if Carddav supports the easy connect at all.

Anyway to get back to the original auth problem. See the log of my local radicale here - Maybe its a problem when the addressbook.vcf resource doesn't yet exist? Therefore can you pls create another test user without any items created - I'll test my clients again. Thx

[chripo]

@rngtng i think the carddavmate issues are CORS related. i added some headers, pls give it another try.

i created a second account without collections: user: test2 password: test2

thanks for sharing your log files. i tracked down one issue / improvement. it's related to .well-known URIs rfc6764 and is fixed in 4c22edb

something is wrong with your test setup or data. there is a false resourcetype in your last PROPFIND request, it should be addressbook instead of calendar. probably this will terminate the collection discovery. you have to edit /test/addressbook.vcf.props and set the tag to "VADDRESSBOOK" instead of "VCALENDAR"

[chripo]

i couldn't get work caldavzap/carddavmate from a foreign origin, because of some eval() calls that get blocked by some CSP (X-Content-Security-Policy). i applied some header-foo but couldn't find a fix.

therefore i set it up on a same origin as radicale and it worked like a charm. it uses "easy connect" to discover collections.

[rngtng]

@chripo thanks. I tried to connect as test2 but now I even can't access your server as test anymore. Auth works, but Addressbook is empty and I can't create any entries... :(

[chripo]

@rngtng you can't create entities because of there is no addressbook, i removed all collections. now i have created a few. feel free to edit the resources. curl is your friend.

~  » curl -X GET https://pim.christoph-polcin.com/test/my-addressbook-3.vcf/ -u 'test:test' 
~  » curl -X DELETE https://pim.christoph-polcin.com/test/my-addressbook-3.vcf/ -u 'test:test'

[chripo]

i will remove my last commit (locating services with help of .well-known URIs (rfc6764)) because the implementation is not correct and could be done by the frontend server.

[rngtng]

@chripo I did some more testing on my server and I guess its a problem with authorization:

1. with current master, connecting with https://myhost.com/<user>/addressbook.vcf/ works
2. with your branch, connecting with https://myhost.com/ fails due to auth issues. See logs - somehow the client did send any auth headers, so user is Anonymous.

Anythings wrong with my setup & config ? The client talks diretly to the radicale server. Do you have any other server in between? how come you can use .htaccess?

[rngtng]

@chripo Good News: it works! I started once again from a clean setup (client, server + DB) and suddenly it works perfectly. I'm confused. Anyway 👍 for latest Mac clients

[chripo]

the anonymous user requests are related to the HTTP specs, or security best practice. the client will offer credentials with each second request if the anonymous request fails with a 401 and WWW-Authenticate header. (configurable by eg. HttpClientParams.setAuthenticationPreemptive)

i think your configuration / setup is messed up some how. the configfile seems to be ok, maybe start over with a fresh one and try the file backend first, it's a much simpler setup and time-proven ;)

you running a different version, the log files dosen't match this branch log output!!! principal-URL and current-user-principal are equal in most cases!!
i'm running apache as frontend server / proxy RewriteRule (.*) http://localhost:5323/$1 [P] thats all.

there is one case that could be cause an (this) issues. i'll do a test tomorrow.

try some simple curl requests.

~  ? curl https://foo.christoph-polcin.com/ -X PROPFIND -H 'Depth: 0' -i     
HTTP/1.1 401 Unauthorized
Date: Tue, 14 Jan 2014 20:07:44 GMT
Server: WSGIServer/0.1 Python/2.7.3
WWW-Authenticate: Basic realm="Radicale - Password Required"
Content-Length: 0
Cache-Control: max-age=0
Expires: Tue, 14 Jan 2014 20:07:44 GMT
Vary: Accept-Encoding
Connection: close
Content-Type: httpd/unix-directory

~  ? curl https://foo.christoph-polcin.com/test/ -X PROPFIND -H 'Depth: 0' -i     
HTTP/1.1 401 Unauthorized
Date: Tue, 14 Jan 2014 20:07:50 GMT
Server: WSGIServer/0.1 Python/2.7.3
WWW-Authenticate: Basic realm="Radicale - Password Required"
Content-Length: 0
Cache-Control: max-age=0
Expires: Tue, 14 Jan 2014 20:07:50 GMT
Vary: Accept-Encoding
Connection: close
Content-Type: application/octet-stream

~  ? curl https://foo.christoph-polcin.com/test/test/ -X PROPFIND -H 'Depth: 0' -i
HTTP/1.1 401 Unauthorized
Date: Tue, 14 Jan 2014 20:07:55 GMT
Server: WSGIServer/0.1 Python/2.7.3
WWW-Authenticate: Basic realm="Radicale - Password Required"
Content-Length: 0
Cache-Control: max-age=0
Expires: Tue, 14 Jan 2014 20:07:55 GMT
Vary: Accept-Encoding
Connection: close
Content-Type: application/octet-stream

~  ? curl https://foo.christoph-polcin.com/test/test/ -X PROPFIND -H 'Depth: 0' -i -u 'test:test'
HTTP/1.1 207 Unknown
Date: Tue, 14 Jan 2014 20:08:05 GMT
Server: WSGIServer/0.1 Python/2.7.3
DAV: 1, 2, 3, calendar-access, addressbook, extended-mkcol
Content-Type: text/xml
Content-Length: 845
Cache-Control: max-age=0
Expires: Tue, 14 Jan 2014 20:08:05 GMT
Vary: Accept-Encoding
Connection: close

<?xml version="1.0"?>
<multistatus xmlns="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:CS="http://calendarserver.org/ns/" xmlns:ICAL="http://apple.com/ns/ical/">
  <response>
    <href>/test/test/</href>
    <propstat>
      <prop>
        <getcontenttype>text/calendar</getcontenttype>
        <resourcetype>
          <C:calendar />
          <collection />
        </resourcetype>
        <displayname>test</displayname>
        <owner>/test/</owner>
        <getetag>"d41d8cd98f00b204e9800998ecf8427e"</getetag>
        <current-user-principal>
          <href>/test/</href>
        </current-user-principal>
        <ICAL:calendar-color>#781d32</ICAL:calendar-color>
        <CS:getctag>"d41d8cd98f00b204e9800998ecf8427e"</CS:getctag>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
</multistatus>

~  ? curl https://foo.christoph-polcin.com/ -X PROPFIND -H 'Depth: 0' -i -u 'test:test' 
HTTP/1.1 207 Unknown
Date: Tue, 14 Jan 2014 20:09:39 GMT
Server: WSGIServer/0.1 Python/2.7.3
DAV: 1, 2, 3, calendar-access, addressbook, extended-mkcol
Content-Type: text/xml
Content-Length: 657
Cache-Control: max-age=0
Expires: Tue, 14 Jan 2014 20:09:39 GMT
Vary: Accept-Encoding
Connection: close

<?xml version="1.0"?>
<multistatus xmlns="DAV:" xmlns:CS="http://calendarserver.org/ns/" xmlns:ICAL="http://apple.com/ns/ical/">
  <response>
    <href>/</href>
    <propstat>
      <prop>
        <resourcetype />
        <current-user-principal>
          <href>/test/</href>
        </current-user-principal>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
    <propstat>
      <prop>
        <getcontenttype />
        <displayname />
        <owner />
        <getetag />
        <ICAL:calendar-color />
        <CS:getctag />
      </prop>
      <status>HTTP/1.1 404 Not Found</status>
    </propstat>
  </response>
</multistatus>

~  ? curl https://foo.christoph-polcin.com/ -X PROPFIND --data '<?xml version="1.0" encoding="UTF-8"?><A:propfind xmlns:A="DAV:"><A:prop><A:current-user-principal/><A:principal-URL/></A:prop></A:propfind>' -H 'Depth: 0' -i -u 'test:test'
HTTP/1.1 207 Unknown
Date: Tue, 14 Jan 2014 20:15:08 GMT
Server: WSGIServer/0.1 Python/2.7.3
DAV: 1, 2, 3, calendar-access, addressbook, extended-mkcol
Content-Type: text/xml
Content-Length: 383
Cache-Control: max-age=0
Expires: Tue, 14 Jan 2014 20:15:08 GMT
Vary: Accept-Encoding
Connection: close

<?xml version="1.0"?>
<multistatus xmlns="DAV:">
  <response>
    <href>/</href>
    <propstat>
      <prop>
        <current-user-principal>
          <href>/test/</href>
        </current-user-principal>
        <principal-URL>
          <href>/test/</href>
        </principal-URL>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
</multistatus>

[chripo]

fix some path issues for the last commit (Fix calendar/addressbook-home-set in PROPFIND response) if the base_prefix is set to something else then '/'

we did some test with davdroid too. collection discovery succeeded.

[chripo]

fixed a path issue. hopefully the last one ;)

[rngtng]

Cool thanks. 👍

[hhenkel]

I can confirm this version is working for me - my setup is as follows:

• radicale master from github + patches submitted to pr Fix access #100
• Nginx as a proxy
• base_prefix = /radicale/
• users from ldap authenticting with mail address

I tested this on an android device with davdroid and I'm now able to sync calendars and addressbooks.

[liZe]

The code is clean, everybody is happy, time to merge!

[liZe]

As this feature has broken support of Apple clients (see #111 and maybe #110), I've decided to revert the commits of this pull request (see #189) to release a 0.9 version soon. Please open a new pull request with this code and the fixes for Apple clients, I'll merge it for 1.0.

That's sad, because this feature is really useful, but I can't fix the bugs with Apple clients. I'm really sorry, but we can't be stuck at 0.9b1 waiting for a solution. I hope that someone will be interested enough to test these clients and find a solution.