Define what's in Radicale 2.0

[liZe]

Things have been explained here: http://librelist.com/browser//radicale/2015/8/21/radicale-1-0-is-coming-what-s-next/

I think that Radicale 2.0 should:

• rely on a few external libraries for simple critical points (dealing
  with
  HTTP and iCal for example),
• be thread-safe,
• be small,
• be documented in a different way (for example by splitting the client
  part
  from the server part, and by adding use cases),
• let most of the "auth" modules outside in external modules,
• have more and more tests,
• have reliable and faster filesystem and database storage mechanisms,
• get a new design [for the website] :).

I'd also secretly love to drop the Python 2.x support.

---

Edited on 2016-03-14.

I've got some time to work on Radicale during the 3 next months (except 2 weeks of holidays in April), so expect big changes!

Here are the main goals of Radicale 2.0, after reading the comments and the open tickets:

• Stay simple.
• Become solid.
• Provide needed new features.

It's easy, isn't it? Well, we must now decide what to do to reach these
goals. Here are my current choices:

• Stay simple:
  • Keep only what's needed:
    • Remove Python 2.x support (Disable SSLv2, SSLv3 #316).
    • Keep only one simple filesystem-based storage system (OSx Calendar and iOS Calendar database storages on PROPFIND #83, Fix the SQL backend #114, Discussion: Proposed changes to storage api for 1.0 #130, Trying to fix database issues. #134, IOError: [Errno 20] Not a directory #161, Problem with saving to MySQL storage #206, Missing documentation of database backends #207, Fixes for multifilesystem backend #215, Create the missing tables if necessary #357).
    • Remove built-in Git support (Improve Git commits #286, Dulwich has a python3-supported release now #317).
    • Remove built-in authentication modules (IMAP auth fails with SSLError #323).
  • Clean the documentation:
    • Split the documentation into client/server.
    • Split the server part of the documentation into quick/normal/complex (Add a tutorial #112).
    • Mention versions covered by the documentation (Mention version(s) covered on user documentation and/or ship with source #157).
    • Use GitHub to host the website (radicale.org uses an invalid SSL certificate #310).
    • Create a new design for the website!
• Become solid:
  • Use standards:
    • Keep the WSGI interface, use Python HTTP server by default.
    • Use a real iCal parser (icalendar).
    • Don't rely on slashes (Documentation improvement: Stress importance of URI endings #322).
    • Build a solid calendar discovery (Collections discovery not working #292).
    • Respect the difference between "files" and "folders" (Creating a /user/ calendar prevents radicale from allowing further calendars #366).
    • Remove the calendar creation with GET requests (Add a configuration entry to disable collection creation on GET/PROPFIND requests #240, Do well-known URLs create collections? #248, Thunderbird / SOGO Connector creates empty calendar instead of addressbook #252).
  • Be thread-safe:
    • Use a file locker (filelock) (Add locking functionality #14).
    • Recommend external HTTP tools (including Python-based and other HTTP servers supporting multiple processes and threads) (WSGI or similar deployment with Nginx #197, Radicale repeatedly hanging / locking #266, CPU activity when idle #270, It there a limitation of users, calenders, file size or parallel server access? #276).
• Provide needed new features:
  • Built-in filters (Support filters #33, Support free-busy-related requests #34, CalDAV 'REPORT' with VTODO also sends back VEVENT blocks (and viceversa) #101).
  • Sync-collection REPORT requests (Support for sync-collection REPORT #306).
  • Read-only permissions (Handle read-only permissions in PROPFIND request #290).
  • External authentication modules allowed (Implementing group restrictions #187, Crashes with auth type = remote_user and lighttpd's htdigest authentication method #227, Allow full path in custom_handler config #247, Use ldap3 for LDAP support #278, Add LDAP support for python 3.x #325, New parameter "user_strip_at" to remove @domain from user. #349).
  • External rights modules allowed (Support storage backend requiring per-user authentication #242).
  • External version control tools allowed (like gitwatch) (Git hooks are not being executed #327).

I know that the storage is a difficult question, but I like

• files more than databases, because they are easier to use and hack, even if it's a bit slower and even if the thread-safe accesses is a little bit more difficult to implement with files,
• one-file-per-event more than one-file-per-collection, because it's really easier to edit but still easy to hack.

Comments welcome!

[untitaker]

have reliable and faster filesystem and database storage mechanisms,

Personally I'd like Radicale to focus on only one storage mechanism to make this goal easier.

Otherwise I think each of those points should have its own issue to facilitate on-topic discussion.

[liZe]

Personally I'd like Radicale to focus on only one storage mechanism to make this goal easier.

Yes, I agree, I think that one file per event is the best solution. When we finally get a thread-safe solution, there will be no discussion about databases, right?

[untitaker]

Well, using a database might be easier than manually acquiring locks.

[Ede123]

I always liked Radicale using plain vCard/vCalendar files that can be modified by any text editor if necessary and merged/split/diffed/etc. however one likes.

• If Radicale required a database I'd probably picked another server to start with
• If Radicale broke up each calendar / address book into many files it would make usage at least much less comfortable.

For small (likely private) installations the current filesystem backend is certainly the best solution and I always thought that was the target audience of Radicale?

[untitaker]

Do you actually directly modify Radicale's data backend for any practical purpose?

[Ede123]

Sure, I used it a lot for debugging purposes (to emulate changes from another client)

Also it's really helpful if you happen to have to merge vCards / vCalendars from sources/devices that for whatever reason can not directly sync with Radicale.

In general I always like to be able to "repair" any potential issues by hand (just in case).
For example I already had a scenario where an application used non-standard formatting of dates. Since all files involved were plain vCards, I could simply use a text editor and some regular expressions and have it fixed in no time so I could merge it into my Radicale vCard.
If Radicale started to use any other database format that would've been a lot harder and much less straightforward.

[untitaker]

I suppose a storage model like multifilesystem has is a reasonable compromise.

For larger calendars I don't think you really want to load the entire textfile into your editor anyway. And for searching, I've often used grep -r.

[jaesivsm]

Well, personally, I'd be all in favor for improving the database support. Maybe even support new backend like mongo. That one seems especially well suited for Radicale.
But even if we stick with SQL, it would definitely make Radicale race condition proof and would fix any lock issues.
I also happen to have witnessed corrupt calendar file with closing tag not present making Radicale return incoherent entry requested (I fixed it by editing the calendar by hand), databases would definitely make that a thing of the past.
All in all, scale, multiple instance running simultaneously and stuff like wouldn't be issues anymore.

@Ede123 I'm quite sure this kind of fixes would be just as doable with databases, but yeah, you'll need to use the database client. It won't be quite as "user" friendly, but it'll still be a sys-admin job...

One last point, even if I'm in favor of supporting more databases backend, I have to admit that extending support to a new backends may create a lot of bugs and user feedback / bugs, it might not be the direction you want for the project.

[liZe]

But even if we stick with SQL, it would definitely make Radicale race condition proof and would fix any lock issues.

The difference between "being race-condition proof" and "using a database" is discussed in #14. TL;DR using a DB doesn't magically make Radicale thread-safe (even if it can help) and using files in not blocking to eradicate race conditions.

The locking issues we experience in #266 (and many other previous bugs) are caused by the HTTP server, not the storage. I don't know any locks caused by the storage in Radicale.

I also happen to have witnessed corrupt calendar file with closing tag not present making Radicale return incoherent entry requested (I fixed it by editing the calendar by hand), databases would definitely make that a thing of the past.

I disagree. The current implementation of the database storage is as stupid (even worse?) as the filesystem storage, and the "garbage in garbage out" strategy used by Radicale is not fixed by using databases. A missing closing tag would have lead to incoherent data in the database, and it would have been way more difficult to fix than editing a file.

The way to fix the problem is to (1) fix race conditions, and (2) really read and understand ics and vcf formats (believe it or not, Radicale currently doesn't really understand what an event is!).

One last point, even if I'm in favor of supporting more databases backend, I have to admit that extending support to a new backends may create a lot of bugs and user feedback / bugs, it might not be the direction you want for the project.

You're right. Supporting other storages actually created more problems than they helped users. The bug tracker is filled by tickets related to the multifilesystem and database backends.

Thanks for your comment. Even if I disagree, I hear that you prefer databases, and you're not the only one. I think that the wonderful confidence given by databases to users is related to good reasons, mainly speed and concurrency. But these reasons cannot be applied to Radicale: it's currently "slow" because of the way data are stored, not because of the file storage, and the race conditions can be fixed without requiring a database.

[untitaker]

The performance issue is not in any particular storage implementation (well filesystem maybe), but rather the API they implement. This API has two main ops as far as I understand: Load the entire collection, and write the entire collection.

EDIT: #130

Also I don't think Radicale would have to start to know what a calendar is if it doesn't have to merge/split them into/from a large .ics file.

[liZe]

The performance issue is not in any particular storage implementation (well filesystem maybe), but rather the API they implement. This API has two main ops as far as I understand: Load the entire collection, and write the entire collection.

Right! #130 + multifilesystem seem to be a good start.

Also I don't think Radicale would have to start to know what a calendar is if it doesn't have to merge/split them into/from a large .ics file.

Splitting and merging is the easy part, and Radicale is already able to do so: it can GET and PUT both collections and event with filesystem and multifilesystem. But knowing what a calendar is is necessary if you want filters, and that's awful: the server must know dates, timezones, repeats… Hopefully, some libraries are already able to handle this.

[untitaker]

Splitting and merging is the easy part, and Radicale is already able to do so

My point is that this knowledge shouldn't be in Radicale, but only in the implementation of filesystem.

AFAICT Radicale doesn't currently implement filters at all, and I don't think it's a critical feature to have.

[liZe]

AFAICT Radicale doesn't currently implement filters at all, and I don't think it's a critical feature to have.

It's not critical, but it would be really useful to:

• work out of the box with OSX and iOS clients,
• highly reduce the amount of data transferred through the network,
• help the clients to be fast.

[liZe]

I've added some proposals and links to open tickets in the ticket description.