Follow roadmap for 2.0

[liZe]

As defined in #364.

• Stay simple:
  • Keep only what's needed:
    • Remove Python 2.x support (Disable SSLv2, SSLv3 #316).
    • Keep only one simple filesystem-based storage system (OSx Calendar and iOS Calendar database storages on PROPFIND #83, Fix the SQL backend #114, Discussion: Proposed changes to storage api for 1.0 #130, Trying to fix database issues. #134, IOError: [Errno 20] Not a directory #161, Problem with saving to MySQL storage #206, Missing documentation of database backends #207, Fixes for multifilesystem backend #215, Init Database #324, Create the missing tables if necessary #357).
    • Remove built-in Git support (Improve Git commits #286, Dulwich has a python3-supported release now #317).
    • Remove built-in authentication modules (IMAP auth fails with SSLError #323).
  • Clean the documentation:
    • Mention versions covered by the documentation (Mention version(s) covered on user documentation and/or ship with source #157).
    • Use GitHub to host the website (radicale.org uses an invalid SSL certificate #310).
    • Create a new design for the website.
  • Write the documentation:
    • Migration guide.
    • Install (Add a tutorial #112)
      • What is Radicale.
      • Tutorial.
      • Simple installation.
      • Production-ready installation (WSGI or similar deployment with Nginx #197, Radicale repeatedly hanging / locking #266, CPU activity when idle #270, It there a limitation of users, calenders, file size or parallel server access? #276, Radicale stops working after some time #388).
      • Versioning.
    • Use
      • Configuration of clients.
    • Configure
      • What can be configured?
      • Auth and rights. (Need better documentation on the rights file. #538)
      • Storage.
      • Logging.
    • Hack
      • How does Radicale work.
      • Plugins.
      • Debugging.
      • Documentation.
• Become solid:
  • Use standards:
    • Keep the WSGI interface, use Python HTTP server by default.
    • Use a real iCal parser (icalendar, vobject).
    • Don't rely on slashes (Documentation improvement: Stress importance of URI endings #322).
    • Build a solid calendar discovery (Collections discovery not working #292).
    • Respect the difference between "files" and "folders" (Creating a /user/ calendar prevents radicale from allowing further calendars #366).
    • Remove the calendar creation with GET requests (Add a configuration entry to disable collection creation on GET/PROPFIND requests #240, Do well-known URLs create collections? #248, Thunderbird / SOGO Connector creates empty calendar instead of addressbook #252).
    • Define how to create collections (Clarification: How to create a user #384).
  • Be thread-safe:
    • Be stateless (Global state in radicale #122).
    • Use a file locker (filelock) (Add locking functionality #14).
    • Add threading (Radicale repeatedly hanging / locking #266, Radicale stops working after some time #388).
    • Get atomic writes (Data loss due to non-atomic writes #440).
• Provide needed new features:
  • Built-in filters (Support filters #33, CalDAV 'REPORT' with VTODO also sends back VEVENT blocks (and viceversa) #101).
  • Read-only permissions (Handle read-only permissions in PROPFIND request #290, Fix privilege set for read only collection #363).
  • External authentication modules allowed (Implementing group restrictions #187, Crashes with auth type = remote_user and lighttpd's htdigest authentication method #227, Allow full path in custom_handler config #247, Use ldap3 for LDAP support #278, Add LDAP support for python 3.x #325, New parameter "user_strip_at" to remove @domain from user. #349).
  • External rights modules allowed (Support storage backend requiring per-user authentication #242, Problem using the "first matching rule" for the rights #374).
  • External storage modules allowed (Discussion: Proposed changes to storage api for 1.0 #130, Provide data migration documentation or scripts #392).
  • External version control tools allowed (like gitwatch) (Git hooks are not being executed #327).

[liZe]

I've removed the extra storage, auth and rights modules. Anyone is insterested in testing the new code and config file?

[untitaker]

Could you keep the refactor out of master until it is finished? Thanks!

Will test soon, but right now I have to deal with the dev version being broken in my CI. :(

[liZe]

Could you keep the refactor out of master until it is finished? Thanks!

I won't, sorry :/.

I've had really bad experiences with keeping ongoing development out of master, and good experiences after with big changes in master early:

• no pull requests on the "old" branch when they're outdated/irrelevent with the "new" one,
• lots of people understand early that "something is going on",
• more early testers and early discussions,
• no "big surprise" when the new branch is merged, with endless discussions on what should have been done after weeks/months of work.

By the way, I've added a 1.1.x branch, and I'm trying to keep master working even with big changes (Travis is watching us!)

Will test soon, but right now I have to deal with the dev version being broken in my CI. :(

Thank you, good luck!

[liZe]

I've tested the current version (af19377) with Thunderbird, DAVdroid and CalDAV Sync Adapter. Collection discovery seems to work well with and without authentication.

[liZe]

The Collection API has been changed as proposed in #130.

[untitaker]

BTW I think skarim/vobject#23 should be considered a blocker issue.

[liZe]

The file locker has been implemented in #402.

I've tested (not for a long time) the current master branch with many clients, including Cadaver, Thunderbird, iOS Calendar, OSX Calendar, and it seems to work well.

[liZe]

I've removed freebusy (#34) and sync-collection REPORT (#306) from the roadmap, as they need a lot of time. I'll fix the two last technical tickets (#440 and #327) and realease a new RC soon.

The next step is testing and documentation writing. Anyone interested?

[bandali0]

Really looking forward to 2.0!

@liZe Any pointers or recommendations for "simple installation" for personal use? I'm about to re-setup my VPS from scratch and would love to use Radicale 2.0 for my calendars and contacts.

[liZe]

Any pointers or recommendations for "simple installation" for personal use? I'm about to re-setup my VPS from scratch and would love to use Radicale 2.0 for my calendars and contacts.

Installing from scratch for personal use is really easy. You can follow what's in the future tutorial :

• install with pip install git+https://github.com/Kozea/Radicale.git:
• store your configuration in ~/.config/radicale/config,
• use nohup or something like this.

That's all!

[bandali0]

Ha I see, thanks! I'll try and see if I can run it behind nginx.

[pbiering]

Dropping Phyton 2.x support and enforcing Phyton 3.x leads to installation problems on Enterprise Linux version 7 (latest RHEL or CentOS).
Biggest issue is that I haven't found a mod_wsgi module for EL7 -> radicale "must" run in standalone mode with Apache in front using ProxyPass
Major issue is missing "python34-vobject", but this is solvable by an adjusted rebuild of SRPM of latest Fedora version.

I can confirm now proper working with "Thunderbird Inverse SoGo Connector", for migration from old multi-filesystem one should perhaps document, that calender "folder" layout changed:

Old:
/user/calendar-1/id-of-entry-1.ics
/user/calendar-1/id-of-entry-2.ics
...
/user/calendar-1.props containing "VCALENDAR"

New:
/user/calendar-1/id-of-entry-1.ics
/user/calendar-1/id-of-entry-2.ics
...
/user/calendar-1/.Radicale.props containing "VCALENDAR"

BTW: in case a manual calendar folder was created on file system but missing .Radicale.props and then using SoGo connector with folder URL leads to a very strange result:
new calendar entries are created as folders, entry contents is a file in that folder having a short hashed name, .Radicale.props is also stored in the folder, directory/file permissions on Unix are 777 and SoGo can't read back the entry (because dot-props for folder is missing):
/user/calendar-with-no-dot-props/id-of-entry-1.ics/shortid
/user/calendar-with-no-dot-props/id-of-entry-1.ics/.Radicale.props
-> somehow not expected, I would suggest rejecting creation of entries in a folder in case this is not explicitly defined in .Radicale.props as VCALENDAR or VADRESSBOOK.

Permissions of 777 are probably the result of umask(0), see #540

[liZe]

new calendar entries are created as folders

It's normal. There's no difference between an event and a calendar in iCal, so PUTting an event in a non-calendar collection is just like creating a calendar in this collection. Nothing strange here 😉.

migration from old multi-filesystem one should perhaps document that calender "folder" layout changed

Of course! We just need to write the Storage page of the new configuration.

Dropping Phyton 2.x support and enforcing Phyton 3.x leads to installation problems on Enterprise Linux version 7 (latest RHEL or CentOS).

With "serious" distributions, it's probably not a good idea to use Radicale 2 before it's been officially packaged. But if anybody wants to try, using pip to install vobject and putting Radicale behind nginx+uwsgi is a solution that should be available in any current distribution.

[pbiering]

new calendar entries are created as folders

It's normal. There's no difference between an event and a calendar in iCal, so PUTting an event in a non-calendar collection is just like creating a calendar in this collection. Nothing strange here 😉

Can one implement a feature (perhaps optionally like "strict_subfolder_usage") to prevent creating Items in the root folder which aren't collections? Because this turns to totally strange situation:

WP run calendar autodiscovery and suddenly has more calendars, the original "folder-based" ones + for each new Item created in the root-folder...means calendars in WP are increasing over time...one per new Item created by a client which (for whatever reason) places entries in the / folder

[pbiering]

Please take a look on my pull request on top of master, I would consider it as stable now, found no issues so far: #526

[liZe]

I'll release a beta version of 2.0.0 soon. The 2.0.0 milestone has some issues open, but some of them can wait after the beta is released.

The future documentation can be updated on the gh-pages branch, take a look if you're interested!

[liZe]

I have released 2.0.0rc1 on PyPI! The version is hidden but available.

There are lots of cool features waiting in some pull requests, but I find them a little bit too dangerous to merge them just now. 2.0.0 is working quite well right now, I'll only merge bugfixes before 2.0.0.

Please check that everything works for you!

[klonos]

The version is hidden but available.

Can you please share the secret 😄 ...links to binaries would be awesome 😉

[liZe]

https://pypi.python.org/pypi/Radicale/2.0.0rc2 ;)

[liZe]

Please read carefully if you're interested in Radicale's future.

Now that #591 is (almost) fixed, I'll release 2.0.0 using the git master branch on Saturday or on Sunday this week.

I know: I'm late.

Before the release, I'd like you (@untitaker, @pbiering, @Unrud, and anyone interested of course) to help me solving a couple of problems:

• I think that I can postpone the bugs tagged for 2.0.0 for a future release.
• I must release the website for 2.0.0. I think that I will put the current http://kozea.github.io/Radicale/ to http://radicale.org/. The documentation for 1.x will be available in the website branch, I'll add a link to this page on the Documentation page.
• There are lots of issues open about the documentation. Some of them can be solved later, but the missing ticks of Follow roadmap for 2.0 #372 sould be written as soon as possible. If anyone is interested, the website is on the gh-pages branch.

What do you think about that?

Another important point: THANK YOU.

It's been a pleasure to merge your pull requests and discuss all these issues and feature requests. After 2.0.0 is released, it's time for Radicale to find new co-maintainers with superpowers. Radicale has become a lot bigger and powerful than it was before. There are really interesting pull requests waiting to be merged, but I can't find the time and the motivation to read, review and merge them anymore. Moreover, I've always loved really simple code with a small amount of features, and I think that Radicale now wants to include features that I'm not ready to clean, debug and maintain alone. Of course, I'll keep an eye on the project and tell what I think, but I'd like to give admin rights to the repository and to PyPI to some current contributors. If anyone is interested, just tell me!

💓

[Unrud]

I'll release 2.0.0 using the git master branch on Saturday or on Sunday

Before you do this, please cherry-pick fc05e55
(#598 is the same fix, but checking path.startswith(base_prefix) was forgotten). The SCRIPT_NAME environment variable is currently not working at all.

I propose to merge (or reject) #604 and #608 before the release. They are very small, but change the default configuration and behavior of the --config command line argument slightly. Therefore, they cannot be merged in a minor release later.

I think that all the other open pull requests don't change anything in a manner that is not backwards compatible.

There are lots of issues open about the documentation.

I started writing the documentation for 2.0.0 in #607 and created an easier way to upgrade from 1.x.x in #606.

it's time for Radicale to find new co-maintainers with superpowers.

I would be interested. I was thinking about forking Radicale anyway.

[liZe]

Before you do this, please cherry-pick fc05e55

It's done, thanks.

I propose to merge (or reject) #604 and #608 before the release.

I've merged both, better now than later.

I think that all the other open pull requests don't change anything in a manner that is not backwards compatible.

I'm a coward, some of them are too big for me 😉. I'd prefer to merge them in a future version.

I started writing the documentation for 2.0.0 in #607 and created an easier way to upgrade from 1.x.x in #606.

I've merged #606 and I'll merge #607 just before releasing 2.0.0. I'll release a new 1.1.x version too.

I would be interested. I was thinking about forking Radicale anyway.

Good news! I think that you have the skills and motivation to bring Radicale to a higher level. No need to fork then, here is a copy of the key 🔑.

[liZe]

Radicale 2.0.0 has been released 🎉.

[djmattyg007]

Congratulations :)