End-to-end encrypted storage type

[untitaker]

Flock was shut down (see https://gist.github.com/rhodey/873ae9d527d8d2a38213), but I'd like to see a successor.

See also https://github.com/WhisperSystems/Flock/issues/93

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[untitaker]

In the meantime I took a look at the remotestorage protocol, and can say that it's a perfect fit (as already said by rhodey). Apart from the crypto, it does everything right, both in simplicity and security/usability. It's a perfect drop-in replacement for the WebDAV that Flock uses. There has been some discussion about adding an transparent encryption layer to the remoteStorage JS library.

I have a local branch that implements a (unencrypted) remoteStorage storage type, with OAuth support and everything. It's not ready yet:

• #233 should be resolved in a manner that makes it possible to save auth tokens in the keychain
• It doesn't do any encryption at all

What I'd like to see is (in that order):

• See how the RS-people's discussion about end-to-end encryption ends. I'd like to implement a form of encryption that is compatible with whatever they come up with.
• remoteStorage javascript apps that request the vdir:rw scope like vdirsyncer does. In other words, I'd like to see some interopability between vdirsyncer and other apps, but over remoteStorage instead of CardDAV/CalDAV.

[untitaker]

In the meantime I took a look at the remotestorage protocol, and can say that it's a perfect fit (as already said by rhodey). Apart from the crypto, it does everything right, both in simplicity and security/usability. It's a perfect drop-in replacement for the WebDAV that Flock uses. There has been some discussion about adding an transparent encryption layer to the remoteStorage JS library.

I have a local branch that implements a (unencrypted) remoteStorage storage type, with OAuth support and everything. It's not ready yet:

• #233 should be resolved in a manner that makes it possible to save auth tokens in the keychain
• It doesn't do any encryption at all

What I'd like to see is (in that order):

• See how the RS-people's discussion about end-to-end encryption ends. I'd like to implement a form of encryption that is compatible with whatever they come up with.
• remoteStorage javascript apps that request the vdir:rw scope like vdirsyncer does. In other words, I'd like to see some interopability between vdirsyncer and other apps, but over remoteStorage instead of CardDAV/CalDAV.

[untitaker]

By the way, it would only make sense for this to get implemented if there were mobile apps supporting the same protocol. Otherwise you might as well just use BitTorrent Sync or Syncthing.

[untitaker]

By the way, it would only make sense for this to get implemented if there were mobile apps supporting the same protocol. Otherwise you might as well just use BitTorrent Sync or Syncthing.

[untitaker]

EteSync looks like a very promising successor to Flock. It is a fork of DavDroid, but communicating over a non-standard protocol with a custom (OSS) server.

@tasn, two questions:

• Does this fork of DavDroid still include the OpenTasks provider, and does it work?
• I'm interested in adding this to vdirsyncer, is the protocol design stable enough for me to (hypothetically) reverse and re-implement it?

[untitaker]

EteSync looks like a very promising successor to Flock. It is a fork of DavDroid, but communicating over a non-standard protocol with a custom (OSS) server.

@tasn, two questions:

• Does this fork of DavDroid still include the OpenTasks provider, and does it work?
• I'm interested in adding this to vdirsyncer, is the protocol design stable enough for me to (hypothetically) reverse and re-implement it?

[tasn]

I'm very happy to discover this project, it looks like a perfect match for desktop support.

Does this fork of DavDroid still include the OpenTasks provider, and does it work?

I removed it when I changed the protocol, just so I can develop faster, but I plan on adding it back. I can look into it soon.

I'm interested in adding this to vdirsyncer, is the protocol design stable enough for me to (hypothetically) reverse and re-implement it?

There is one change I would like to make to the protocol before I can consider it stable, though this changed has been planned for a while, and apart from it I think the protocol is pretty future proof, and even this change can (and will) be done in a backwards compatible way.

Essentially, at the moment I derive the encryption password from the user password and use it for all of the journals. This is absolutely fine in a single-user environment, but I would like to be able to let users share journals with other users (think shared calendars), so generating a secure password per-journal is needed. I plan on changing it very soon, so it'll probably already be fixed by the time you get to this part.

As for reimplementing it: I'd be more than happy to help creating a reusable python library that interacts with EteSync. It should be fairly simple. I made sure to keep the protocol as simple as possible. The java code that deals with the journal is tiny, and vCard and iCal libraries are readily available for many languages. Please let me know if you need anything else from me.

[tasn]

I'm very happy to discover this project, it looks like a perfect match for desktop support.

Does this fork of DavDroid still include the OpenTasks provider, and does it work?

I removed it when I changed the protocol, just so I can develop faster, but I plan on adding it back. I can look into it soon.

I'm interested in adding this to vdirsyncer, is the protocol design stable enough for me to (hypothetically) reverse and re-implement it?

There is one change I would like to make to the protocol before I can consider it stable, though this changed has been planned for a while, and apart from it I think the protocol is pretty future proof, and even this change can (and will) be done in a backwards compatible way.

Essentially, at the moment I derive the encryption password from the user password and use it for all of the journals. This is absolutely fine in a single-user environment, but I would like to be able to let users share journals with other users (think shared calendars), so generating a secure password per-journal is needed. I plan on changing it very soon, so it'll probably already be fixed by the time you get to this part.

As for reimplementing it: I'd be more than happy to help creating a reusable python library that interacts with EteSync. It should be fairly simple. I made sure to keep the protocol as simple as possible. The java code that deals with the journal is tiny, and vCard and iCal libraries are readily available for many languages. Please let me know if you need anything else from me.

[untitaker]

I don't parse vCard and iCal very extensively because CalDAV and CardDAV can be treated like glorified BLOB storages. Is there anything in your protocol that would prevent me from downloading and uploading entire VEVENTs or VCARDs?

EDIT: Yes, it is end-to-end encrypted but it could be that you encrypt per-property or something like that.

[untitaker]

I don't parse vCard and iCal very extensively because CalDAV and CardDAV can be treated like glorified BLOB storages. Is there anything in your protocol that would prevent me from downloading and uploading entire VEVENTs or VCARDs?

EDIT: Yes, it is end-to-end encrypted but it could be that you encrypt per-property or something like that.

[tasn]

What do you mean by entire VEVENT or VCARD? You mean not dealing with the content? If so, then yes.
EteSync is essentially a change journal, with each entry having a type of action and content.
The types currently supported are: add, delete and change, and the content is the current snapshop of the vevent/vcard. So you don't need to understand vcard, you can just treat it as a blob. All you need to do is:
if action == add/change: save blob
else if action == delete: delete blob

That's it.

[tasn]

What do you mean by entire VEVENT or VCARD? You mean not dealing with the content? If so, then yes.
EteSync is essentially a change journal, with each entry having a type of action and content.
The types currently supported are: add, delete and change, and the content is the current snapshop of the vevent/vcard. So you don't need to understand vcard, you can just treat it as a blob. All you need to do is:
if action == add/change: save blob
else if action == delete: delete blob

That's it.

[untitaker]

Yeah that's what I mean. 👍

[untitaker]

Yeah that's what I mean. 👍

[tasn]

Just adding it here for reference, work seems to be well on the way (by @untitaker)
Pull request: #614

[tasn]

Just adding it here for reference, work seems to be well on the way (by @untitaker)
Pull request: #614