Add JMAP support (as an alternative to IMAP + SMTP)

[marclaporte]

"JMAP is intended to be a new standard for email clients to connect to mail stores. It therefore intends to primarily replace IMAP + SMTP submission. It is also designed to be more generic such that it can be extended with contacts, calendars in the future (replacing CardDAV/CalDAV). It does not replace MTA-to-MTA SMTP transmission."

http://jmap.io/

[jasonmunro]

Thanks for the feedback. I have heard of jmap before, and briefly reviewed it a while back, but I'm not super familiar with the specification. Could be a cool addition to Cypht as either a stand alone module set or as part of the imap modules.

[marclaporte]

Super!

Someone else asked me "Who uses or supports it?" My answer copied below:

Cyrus IMAP and Apache James are the notable ones here: http://jmap.io/software.html

OpenPaaS is an open source Entreprise Social Platform which is led by LINAGORA (which has more than 150 employees)
https://linagora.com/
http://www.open-paas.org

OpenPaaS is using Apache James. And you can see JMAP used in the code:
https://github.com/linagora/openpaas-esn/search?utf8=%E2%9C%93&q=jmap&type=

In this video: https://www.youtube.com/watch?v=2QZIi3EO6rQ , Benoit Tellier explains why they decided to invest in James and JMAP (among other things). LINAGORA has 5 full time employees which are devoted to the project.

Why LINAGORA picked JMAP:
https://medium.com/linagora-engineering/you-cant-do-imap-in-the-browser-can-you-208838b0ba17

FastMail started this initiative:
"JMAP is FastMail's protocol with the warts removed. We leverage existing standards like HTTP, JSON, and native push channels on platforms which have them - making it easy for developers to work with."
https://blog.fastmail.com/2016/12/22/cyrus-development-and-release-plans/
https://blog.fastmail.com/2016/12/12/why-we-contribute/
https://blog.fastmail.com/2016/12/09/jmap-test-suite/
https://blog.fastmail.com/2015/12/23/the-jmap-momentum-builds/
https://blog.fastmail.com/2015/08/11/an-open-source-jmap-proxy-javascript-library-and-webmail-demo/
https://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-email/

Roundcube Next will support JMAP as well.
https://github.com/search?q=org%3Aroundcube-next+jmap&type=Code

Thanks!

[jasonmunro]

Thanks for that info. I took another peek at the JMAP spec after you submitted this. Looks like a pretty straight forward rest-style api. Cypht was designed to aggregate data from different sources, so a lot of the work of uncoupling from a particular api or protocol is baked in already. Having worked with the IMAP protocol for 15+ years, I understand more than most why something like JMAP is needed :) I think it differs enough from IMAP that it would make sense to create a stand alone module set (also the IMAP module set is already a tremendous amount of code).

I can't promise I will get to this real soon, but I am intrigued. I suspect initial support for this would entail a similar amount of work that the Github module set took - I also suspect we could abstract various bits of the existing IMAP code where the two protocols might overlap.

[marclaporte]

Dear Jason,

Now is a good time to start the implementation:
https://fastmail.blog/2018/12/27/jmap-is-on-the-home-straight/

Thanks!!!

[dumblob]

@marclaporte how I see it, the most important information is, that during the upcoming weeks/months, JMAP will finally become an IETF standard. Which is a huge step from my point of view, but seems to be also a huge step forward form the JMAP developers point of view (the IETF processes increased the JMAP quality significantly).

[jasonmunro]

@dumblob @marclaporte I agree and that was an interesting read, thanks for the link. It's a new year and I'm feeling ambitious so I'm going to bump this up and see what I can get done. The other new feature in the works is cardDav, which has read only support now, and I want to expand that to read/write then move on to calDav support. No reason we can't push both that and JMAP forward at the same time :)

One prerequisite for this is I want to create a "skeleton" module set for new data sources. Sort of a template of all (or at least most) of the entry points required to create a new data source inside Cypht. Then I can use that as the foundation for supporting JMAP, as well as making it easier to build support for any other protocol we want to implement.

[ulfgebhardt]

A slim PHP Library is missing at current time.

There is one for Zend:
https://github.com/WikiSuite/zend-jmap

There is many for JS (which is not compatible with current architecture of cypht?!)
https://github.com/search?l=JavaScript&q=jmap&type=Repositories

Protocol Meta for Client:
https://github.com/jmapio/jmap/blob/master/client-guide/jmap-client-guide.mdown

I think it is possible to extract the Implementation out of the Zend Framework from here:
https://github.com/WikiSuite/zend-jmap/blob/master/src/Jmap.php
(and parent folder respectively)

But the License is quite limiting:
https://github.com/WikiSuite/zend-jmap/blob/master/LICENSE.md

[dumblob]

For the record, there is also a JMAP<->IMAP proxy which could be useful for testing of some corner cases.

[jasonmunro]

@ulfgebhardt and @dumblob thanks for the resources! I suspect we will be writing our own JMAP library in PHP for this. Once we have the "skeleton" module set in place for data sources, most of the boilerplate required to set this up in Cypht is in place. Since JMAP is very RESTful, we can leverage our existing API wrapper code that makes doing RESTy things easy. The final bit is the actual work of deciphering the spec and building out the parsing and logical flows which I suspect won't be nearly as hard as IMAP. We also have a nice set of functions in the core module set for common operations so we can just re-use things we use now (like decoding mime encoded header values for example).

[jasonmunro]

As I dig into this I think I might put JMAP support into the IMAP and SMTP module sets. It will likely be a LOT easier with the only downside being you need those 2 sets enabled to use JMAP.

After much head scratching and trial and error, I have a git checkout of Cyrus IMAP running locally with JMAP support - which should be the most up to date WRT JMAP specs as we can get :)

[jasonmunro]

Update: spent some time on this over the weekend. Some random thoughts:

• It's a pretty cool API. Docs have been spot on so far and the design is consistent
• It's not as "folder" based as IMAP. Messages are still grouped by folders and folders have hierarchy of course, but message uuids are globally unique so there is no need to select a folder before accessing a message
• Performance is great, though it IS Cyrus IMAP running locally so I would expect that :)
• I started hacking around by swapping out the public interface to our IMAP lib with a matching JMAP version. This has been ... interesting. It's a hell of a lot less duplicate code than creating a stand alone module set, and maybe it could be sort of a "drop in" replacement. But It's quickly becoming obvious the protocol differences are going to require some better abstraction at the module layer that interacts with the library for that to work :)
• Pagination support for message lists!
• Our API wrapper for http requests using curl is working out nicely for this. No changes required so far.
• Managed to cobble together folder navigation. Barely tested but it's functional.

After getting folder navigation going I tried to add old school mailbox browsing, but can't seem to get quite what I need yet. Surely I'm doing something wrong but I filed an issue anyway because whatever is not working I just don't see it yet: cyrusimap/cyrus-imapd#2627

[jasonmunro]

As expected the issue I was having with Cyrus JMAP was not a bug, but a configuration issue on my end. With there help we are back in business :)

[jasonmunro]

Update: Now able to browse mailboxes and have a basic message view working. Here is the roadmap to MVP:

• Finish the hm-jmap.php lib to include all the methods the IMAP lib uses
• Add a new section on the servers page to add a JMAP server
• incorporate JMAP into the SMTP module set for outbound mail
• include JMAP as a possible IMAP or SMTP server in profiles

Once we get that far, I want to circle back around to the library code and see what can be done to improve the performance. Right now we are using JMAP in a very IMAP like way, but we can streamline that in a number of places to take advantage of JAMP features like multiple "commands" in a single API call.

[jasonmunro]

Update: Ran into another problem with Cyrus JMAP that maybe is not me doing something wrong :) filed an issue (cyrusimap/cyrus-imapd#2631).

In the meantime, Cypht JMAP support now works with combined views (unread, flagged, everything). I also reworked the folder logic in the library and was able to remove some module level work-arounds that were icky. Then I started on search integration but was thwarted by the above issue.

JMAP is complicated (E-mail is complicated!), but it truly is IMAP simplified and modernized. There is lots more to do but we are making great progress a lot faster than I expected. And I'm sort of obsessed with making it feature complete now that I started working on it :)

[jasonmunro]

Update: Wrong again. My latest issue was me doing something wrong (specifically not configuring the Cyrus search backend properly). Thanks to the helpful feedback from Cyrus devs it is resolved and search works now with JMAP, as does some message operations (flag/read/unread/etc). For core IMAP stuff needing to be converted, I have move, copy, delete, and folder management up next, following by downloading and then I think we are pretty close to IMAP feature complete!

[jasonmunro]

Compatibility layer for JMAP is feature complete! Needs more testing and love but it's working pretty well so far. Here is the code:

https://github.com/jasonmunro/cypht/blob/master/modules/imap/hm-jmap.php

Comparing this to the IMAP lib:
IMAP uses 3036 lines of code excluding comments
JMAP uses 861 lines of code excluding comments

WOW! Up next is wiring the UI to add/delete JMAP servers from the Settings -> Servers page.

[jasonmunro]

UI update to add/delete JAMP servers is pushed to the master branch :) Next I'm going to scope out using JMAP for SMTP. This should be way easier as we only communicate with SMTP in 2 places of the code if I recall.

[dumblob]

Is anyone of you in contact with JMAP developers or just people around JMAP?

It seems Cypht will be one of the very few first clients supporting JMAP - that could be used to raise awareness of Cypht and its security and mobile advantages. We could start by letting Cypht being linked from JMAP websites. Then maybe write a blog post on JMAP sites, etc.

[jasonmunro]

@dumblob I have been in contact with the JMAP folks, and we are now listed here: https://jmap.io/software.html. Also I wrote a blog post on JMAP that made it to the front page of hacker news last night and it's driving a TON of traffic :)

https://unencumberedbyfacts.com/2019/01/24/jmap-its-like-imap-but-not-really/

[marclaporte]

Bron Gondwana presented JMAP at FOSDEM 2019.

Slides with Speakers' Notes:
https://fosdem.org/2019/schedule/event/email_standards/attachments/paper/3116/export/events/attachments/email_standards/paper/3116/jmap_slides_fosdem_2019_notes.pdf

General page: https://fosdem.org/2019/schedule/event/email_standards/

Well done Bron!

And well done Jason for such a quick implementation!

[OJFord]

For the benefit of others linked to this issue from the JMAP site, my understanding is that this issue remains open only because the JMAP feature isn't yet in a (non-RC) release?

I.e. it is in master, and the (at time of writing) latest 'release', v1.2.0-rc1.

I look forward to trying it, thanks!

[marclaporte]

@OJFord : yes, that is correct. And as this is a new feature, on a new protocol, expect it to be bumpy, and please file new issues for bugs and missing features.

I encourage you to get Cypht master and start testing it out. I feel master is stable enough to be released.

In Tiki, we get Cypht master from Packagist and it works quite well:
#311

If you have trouble installing Cypht master, you can install a daily build of Tiki
20.x is stable with Cypht (I use this daily with 3 mail accounts)
21.x is not released yet and has CalDAV / CardDAV as well (also expect this to be bumpy, and UI is still uncoordinated)
https://dev.tiki.org/Daily-Build

[marclaporte]

Cyrus IMAP 3.2.0-rc1 released (with JMAP support)
https://lists.andrew.cmu.edu/pipermail/info-cyrus/2020-April/041191.html

https://www.cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.0-rc1.html#major-changes-since-the-3-0-x-series

• Support for JMAP core protocol (RFC 8620)
• Support for JMAP Mail (RFC 8621)
• Experimental support for JMAP Contacts (requires jmap_nonstandard_extensions: yes in imapd.conf(5))
• Experimental support for JMAP Calendars (requires jmap_nonstandard_extensions: yes in imapd.conf(5))

[marclaporte]

Cyrus IMAP 3.2.1 released (with JMAP support)
https://lists.andrew.cmu.edu/pipermail/info-cyrus/2020-May/041269.html
https://www.cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.1.html

[marclaporte]

JMAP online meetings/hackathons occur from time to time. We could join one at one point to review our implementation:
https://mailarchive.ietf.org/arch/browse/jmap/

[marclaporte]

After over 1100 commits, we can see that the JMAP mail spec has stabilized:

• https://github.com/jmapio/jmap
• https://datatracker.ietf.org/doc/rfc8620/

The JMAP community is now focusing on other aspects of JMAP, like contacts, calendars, etc.

• https://datatracker.ietf.org/doc/draft-ietf-jmap-calendars/
• https://datatracker.ietf.org/doc/draft-ietf-jmap-jscontact/
• https://tools.ietf.org/html/draft-ietf-jmap-sieve-00
• and others at https://tools.ietf.org/wg/jmap/

So it's a good time to get back to JMAP in Cypht. AFAICT, the most complete server implementation is with Cyrus. There are over 1800 commits for the term "JMAP":
https://github.com/cyrusimap/cyrus-imapd/search?o=desc&q=jmap&s=committer-date&type=commits

Maybe we can use https://github.com/cyrusimap/cyrus-docker/ which @ajaysusarla has recently worked on?

Thanks!

[marclaporte]

I was recently made aware of https://github.com/barnslig/jmap-proxy

[mildred]

Hi, Thank you for the great work. I'm in the process of integrating cypht in a email stack. I just tested JMAP by manually configuring it and it works great.

However, there is no docker configuration options for JMAP (as there is for IMAP) and nothing configurable in cypht/hm3.ini either.

How can JMAP be globally configured?

Thanks.

[kroky]

@jasonmunro

RE:

One prerequisite for this is I want to create a "skeleton" module set for new data sources. Sort of a template of all (or at least most) of the entry points required to create a new data source inside Cypht. Then I can use that as the foundation for supporting JMAP, as well as making it easier to build support for any other protocol we want to implement.

AND

As I dig into this I think I might put JMAP support into the IMAP and SMTP module sets. It will likely be a LOT easier with the only downside being you need those 2 sets enabled to use JMAP.

I see that you postponed the idea of a "skeleton" module set but now we need it to support https://dev.tiki.org/Email-as-a-first-class-citizen - we want a Tiki-based data storage for emails and still use the excellent Cypht UI to handle the email-related actions like displaying, replying, etc. I started adding as part of the imap module but that quite heavily depends on Hm_IMAP and Hm_JMAP. I think I have a couple of options integrating here:

1. Add Hm_Tiki and plug the adapter into the existing code (easier to do but not extensible down the road).
2. Proceed with implementing the skeleton module but that will be a lot of rewrite of the current imap module. I imagine we will decouple UI/presentation from the IMAP source - i.e. all handlers parsing IMAP messages as well as depending on IMAP struct will be abstracted, so the UI layer depends on a clearly defined interface with data access methods. Then, we can add an adapter for IMAP, JMAP and our own as we need. Are you willing to accept such a big PR or have you any plans on doing that skeleton module soon?

[henrique-borba]

Far as I know my friends, JMAP is just a mirror of the HM-IMAP module. That said, supporting it will not he a hard work.

[marclaporte]

@josaphatim will soon test Cypht's JMAP against https://stalw.art/jmap/ via https://jmap.cloud/

[landryb]

@josaphatim will soon test Cypht's JMAP against https://stalw.art/jmap/ via https://jmap.cloud/

fwiw, i've tested current HEAD against a selfhosted stalwart-jmap instance, and it doesnt look good (but i might have done something wrong in my setup):

• it seems cypht hardcodes sending all queries to .well-known/jmap instead of using the value of apiUrl given by the capabilities document (which seems to be /jmap/ in stalwart-jmap's case, at least that's what i understand from comparing a working JMAP session done by meli)
• queries are set as GET instead of POST ? thus the POST params seems lost, and i get this in the php fpm stderr:

"PHP message: PHP Warning:  Undefined array key "totalEmails" in /cypht/modules/imap/hm-jmap.php on line 126PHP message: PHP Warning:  Undefined array key "unreadEmails" in /cypht/modules/imap/hm-jmap.php on line 127"

because all calls are GET to .well-known/jmap are without args, of course the capabilities document don't have those values in the returned json.

here's the complete json reply for .well-known/jmap:

{"capabilities":{
"urn:ietf:params:jmap:core":{"maxSizeUpload":50000000,"maxConcurrentUpload":4,"maxSizeRequest":10000000,
    "maxConcurrentRequests":4,"maxCallsInRequest":16,"maxObjectsInGet":500,"maxObjectsInSet":500,"collationAlgorithms":["i;ascii-numeric","i;ascii-casemap","i;unicode-casemap"]},
"urn:ietf:params:jmap:mail":{"maxMailboxesPerEmail":null,"maxMailboxDepth":10,"maxSizeMailboxName":255,
"maxSizeAttachmentsPerEmail":50000000,
"emailQuerySortOptions":["receivedAt","size","from","to","subject","sentAt","hasKeyword","allInThreadHaveKeyword","someInThreadHaveKeyword"],
"mayCreateTopLevelMailbox":true},
"urn:ietf:params:jmap:websocket":{"url":"ws://jmap.test:8081/jmap/ws","supportsPush":true}},
"accounts":{"d":{"name":"accountname","isPersonal":true,"isReadOnly":false,"accountCapabilities":{"urn:ietf:params:jmap:core":{"maxSizeUpload":50000000,"maxConcurrentUpload":4,"maxSizeRequest":10000000,
"maxConcurrentRequests":4,
"maxCallsInRequest":16,"maxObjectsInGet":500,"maxObjectsInSet":500,"collationAlgorithms":["i;ascii-numeric","i;ascii-casemap","i;unicode-casemap"]},
"urn:ietf:params:jmap:mail":{"maxMailboxesPerEmail":null,"maxMailboxDepth":10,"maxSizeMailboxName":255,"maxSizeAttachmentsPerEmail":50000000,
"emailQuerySortOptions":["receivedAt","size","from","to","subject","sentAt","hasKeyword","allInThreadHaveKeyword","someInThreadHaveKeyword"],
"mayCreateTopLevelMailbox":true},
"urn:ietf:params:jmap:websocket":{"url":"ws://jmap.test:8081/jmap/ws","supportsPush":true}}}},
"primaryAccounts":
{"urn:ietf:params:jmap:core":"d","urn:ietf:params:jmap:mail":"d","urn:ietf:params:jmap:websocket":"d"},
"username":"user@jmap.test",
"apiUrl":"http://jmap.test:8081/jmap/",
"downloadUrl":"http://jmap.test:8081/jmap/download/{accountId}/{blobId}/{name}?accept={type}",
"uploadUrl":"http://jmap.test:8081/jmap/upload/{accountId}/",
"eventSourceUrl":"http://jmap.test:8081/jmap/eventsource/?types={types}&closeafter={closeafter}&ping={ping}","state":"753b9c77"}

i'll poke at various things to understand better how it's supposed to work

[landryb]

well reading through hm-jmap.php, it seems apiUrl is correctly found and a var is initialized with its value, so ... will dig more. Maybe just php misconfiguration from my side with php-fpm & nginx...

edit: seems api_url assignation is wrong, the line in https://github.com/jasonmunro/cypht/blob/master/modules/imap/hm-jmap.php#L859 seems to append something weird.

• $url is http://fqdn/
• $data[apiUrl] is http://fqdn/jmap/
• $this->api_url end up with http://fqdnhttp://fqdn/jmap/ which doesnt look right at all

hardcoding $this->api_url to be http://fqdn/jmap/ i see the queries sent as POST but i'm getting a 400 code:

    [type] => urn:ietf:params:jmap:error:notRequest
    [status] => 400
    [detail] => The Request object is not a valid JMAP request

[landryb]

one of the things i found out from the above json & in testing (but maybe i should file a distinct issues for stalwart-jmap compatibility issues ?):
there's no isPrimary key in accounts hash

  "accounts": {
    "d": {
      "name": "accountname",
      "isPersonal": true,
      "isReadOnly": false,

, but a toplevel primaryAccounts hash with capabilities which each list account names.

  "primaryAccounts": {
    "urn:ietf:params:jmap:core": "d",
    "urn:ietf:params:jmap:mail": "d",
    "urn:ietf:params:jmap:websocket": "d"
  },

Thus account_id ends up null.. hacking it to not check if accounts are primary, the server still replies with a 400 when listing folders (from get_folder_list)

here's the post sent as logged by nginx via $request_body:

"POST /jmap/ HTTP/1.1" 400 120 {\x22using\x22:[\x22urn:ietf:params:jmap:core\x22,\x22urn:ietf:params:jmap:mail\x22],\x22methodCalls\x22:[[\x22Mailbox\x5C/get\x22,{\x22accountId\x22:\x22d\x22,\x22ids\x22:null},\x22fl\x22]]}

[landryb]

playing with curl, it seems what is sent is escaped too much ? "Method/get" works, but cypht sends "Method\/get" which is what triggers a 400 from the server.

curl -v -o /dev/null -XPOST -d'{"using":["urn:ietf:params:jmap:core","urn:ietf:params:jmap:mail"],"methodCalls":[["Mailbox/get",{"accountId":"d","ids":null},"fl"]]}' -u user@jmap.test:password http://127.0.0.1:8080/jmap/ 2>&1 |grep HTTP

> POST /jmap/ HTTP/1.1
< HTTP/1.1 200 OK

curl -v -o /dev/null -XPOST -d'{"using":["urn:ietf:params:jmap:core","urn:ietf:params:jmap:mail"],"methodCalls":[["Mailbox\/get",{"accountId":"d","ids":null},"fl"]]}' -u user@jmap.test:password http://127.0.0.1:8080/jmap/ 2>&1 |grep HTTP

> POST /jmap/ HTTP/1.1
< HTTP/1.1 400 Bad Request

[landryb]

Fixed the above by adding JSON_UNESCAPED_SLASHES flag to the json_encode call in format_request... now the requests sent to the jmap server all return 200 codes, but cypht returns a 500 to the user. making progress slowly :)