New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add JMAP support (as an alternative to IMAP + SMTP) #180
Comments
Thanks for the feedback. I have heard of jmap before, and briefly reviewed it a while back, but I'm not super familiar with the specification. Could be a cool addition to Cypht as either a stand alone module set or as part of the imap modules. |
Super! Someone else asked me "Who uses or supports it?" My answer copied below: Cyrus IMAP and Apache James are the notable ones here: http://jmap.io/software.html OpenPaaS is an open source Entreprise Social Platform which is led by LINAGORA (which has more than 150 employees) OpenPaaS is using Apache James. And you can see JMAP used in the code: In this video: https://www.youtube.com/watch?v=2QZIi3EO6rQ , Benoit Tellier explains why they decided to invest in James and JMAP (among other things). LINAGORA has 5 full time employees which are devoted to the project. Why LINAGORA picked JMAP: FastMail started this initiative: Roundcube Next will support JMAP as well. Thanks! |
Thanks for that info. I took another peek at the JMAP spec after you submitted this. Looks like a pretty straight forward rest-style api. Cypht was designed to aggregate data from different sources, so a lot of the work of uncoupling from a particular api or protocol is baked in already. Having worked with the IMAP protocol for 15+ years, I understand more than most why something like JMAP is needed :) I think it differs enough from IMAP that it would make sense to create a stand alone module set (also the IMAP module set is already a tremendous amount of code). I can't promise I will get to this real soon, but I am intrigued. I suspect initial support for this would entail a similar amount of work that the Github module set took - I also suspect we could abstract various bits of the existing IMAP code where the two protocols might overlap. |
Dear Jason, Now is a good time to start the implementation: Thanks!!! |
@marclaporte how I see it, the most important information is, that during the upcoming weeks/months, JMAP will finally become an IETF standard. Which is a huge step from my point of view, but seems to be also a huge step forward form the JMAP developers point of view (the IETF processes increased the JMAP quality significantly). |
@dumblob @marclaporte I agree and that was an interesting read, thanks for the link. It's a new year and I'm feeling ambitious so I'm going to bump this up and see what I can get done. The other new feature in the works is cardDav, which has read only support now, and I want to expand that to read/write then move on to calDav support. No reason we can't push both that and JMAP forward at the same time :) One prerequisite for this is I want to create a "skeleton" module set for new data sources. Sort of a template of all (or at least most) of the entry points required to create a new data source inside Cypht. Then I can use that as the foundation for supporting JMAP, as well as making it easier to build support for any other protocol we want to implement. |
A slim PHP Library is missing at current time. There is one for Zend: There is many for JS (which is not compatible with current architecture of cypht?!) Protocol Meta for Client: I think it is possible to extract the Implementation out of the Zend Framework from here: But the License is quite limiting: |
For the record, there is also a JMAP<->IMAP proxy which could be useful for testing of some corner cases. |
@ulfgebhardt and @dumblob thanks for the resources! I suspect we will be writing our own JMAP library in PHP for this. Once we have the "skeleton" module set in place for data sources, most of the boilerplate required to set this up in Cypht is in place. Since JMAP is very RESTful, we can leverage our existing API wrapper code that makes doing RESTy things easy. The final bit is the actual work of deciphering the spec and building out the parsing and logical flows which I suspect won't be nearly as hard as IMAP. We also have a nice set of functions in the core module set for common operations so we can just re-use things we use now (like decoding mime encoded header values for example). |
As I dig into this I think I might put JMAP support into the IMAP and SMTP module sets. It will likely be a LOT easier with the only downside being you need those 2 sets enabled to use JMAP. After much head scratching and trial and error, I have a git checkout of Cyrus IMAP running locally with JMAP support - which should be the most up to date WRT JMAP specs as we can get :) |
Update: spent some time on this over the weekend. Some random thoughts:
After getting folder navigation going I tried to add old school mailbox browsing, but can't seem to get quite what I need yet. Surely I'm doing something wrong but I filed an issue anyway because whatever is not working I just don't see it yet: cyrusimap/cyrus-imapd#2627 |
As expected the issue I was having with Cyrus JMAP was not a bug, but a configuration issue on my end. With there help we are back in business :) |
Update: Now able to browse mailboxes and have a basic message view working. Here is the roadmap to MVP:
Once we get that far, I want to circle back around to the library code and see what can be done to improve the performance. Right now we are using JMAP in a very IMAP like way, but we can streamline that in a number of places to take advantage of JAMP features like multiple "commands" in a single API call. |
Update: Ran into another problem with Cyrus JMAP that maybe is not me doing something wrong :) filed an issue (cyrusimap/cyrus-imapd#2631). In the meantime, Cypht JMAP support now works with combined views (unread, flagged, everything). I also reworked the folder logic in the library and was able to remove some module level work-arounds that were icky. Then I started on search integration but was thwarted by the above issue. JMAP is complicated (E-mail is complicated!), but it truly is IMAP simplified and modernized. There is lots more to do but we are making great progress a lot faster than I expected. And I'm sort of obsessed with making it feature complete now that I started working on it :) |
Update: Wrong again. My latest issue was me doing something wrong (specifically not configuring the Cyrus search backend properly). Thanks to the helpful feedback from Cyrus devs it is resolved and search works now with JMAP, as does some message operations (flag/read/unread/etc). For core IMAP stuff needing to be converted, I have move, copy, delete, and folder management up next, following by downloading and then I think we are pretty close to IMAP feature complete! |
Initial feature complete drop in replacement for the Hm_IMAP class issue #180
Compatibility layer for JMAP is feature complete! Needs more testing and love but it's working pretty well so far. Here is the code: https://github.com/jasonmunro/cypht/blob/master/modules/imap/hm-jmap.php Comparing this to the IMAP lib: WOW! Up next is wiring the UI to add/delete JMAP servers from the Settings -> Servers page. |
UI update to add/delete JAMP servers is pushed to the master branch :) Next I'm going to scope out using JMAP for SMTP. This should be way easier as we only communicate with SMTP in 2 places of the code if I recall. |
Is anyone of you in contact with JMAP developers or just people around JMAP? It seems Cypht will be one of the very few first clients supporting JMAP - that could be used to raise awareness of Cypht and its security and mobile advantages. We could start by letting Cypht being linked from JMAP websites. Then maybe write a blog post on JMAP sites, etc. |
@dumblob I have been in contact with the JMAP folks, and we are now listed here: https://jmap.io/software.html. Also I wrote a blog post on JMAP that made it to the front page of hacker news last night and it's driving a TON of traffic :) https://unencumberedbyfacts.com/2019/01/24/jmap-its-like-imap-but-not-really/ |
Bron Gondwana presented JMAP at FOSDEM 2019. Slides with Speakers' Notes: General page: https://fosdem.org/2019/schedule/event/email_standards/ Well done Bron! And well done Jason for such a quick implementation! |
For the benefit of others linked to this issue from the JMAP site, my understanding is that this issue remains open only because the JMAP feature isn't yet in a (non-RC) release? I.e. it is in master, and the (at time of writing) latest 'release', v1.2.0-rc1. I look forward to trying it, thanks! |
@OJFord : yes, that is correct. And as this is a new feature, on a new protocol, expect it to be bumpy, and please file new issues for bugs and missing features. I encourage you to get Cypht master and start testing it out. I feel master is stable enough to be released. In Tiki, we get Cypht master from Packagist and it works quite well: If you have trouble installing Cypht master, you can install a daily build of Tiki |
Cyrus IMAP 3.2.0-rc1 released (with JMAP support)
|
Cyrus IMAP 3.2.1 released (with JMAP support) |
JMAP online meetings/hackathons occur from time to time. We could join one at one point to review our implementation: |
After over 1100 commits, we can see that the JMAP mail spec has stabilized: The JMAP community is now focusing on other aspects of JMAP, like contacts, calendars, etc.
So it's a good time to get back to JMAP in Cypht. AFAICT, the most complete server implementation is with Cyrus. There are over 1800 commits for the term "JMAP": Maybe we can use https://github.com/cyrusimap/cyrus-docker/ which @ajaysusarla has recently worked on? Thanks! |
I was recently made aware of https://github.com/barnslig/jmap-proxy |
Hi, Thank you for the great work. I'm in the process of integrating cypht in a email stack. I just tested JMAP by manually configuring it and it works great. However, there is no docker configuration options for JMAP (as there is for IMAP) and nothing configurable in cypht/hm3.ini either. How can JMAP be globally configured? Thanks. |
RE:
AND
I see that you postponed the idea of a "skeleton" module set but now we need it to support https://dev.tiki.org/Email-as-a-first-class-citizen - we want a Tiki-based data storage for emails and still use the excellent Cypht UI to handle the email-related actions like displaying, replying, etc. I started adding as part of the imap module but that quite heavily depends on Hm_IMAP and Hm_JMAP. I think I have a couple of options integrating here:
|
Far as I know my friends, JMAP is just a mirror of the HM-IMAP module. That said, supporting it will not he a hard work. |
@josaphatim will soon test Cypht's JMAP against https://stalw.art/jmap/ via https://jmap.cloud/ |
fwiw, i've tested current HEAD against a selfhosted stalwart-jmap instance, and it doesnt look good (but i might have done something wrong in my setup):
because all calls are GET to here's the complete json reply for {"capabilities":{
"urn:ietf:params:jmap:core":{"maxSizeUpload":50000000,"maxConcurrentUpload":4,"maxSizeRequest":10000000,
"maxConcurrentRequests":4,"maxCallsInRequest":16,"maxObjectsInGet":500,"maxObjectsInSet":500,"collationAlgorithms":["i;ascii-numeric","i;ascii-casemap","i;unicode-casemap"]},
"urn:ietf:params:jmap:mail":{"maxMailboxesPerEmail":null,"maxMailboxDepth":10,"maxSizeMailboxName":255,
"maxSizeAttachmentsPerEmail":50000000,
"emailQuerySortOptions":["receivedAt","size","from","to","subject","sentAt","hasKeyword","allInThreadHaveKeyword","someInThreadHaveKeyword"],
"mayCreateTopLevelMailbox":true},
"urn:ietf:params:jmap:websocket":{"url":"ws://jmap.test:8081/jmap/ws","supportsPush":true}},
"accounts":{"d":{"name":"accountname","isPersonal":true,"isReadOnly":false,"accountCapabilities":{"urn:ietf:params:jmap:core":{"maxSizeUpload":50000000,"maxConcurrentUpload":4,"maxSizeRequest":10000000,
"maxConcurrentRequests":4,
"maxCallsInRequest":16,"maxObjectsInGet":500,"maxObjectsInSet":500,"collationAlgorithms":["i;ascii-numeric","i;ascii-casemap","i;unicode-casemap"]},
"urn:ietf:params:jmap:mail":{"maxMailboxesPerEmail":null,"maxMailboxDepth":10,"maxSizeMailboxName":255,"maxSizeAttachmentsPerEmail":50000000,
"emailQuerySortOptions":["receivedAt","size","from","to","subject","sentAt","hasKeyword","allInThreadHaveKeyword","someInThreadHaveKeyword"],
"mayCreateTopLevelMailbox":true},
"urn:ietf:params:jmap:websocket":{"url":"ws://jmap.test:8081/jmap/ws","supportsPush":true}}}},
"primaryAccounts":
{"urn:ietf:params:jmap:core":"d","urn:ietf:params:jmap:mail":"d","urn:ietf:params:jmap:websocket":"d"},
"username":"user@jmap.test",
"apiUrl":"http://jmap.test:8081/jmap/",
"downloadUrl":"http://jmap.test:8081/jmap/download/{accountId}/{blobId}/{name}?accept={type}",
"uploadUrl":"http://jmap.test:8081/jmap/upload/{accountId}/",
"eventSourceUrl":"http://jmap.test:8081/jmap/eventsource/?types={types}&closeafter={closeafter}&ping={ping}","state":"753b9c77"} i'll poke at various things to understand better how it's supposed to work |
well reading through edit: seems
hardcoding
|
one of the things i found out from the above json & in testing (but maybe i should file a distinct issues for stalwart-jmap compatibility issues ?):
, but a toplevel
Thus here's the post sent as logged by nginx via
|
playing with curl, it seems what is sent is escaped too much ? "Method/get" works, but cypht sends "Method\/get" which is what triggers a 400 from the server.
|
Fixed the above by adding |
"JMAP is intended to be a new standard for email clients to connect to mail stores. It therefore intends to primarily replace IMAP + SMTP submission. It is also designed to be more generic such that it can be extended with contacts, calendars in the future (replacing CardDAV/CalDAV). It does not replace MTA-to-MTA SMTP transmission."
http://jmap.io/
The text was updated successfully, but these errors were encountered: