JMAP Email/import setting hasAttachment when it shouldn't?

[wolfsage]

Email/import of file without attachments results in an Email object with hasAttachment true (but no attachments):

=== BEGIN UPLOAD REQUEST 3455.14 ===
POST http://localhost/jmap/upload/jt-70578ae4-6e66-11e8-b448-533ca35aa5ac@localhost/
Authorization: Basic ...
User-Agent: libwww-perl/6.33
Content-Type: message/rfc822

From: example@example.com
To: example@example.biz
Subject: This is a test
Message-Id: <15288246899.CBDb71cE.3455@cyrus-dev>
Date: Tue, 12 Jun 2018 13:31:29 -0400
MIME-Version: 1.0

This is a very simple message.
=== END UPLOAD REQUEST 3455.14 ===
=== BEGIN UPLOAD RESPONSE 3455.14 ===
HTTP/1.1 201 Created
Connection: close
Date: Tue, 12 Jun 2018 17:31:29 GMT
Vary: Accept-Encoding
Content-Length: 186
Content-Type: application/json; charset=utf-8
Client-Date: Tue, 12 Jun 2018 17:31:29 GMT
Client-Peer: ::1:80
Client-Response-Num: 1

{"accountId":"jt-70578ae4-6e66-11e8-b448-533ca35aa5ac@localhost","blobId":"Gd77fc7cfb6c00cd94b21da64e827fab35913e2c2","size":218,"expires":"2018-06-13T17:31:29Z","type":"message/rfc822"}
=== END UPLOAD RESPONSE 3455.14 ===
=== BEGIN JMAP REQUEST 3455.48 ===
POST http://localhost/jmap/
Authorization: Basic ...
User-Agent: libwww-perl/6.33
Content-Type: application/json

{"methodCalls":[["Email/import",{"emails":{"msg":{"isDraft":false,"isUnread":false,"mailboxIds":{"d9dde7eb-7a53-4812-8ef7-c4ebe9a010e7":true},"isAnswered":false,"isFlagged":false,"blobId":"Gd77fc7cfb6c00cd94b21da64e827fab35913e2c2"}}},"a
w"]],"using":["ietf:jmapmail"]}
=== END JMAP REQUEST 3455.48 ===
=== BEGIN JMAP RESPONSE 3455.48 ===
HTTP/1.1 200 OK
Connection: close
Date: Tue, 12 Jun 2018 17:31:29 GMT
Vary: Accept-Encoding
Content-Length: 263
Content-Type: application/json; charset=utf-8
Client-Date: Tue, 12 Jun 2018 17:31:29 GMT
Client-Peer: ::1:80
Client-Response-Num: 1

{"methodResponses":[["Email/import",{"accountId":"jt-70578ae4-6e66-11e8-b448-533ca35aa5ac","created":{"msg":{"id":"Md77fc7cfb6c00cd94b21da64","blobId":"Gd77fc7cfb6c00cd94b21da64e827fab35913e2c2","threadId":"T468c1e2ebda907cb","size":218}
},"notCreated":{}},"aw"]]}
=== END JMAP RESPONSE 3455.48 ===
=== BEGIN JMAP REQUEST 3455.49 ===
POST http://localhost/jmap/
Authorization: Basic ...
User-Agent: libwww-perl/6.33
Content-Type: application/json

{"using":["ietf:jmapmail"],"methodCalls":[["Email/get",{"ids":["Md77fc7cfb6c00cd94b21da64"]},"ax"]]}
=== END JMAP REQUEST 3455.49 ===
=== BEGIN JMAP RESPONSE 3455.49 ===
HTTP/1.1 200 OK
Connection: close
Date: Tue, 12 Jun 2018 17:31:29 GMT
Vary: Accept-Encoding
Content-Type: application/json; charset=utf-8
Client-Date: Tue, 12 Jun 2018 17:31:29 GMT
Client-Peer: ::1:80
Client-Response-Num: 1
Client-Transfer-Encoding: gzip, chunked

{"methodResponses":[["Email/get",{"state":"74","list":[{"id":"Md77fc7cfb6c00cd94b21da64","blobId":"Gd77fc7cfb6c00cd94b21da64e827fab35913e2c2","threadId":"T468c1e2ebda907cb","mailboxIds":{"d9dde7eb-7a53-4812-8ef7-c4ebe9a010e7":true},"keywords":{"$hasattachment":true},"size":218,"receivedAt":"2018-06-12T17:31:29Z","messageId":["15288246899.CBDb71cE.3455@cyrus-dev"],"inReplyTo":null,"from":[{"name":null,"email":"example@example.com"}],"to":[{"name":null,"email":"example@example.biz"}],"cc":null,"bcc":null,"subject":"This is a test","sentAt":"2018-06-12T17:31:29Z","references":null,"sender":null,"replyTo":null,"bodyValues":{},"textBody":[{"partId":"1","blobId":"Ge9dd999849d79fa314a428a3b8972d4270d2ab5f","size":30,"name":null,"type":"text/plain","charset":"us-ascii","disposition":null,"cid":null,"language":[],"location":null}],"htmlBody":[{"partId":"1","blobId":"Ge9dd999849d79fa314a428a3b8972d4270d2ab5f","size":30,"name":null,"type":"text/plain","charset":"us-ascii","disposition":null,"cid":null,"language":[],"location":null}],"attachedEmails":[],"attachedFiles":[],"hasAttachment":true,"preview":"This is a very simple message."}],"notFound":[],"accountId":"jt-70578ae4-6e66-11e8-b448-533ca35aa5ac"},"ax"]]}
=== END JMAP RESPONSE 3455.49 ===

[brong]

I wonder if this is actually an annotator bug... looks like you're testing against a local instance.

[wolfsage]

Yeah, this is on my local vm. Let me know if you want configs or any info

[rsto]

Yes, looks like an annotator issue for top-level messages:

 7209     if (_wantprop(props, "hasAttachment")) {
 7210         int has_att = 0;
 7211         if (is_toplevel)
 7212             msgrecord_hasflag(msg->mr, JMAP_HAS_ATTACHMENT_FLAG, &has_att);
 7213         else
 7214             has_att = bodies.atts.count > 0;
 7215
 7216         json_object_set_new(email, "hasAttachment", json_boolean(has_att));
 7217     }

But for embedded emails it could be an issue with the JMAP bodypart parser.

[rsto]

Wait a second, this could be a bug in the JMAP Email/import code. I'm just rewriting the code use attachments rather than attachedFiles and attachedEmails, and this is issue is related.

[rsto]

So, this should be fixed in 7677ede (there's also a test at cyrusimap/cassandane@bcfb787).

But there's lots of ceremony in Email/import et al in the JMAP code to handle hasAttachment: I remember now that we decided to provide a Cyrus-native implementation of setting the hasAttachment flag on emails, rather than using the Perl annotator like we do at FastMail.

But currently, it looks as if we set the hasAttachment flag during email creation/import, which triggers the annotator, which again determines the hasAttachment flag. We might choose one of the following options:

• set hasAttachment in both Cyrus and any outside annotator (= keep as is)
• add a flag to Cyrus imapd.conf so the JMAP layer knows if to determine the hasAttachment flag internally (this will save us the cost to determine the flag twice)
• remove the hasAttachment code from Cyrus and require all installations to use an outside annotator (we ship a prototype for this already in the perl subdirectory of the source distribution)

[elliefm]

remove the hasAttachment code from Cyrus and require all installations to use an outside annotator (we ship a prototype for this already in the perl subdirectory of the source distribution)

Wearing release manager hat, I don't like this as an option. People struggle enough with getting fundamentals like smtp and sasl working, without mandating they scaffold up and maintain an annotator script too.

add a flag to Cyrus imapd.conf so the JMAP layer knows if to determine the hasAttachment flag internally (this will save us the cost to determine the flag twice)

This sounds good to me. Default it to on, and places like FastMail that already have that functionality externally can turn it off.