Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to save mail in cyrus imapd #2828

Closed
rcubetrac opened this issue May 2, 2010 · 19 comments

Comments

@rcubetrac
Copy link

commented May 2, 2010

Reported by felfert on 2 May 2010 11:43 UTC as Trac ticket #1486712

I'm using cyrus (2.3.7) and am trying to get RCMail 0.4-beta running.
During my tests, i noticed, that RCM always reports an error, when it is appending a mail to some imap folder (e.g.: saving in Sent after sending the mail). Fortunately, being a developer myself, I was able to dig into the code: It turns out, that RCM does not adhere to RFC when it is using the function appendFromFile() in program/includes/rcube_imap_generic.php. Instead of using CRLF as line-delimiter, it is using LF which in turn triggers an error in cyrus. As you might know, cyrus is very picky about newlines as it adheres very close to RFCs.

In order to show CR and LF in the logs, i instrumented the existing logging slightly (now shown as \r and \n respectively). This change is attached in the first attachment (imap-debug.patch).

Will attach more stuff after submission ...

Migrated-From: http://trac.roundcube.net/ticket/1486712

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 2, 2010

Comment by felfert on 2 May 2010 12:59 UTC

Description of the attached logs:

All Logging was done via syslog with my patch imap-debug.patch applied.

default-setup.log was created by composing a mail with a text attachment in HTML mode and hitting the "save" button. It can be clearly seen, that the default delimiter (automatically assigned to "\n") is utterly wrong and results in cyrus's response "NO Message contains bare newlines".

manual-delimiter.log was created by the same action after changing

$rcmail_config['mail_header_delimiter'] = "\r\n";

In this case, almost all content is sent correctly, except for two linefeeds in the plain-text part.

manual-delimiter-textonly.log finally shows success.

Conclusion:

The problem is twofold:

  • IMHO, the concept of using mail_header_delimiter is wrong. IMAP should not use the mail_header_delimiter. Instead, it should always use CRLF. If i guess why it was introduced in the first place, i can imagine, that on Unix - when piping messages into external commands - it might be necessary to us linefeeds.
  • When composing html messages, the creation of the plain-text part should consider that and always create CRLF line endings.

Will report more findings when I have debugged the issue further ...

Cheers
-Fritz

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 2, 2010

Comment by felfert on 2 May 2010 14:16 UTC

Hi again,

I just attached htmlmail-save-delimiterfix.patch which fixes the error seen in the testcase shown in manual-delimiter.log.

Verbose description:[[BR]]
When composing a html mail, an alternative text/plain part is generated in program/steps/mail/sendmail.inc, around line 430. There, the class html2text is used to parse the html part and produce the plain text. This class always returns single linefeeds in the return value of its method get_text(). Therefore, that value has to be converted to use CRLF before being processed further.

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 3, 2010

Comment by @alecpl on 3 May 2010 06:53 UTC

Could you provide an imap account on your server for testing?

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 3, 2010

Comment by felfert on 3 May 2010 08:31 UTC

Replying to alec:

Could you provide an imap account on your server for testing?
Unfortunately not (it's a customer installation). However i can offer to setup an identical server on VMWare or VirtualBox (your choice) and give you that VM-Image so you can test at your location.
Please let me know if that's ok for you and - if yes - your personal preference (VBox or VMWare)

Cheers
-Fritz

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 12, 2010

Comment by rgalps on 12 May 2010 17:27 UTC

Hi There

I'm having a pretty much identical issue with 0.4-beta and cyrus imap, but instead of when saving a message, it occurs when sending a message that has an attachment:

May 12 18:08:06 mta1 roundcube: 18:08:06 +0100: User russ@lrhosting.net [Message for xxx@xxx; 250: 2.0.0 Ok: queued as D058416D7
May 12 18:08:06 mta1 roundcube: 12-May-2010 18:08:06 +0100: IMAP Error: Could not save message in Sent in /home/webmail2/public_html/program/steps/mail/sendmail.inc on line 621 (POST /?_task=mail&_action=send)

The error returned from cyrus is: "a NO Message contains bare newlines".

I've applied the patch suggested here (even thought it only seems valid for HTML emails) - it's had no affect, same problem. Doesn't matter what the attachment is (I've tried a tiny text file with 'hello' in it, and a 12MB file) always the same error. Save case with HTML / Plain Text emails.

I can provide a test IMAP account to replicate the issue if required.

Please let me know if any further is needed.

Thanks

Russ

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 12, 2010

Comment by felfert on 12 May 2010 19:14 UTC

Replying to rgalps:

[I've applied the patch suggested here (even thought it only seems valid for HTML emails) - it's had no affect, same problem. Doesn't matter what the attachment is (I've tried a tiny text file with 'hello' in it, and a 12MB file) always the same error. Save case with HTML / Plain Text emails.
...
The patch fixes only the conversion routine. In order to make it work completely, you also have to set the mail_header_delimiter config variable to "\r\n" (See my description of logfiles above).
Apart from that there's no difference between saving a mail during (actually after) sending and saving a mail as draft. The code is the same, just the destination folder is (usually) a different one.

Cheers
-Fritz

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 12, 2010

Comment by rgalps on 12 May 2010 19:24 UTC

Replying to felfert:

The patch fixes only the conversion routine. In order to make it work completely, you also have to set the mail_header_delimiter config variable to "\r\n" (See my description of logfiles above).
Apart from that there's no difference between saving a mail during (actually after) sending and saving a mail as draft. The code is the same, just the destination folder is (usually) a different one.

Thanks Fritz

I can confirm that the patch does fix the issue as long as you have the mail_header_delimiter set - that was the step I was missing.

Hopefully this can be fixed a bit more eloquently so we don't have to manually set that config param in a later release.

Russ

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 12, 2010

Comment by felfert on 12 May 2010 20:33 UTC

Replying to rgalps:

Hopefully this can be fixed a bit more eloquently so we don't have to manually set that config param in a later release.

I intentionally left that to the core developers. In my opinion, that variable should not be necessary at all (if all targeted IMAP servers stick to RFCs, it could simply be hard coded to "\r\n"). Unfortunately, there is nothing in the SVN commit logs mentioning the reason why it was made configurable in the first place. Anyway, the current implementation of setting it automatically depending on the OS of the host running the PHP script is definitively wrong. If variable at all, the "automatic" should depend on the IMAP server being used.

Cheers
-Fritz

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 13, 2010

Comment by @alecpl on 13 May 2010 11:55 UTC

On http://pl.php.net/manual/en/function.mail.php is a note "If messages are not received, try using a LF (\n) only. Some poor quality Unix mail transfer agents replace LF by CRLF automatically (which leads to doubling CR if CRLF is used). This should be a last resort, as it does not comply with RFC 2822." This may be why mail_header_delimiter has been provided. If so, I think we could use "\r\n" everywhere and just convert if needed for mail() function (just right before its use).

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 24, 2010

Comment by @thomascube on 24 May 2010 12:30 UTC

I agree to Alec's suggestion. Roundcube doesn't know much about the IMAP backend and therefore should stick to RFC. But (fortunately) we know when an installation is using PHP's crappy mail() function and we could react on that by converting CRLF with the value configured in mail_header_delimiter.

BTW: it's correct that this mail() function and it's unpredictable behavior was the reason why we introduced the mail_header_delimiter config option. There was an issue with Gmail which could not read messages sent with CRLF using mail().

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 29, 2010

Comment by @alecpl on 29 May 2010 17:01 UTC

Fixed in ac8edbe.

@rcubetrac

This comment has been minimized.

Copy link
Author

commented May 29, 2010

Status changed by @alecpl on 29 May 2010 17:01 UTC

new => closed

@rcubetrac

This comment has been minimized.

Copy link
Author

commented Sep 2, 2010

Comment by mschiff on 2 Sep 2010 14:10 UTC

Sorry if reopening here is wrong, but I got the same issue with 0.4stable.

With the default setting of NULL (auto-detect) of mail_header_delimiter I am not able to store messages having attachments to the Sent folder (Cyrus IMAP)

After setting mail_header_delimiter to "\r\n" it works mostly.

But I found one reproducable case where the problem still occurs:

  1. Have RC using IMAP (Cyrus)
  2. Send a mail to yourself with an attachment: That works now.
    (3.) Message arrives at your INBOX
  3. Select that message, then open message as new message
  4. Now in the composer, remove the attachment, then try to send -> Same old error again

Looking at the imap debug logs that last header line looks abit odd (opened text file using vi so I can see the different line endings)

----------------------------------------------------------
[16:02:56 +0200](02-Sep-2010): C: MIME-Version: 1.0^M
Content-Type: text/plain;^M
 charset=UTF-8;^M
 format=flowed^M
Content-Transfer-Encoding: 7bit^M
Date: Thu, 02 Sep 2010 16:02:56 +0200^M
From: Foo Bar <foo@bar.net>^M
To: Foo Bar <foo@bar.net>^M
Subject: test^M
Message-ID: <5f121bda4c1c3aff7368cdcaa55d1bf3@localhost>^M
X-Sender: foo@bar.net^M
User-Agent: Roundcube Webmail/0.4\r\n\r\n
[16:02:56 +0200](02-Sep-2010): C: --
[16:02:56 +0200](02-Sep-2010): C: 8AAC 5F46 83B4 DB70 8317  3723 296C 6CCA 35A6 4134
[16:02:56 +0200](02-Sep-2010): C:
----------------------------------------------------------

^M is a real CR (\r) (vi shows this because it did not convert the file due to existing LF endings)

So it seems RC seems to do some escaping on the last header line... thats why I see "\r\n\r\n" instead of ^M and another one in the next line.

Can anybody confirm?

@rcubetrac

This comment has been minimized.

Copy link
Author

commented Sep 2, 2010

Status changed by mschiff on 2 Sep 2010 14:10 UTC

closed => reopened

@rcubetrac

This comment has been minimized.

Copy link
Author

commented Sep 2, 2010

Severity changed by mschiff on 2 Sep 2010 14:10 UTC

blocker => normal

@rcubetrac

This comment has been minimized.

Copy link
Author

commented Sep 4, 2010

Milestone changed by @alecpl on 4 Sep 2010 07:29 UTC

0.4-stable => 0.4.1

@rcubetrac

This comment has been minimized.

Copy link
Author

commented Sep 4, 2010

Comment by @alecpl on 4 Sep 2010 07:45 UTC

Check your config again and make sure you have no duplicated mail_header_delimiter setting, also remember that "\r\n" and '\r\n' is not the same.

@rcubetrac

This comment has been minimized.

Copy link
Author

commented Sep 4, 2010

Comment by @alecpl on 4 Sep 2010 08:01 UTC

I think [and 272a7e5(086767c]) should fix the issue definitely.

@rcubetrac

This comment has been minimized.

Copy link
Author

commented Sep 4, 2010

Status changed by @alecpl on 4 Sep 2010 08:01 UTC

reopened => closed

@rcubetrac rcubetrac closed this Sep 4, 2010
@rcubetrac rcubetrac added this to the 0.4.1 milestone Mar 20, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.