sieve notify break =?UTF8?Q? encoded texts

[brong]

From: Deniss Gaplevsky
Bugzilla-Id: 3848
Version: 2.4.17
Owner: Bron Gondwana

[brong]

From: Deniss Gaplevsky

i try to use external notify in sieve like

notify :method "external" :message "$from$~~~ $env-from$~~~ $subject$~~~ $text[20]$~~~ ";

when i send message with subject encoded in =?UTF8?Q? (quota printable) like

Subject: =?UTF8?Q?asdasd_=C4=81=C5=BE=C5=86=C4=AB=C4=BC=C5=AB?=

$subject$ is not correctly decoded:

\xef\xbf\xbd~~~

while =?UTF8?B? works just fine

[brong]

From: Dilyan Palauzov

Does the problem disappear, if you use =?UTF-8 instead of =?UTF8 ?

As of http://www.iana.org/assignments/character-sets/character-sets.xhtml, the correct spelling is with dash.

I used :method "log" and from

Subject: A12345 =?UTF-8?Q?asdasd_=C4=81=C5=BE=C5=86=C4=AB=C4=BC=C5=AB?= ABCD

the generated output was

A12345 asdasd ā�ņīļū

which coincides with the RFC2047 decoder provided at http://dogmamix.com/MimeHeadersDecoder/ .

[brong]

From: Deniss Gaplevsky

the messages were received from wild internet

[brong]

From: Dilyan Palauzov

I tested the base64 decoding with notify :method "log":

Subject: A12345 =?utf-8?b?0JrQstC+IA==?=

is decoded correctly by log, and

Subject: A12345 =?utf8?b?0JrQstC+IA==?= or
Subject: A12345 =?UTF8?b?0JrQstC+IA==?=

are not decoded correctly, contrary to your original message.

I tend to close this as not a bug:

I do not see Cyrus imapd's task to deal with misspelled charsets.

Even messages from wild internet are required to encode their headers with a valid charset.

If the subject is not encoded correctly, then the spam filter shall take care to move the message to the spam folder, and the sieve script shall make no further proceedings on the mail.

[brong]

From: Dilyan Palauzov

Fixed on cyrus-imapd-2.4 with commit 63856fc .

In fact on the master branch there was already support for charset aliases and I back-ported it to cyrus-imapd-2.4.

[brong]

From: Leena Heino

(In reply to comment #4)
> Fixed on cyrus-imapd-2.4 with commit 63856fc .
>
> In fact on the master branch there was already support for charset aliases and
> I back-ported it to cyrus-imapd-2.4.

That patch broke the search completely. With this patch installed the search cannot find anything.

When I remove the patch the search begins to work again. Please revert this patch

[brong]

Attachment-Id: 1541
From: Dilyan Palauzov
Type: text/plain
File: lib_mkchartable.patch

Proposed fix

[brong]

From: Deniss Gaplevsky

please fix the git for 2.4 to be stable - rollback the broken patch or apply the fix

[brong]

From: Dilyan Palauzov

I applied the fix, commit 8b59aeb .

[brong]

From: Deniss Gaplevsky

looks fine for me now and may be closed

[brong]

From: Leena Heino

(In reply to comment #8)
> I applied the fix, commit 8b59aeb .

The patch seems to work ok.

Could you also add character set "Latin1" as alias to iso-8859-1?

[brong]

From: Dilyan Palauzov

I added "latin1" as alias of "iso-8859-1" with commit facf3b8 on branch master and with commit b4de7a5 on branch cyrus-imapd-2.4 .