e-mail is not recognised as bouncemail and is skipped

[mrwushu]

I would just like to inquire something regarding Sisimai.

I am currently using this version:

    (no description)
    A/AK/AKXLIX/Sisimai-4.13.1.tar.gz
    /usr/local/share/perl5/Sisimai.pm
    Installed: 4.13.1
    CPAN:      v4.13.1  up to date
    azuma, kuniyuki (AKXLIX)
    akxlix@cpan.org

---

I have been using the above version of Sisimai and have encountered the following problem.

There seem to be a problem in the analysis of an email if it contains multiple Chinese characters.

the return of Sisimai->make( $mailbox ); does not include e-mails that have multiple chinese characters.

the following are the test scenarios I have tried:
(1) Long english email. Result: Success
(2) multiple chinese characters email. Result: Fail
(3) minimal chinese characters email. Result: Success

Also attached is the mailbox copy of the 3 mails above.

As per initial feedback regarding this issue via e-mail consultation:
"Message body of the second mail in inbox.txt is BASE64 encoded string.
As of present, Sisimai does not decode BASE64 string and can not parse such bounce message."

inbox.txt

[azumakuniyuki]

Thank you for the detailed report and sample email above. The entire message body of the second message of "inbox.txt" is BASE64 encoded text:

Date: Wed, 25 Nov 2015 02:10:15 +0000
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: base64

RGVsaXZlcnkgdG8gdGhlIGZvbGxvd2luZyByZWNpcGllbnQgZmFpbGVkIHBlcm1hbmVudGx5OgoK
ICAgICBkb2Vzbm90ZXhpc3RAc2FueXV0ZWNobm9zLXBoLmNvbQoKVGVjaG5pY2FsIGRldGFpbHMg
b2YgcGVybWFuZW50IGZhaWx1cmU6IApHb29nbGUgdHJpZWQgdG8gZGVsaXZlciB5b3VyIG1lc3Nh
...

The following sting is decoded text of the message above.

Delivery to the following recipient failed permanently:

     doesnotexist@sanyutechnos-ph.com

Technical details of permanent failure: 
Google tried to deliver your messa

I think, implementation of additional codes for decoding BASE64 text in a message body required to parse the message like the second mail in "inbox.txt". It seems that implementation is not so difficult.

[azumakuniyuki]

The second message could be parsed by the code of "message-body-is-base64" branch and the codes return the following results:

    [1] Sisimai::Data  {
        internals: {
            action           "failed",
            addresser        Sisimai::Address  {
                internals: {
                    address   "cacco.taiwan@gmail.com",
                    alias     "",
                    host      "gmail.com",
                    user      "cacco.taiwan",
                    verp      ""
                }
            },
            alias            "",
            deliverystatus   "5.1.1",
            destination      "sanyutechnos-ph.com",
            diagnosticcode   "Delivery to the following recipient failed permanently: Technical details of permanent failure: Google tried to deliver your message, but it was rejected by the server for the recipient domain sanyutechnos-ph.com by aspmx.l.google.com. [2a00:1450:4010:c09::1a]. The error that the other server returned was: 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 https://support.google.com/mail/answer/6596 t8si12312940lbb.50 - gsmtp",
            diagnostictype   "SMTP",
            feedbacktype     "",
            lhost            "gmail-pop.l.google.com",
            listid           "",
",          messageid        "CA+7CZmaMHS_g+Kc7LOpyFPmqB8ahkPidfTnR2PqGwRdLeb79bw@mail.gmail.com
            reason           "userunknown",
            recipient        Sisimai::Address  {
                internals: {
                    address   "doesnotexist@sanyutechnos-ph.com",
                    alias     "",
                    host      "sanyutechnos-ph.com",
                    user      "doesnotexist",
                    verp      ""
                }
            },
            replycode        550,
            rhost            "10.25.142.84",
            senderdomain     "gmail.com",
            smtpagent        "US::Google",
            smtpcommand      "",
            softbounce       0,
            subject          "發發發發發",
            timestamp        Wed Nov 25 11:10:15 2015,
            timezoneoffset   +0800,
            token            "e79768a14db7b26d4e1544655c92661a38ad74a8"
        }
    },

[azumakuniyuki]

Created Pull-Request #132