Skip to content

Error in the parser : Object 17 has no method 'toLowerCase' #345

Closed
floross opened this Issue Jan 14, 2014 · 27 comments

8 participants

@floross
floross commented Jan 14, 2014

Hi,

I'm using node-imap to pull every mail from a gmail account fetch them. Sometime I get this error from the imap library when I try to parse the email from gmail.

/home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:426
key = list[i].toLowerCase();
              ^
TypeError: Object 17 has no method 'toLowerCase'
at Parser.parseFetch (/home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:426:19)
at Parser._resUntagged (/home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:258:24)
at Parser._parse (/home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:135:16)
at Parser._tryread (/home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:81:15)
at Readable.Parser._cbReadable [as _read] (/home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:51:12)
at Readable.read (_stream_readable.js:320:10)
at flow (_stream_readable.js:589:52)
at Readable.pipeOnReadable (_stream_readable.js:624:5)
at Readable.EventEmitter.emit (events.js:92:17)
at emitReadable_ (_stream_readable.js:408:10)

I didn't have the content of the email so I can't give you any more clue for this bug. Do you know why this error popup sometimes and not on the same email ?

Moreover maybe this is correlated but I am not able to retrieve all the email from gmail because that's stop aleatory while the downloading.

Thanks for your library and for your time.

@mscdex
Owner
mscdex commented Jan 14, 2014

Hrmm, not sure on this one. Next time temporarily enable debug output by setting debug: console.log in the connection settings and post the output leading up to the error. Hopefully that will give us a better idea of what's not being parsed correctly.

@floross
floross commented Jan 14, 2014

This not gonna be really good to read this but i put you the stack trace before the crach :

 <= '* 112 FETCH (X-GM-THRID 1251723620142606658 X-GM-MSGID 1251723620142606658 X-GM-LABELS ("\\\\Inbox") UID 112 MODSEQ (1424505) INTERNALDATE "30-Oct-2007 09:31:08 +0000" FLAGS (\\Seen) BODY[] {12679}'
 <= '* 112 FETCH (X-GM-THRID 1251723620142606658 X-GM-MSGID 1251723620142606658 X-GM-LABELS ("\\\\Inbox") UID 112 MODSEQ (1424505) INTERNALDATE "30-Oct-2007 09:31:08 +0000" FLAGS (\\Seen) )'
 <= '* 113 FETCH (X-GM-THRID 1252021425741775249 X-GM-MSGID 1252021425741775249 X-GM-LABELS ("\\\\Inbox") UID 113 MODSEQ (1424600) INTERNALDATE "02-Nov-2007 16:24:37 +0000" FLAGS (\\Seen) BODY[] {15289}'
 <= '* 113 FETCH (X-GM-THRID 1252021425741775249 X-GM-MSGID 1252021425741775249 X-GM-LABELS ("\\\\Inbox") UID 113 MODSEQ (1424600) INTERNALDATE "02-Nov-2007 16:24:37 +0000" FLAGS (\\Seen) )'
 <= 'uropoker.com/mail/images/='
 <= 'directlinks_fr.gif" alt=3D"" USEMAP=3D"#direct" width=3D"500" height=3D"='

 ...Contains of the mail in html : for the readeable of the stack trace I removed this part...

 <= '* 111 FETCH (X-GM-THRID 1251109080323478703 X-GM-MSGID 1251109080323478703 X-GM-LABELS ("\\\\Inbox") UID 111 MODSEQ (1424427) INTERNALDATE "23-Oct-2007 14:43:16 +0000" FLAGS (\\Seen) BODY[] {20015}'
 New mail
 New mail
 <= '* 111 FETCH (X-GM-THRID 1251109080323478703 X-GM-MSGID 1251109080323478703 X-GM-LABELS ("\\\\Inbox") UID 111 MODSEQ (1424427) INTERNALDATE "23-Oct-2007 14:43:16 +0000" FLAGS (\\Seen) ge:url(http://www.europoker.com/mail='

 /home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:426
key = list[i].toLowerCase();
              ^
 TypeError: Object http://www.europoker.com/mail= has no method 'toLowerCase'
at Parser.parseFetch (/home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:426:19)
at Parser._resUntagged (/home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:258:24)
at Parser._parse (/home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:135:16)
at Parser._tryread (/home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:81:15)
at CleartextStream.Parser._cbReadable (/home/florian-rossiaud/provider-gmail/node_modules/imap/lib/Parser.js:51:12)
at CleartextStream.EventEmitter.emit (events.js:92:17)
at emitReadable_ (_stream_readable.js:408:10)
at _stream_readable.js:401:7
at process._tickCallback (node.js:415:13)

If you want the contains of the previous mail i can send you via email.

In the stack trace New Mail is a trace from my code when I retrieve a new email

@mscdex
Owner
mscdex commented Jan 14, 2014

Ok, so the contents of string literals (the email bodies in this case) like that shouldn't be showing up in the debug output like that, so something is definitely wrong.

Please send me this particular email exactly as it is, unmodified, to mscdex at mscdex dot net.

Also: what version of node and node-imap are you using?

@floross
floross commented Jan 16, 2014

I didn't have the email (lost while operating on my computer) but what we should know is this error not depends of the email because when I retry on the same bunch of emails, the error doesn't appears wit the same email (compared with the uid).

This error is still popup when my tests.

I use the node version : v0.10.24 and the node-imap : imap@0.8.7

@floross
floross commented Jan 17, 2014

You can see my code on this repo :

https://github.com/Papiel/gmail.provider.anyfetch.com/blob/fixing/lib/helpers/retrieve.js

I used a setTimeout because sometime the imap was just stop between two email without restarting. Moreover the error of the parser was existing before the recursion.

@mscdex
Owner
mscdex commented Feb 28, 2014

If you are able to get a packet trace of when this occurs, that would be helpful too. It must be hitting an edge case somewhere that isn't accounted for.

@mhahn
mhahn commented Apr 18, 2014

i've been coming across the same issue randomly. i can't reproduce it every time.

@mhahn
mhahn commented Apr 18, 2014

ie:

    TypeError: Object 2 has no method 'toLowerCase'
  at Parser.parseFetch (/home/vagrant/jarvis/jarvis-imap-service/node_modules/imap/lib/Parser.js:427:19)
  at Parser._resUntagged (/home/vagrant/jarvis/jarvis-imap-service/node_modules/imap/lib/Parser.js:253:24)
  at Parser._parse (/home/vagrant/jarvis/jarvis-imap-service/node_modules/imap/lib/Parser.js:134:16)
  at Parser._tryread (/home/vagrant/jarvis/jarvis-imap-service/node_modules/imap/lib/Parser.js:80:15)
  at Readable.Parser._cbReadable [as _read] (/home/vagrant/jarvis/jarvis-imap-service/node_modules/imap/lib/Parser.js:51:12)
  at Readable.read (_stream_readable.js:320:10)
  at flow (_stream_readable.js:589:52)
  at Readable.pipeOnReadable (_stream_readable.js:624:5)
  at Readable.EventEmitter.emit (events.js:92:17)
  at emitReadable_ (_stream_readable.js:408:10)
  at emitReadable (_stream_readable.js:404:5)
  at readableAddChunk (_stream_readable.js:165:9)
  at Readable.push (_stream_readable.js:127:10)
  at Parser._parse (/home/vagrant/jarvis/jarvis-imap-service/node_modules/imap/lib/Parser.js:94:14)
  at Parser._tryread (/home/vagrant/jarvis/jarvis-imap-service/node_modules/imap/lib/Parser.js:80:15)
  at CleartextStream.Parser._cbReadable (/home/vagrant/jarvis/jarvis-imap-service/node_modules/imap/lib/Parser.js:51:12)
  at CleartextStream.EventEmitter.emit (events.js:92:17)
  at emitReadable_ (_stream_readable.js:408:10)
  at _stream_readable.js:401:7
  at process._tickDomainCallback (node.js:459:13)
@NathanaelA

I think I know the cause of why this is occurring; don't have a "great" solution yet. I can duplicate the error In the message.on('body') event by piping the stream to something else (Looking at @florian-rossiaud code he was also piping the stream to something else)

When I attempt to pipe the stream to https://github.com/andris9/mailparser (mailparser) then both of the errors reported -- freezing while getting messages and the toLowerCase error appears randomly. The messages it dies on are typically much larger mime(?) encoded messages of some sort and it goes into some endless loop calling _tryread and _parse in .the Parser.js library.

If I do var buffer=''; then inside the message.on('body') do buffer += stream.toString(); then send the buffer to the mailparser in the 'end' event -- I am fine.

I believe either the mailparser or this library is modifying the stream on larger messages and it is causing node-imap to blow up or go into a endless loop because characters are no longer available in the stream(?).

@NathanaelA

My prior comment is partially wrong. The cause of the issues is still piping the stream and not getting the data out before the on.end is done. Not sure why I thought buffer += stream.toString() would work in the above comment. In later testing I figured out that it was not doing what I thought it was. he work around that appears to work perfectly is;

f.on('message', function(msg, seqno) {
  var curStream, mailParser = new MailParser();
  msg.on('body', function (stream) { curStream = stream; });
  msg.on('end', function() { 
     // We need to be completely done using the "curStream before we exit the 'end' function
     var buffer = curStream.read(); // Reads the entire stream into a buffer or string
     mailProcessor.write(buffer);
     mailProcessor.end();
  }
}
@rltvty
rltvty commented Jul 4, 2014

I see this error a few times a day when attempting to fetch the envelope on certain messages. After fetching the raw content for one such message, I have made a simple test script that reproduces the unhandled exception.

Note that HoodieCrow (a library to simulate IMAP) is required for the test. More information about hoodiecrow here: https://www.npmjs.org/package/hoodiecrow.

And the test script:

var hoodiecrow = require('hoodiecrow');
var Imap = require('imap');

//use npm hoodiecrow to simulate an IMAP Inbox with one message
var imap_server = hoodiecrow({
  storage: {
    "INBOX": {
      messages: [
        {raw: "Date: Wed, 30 Mar 2014 02:38:23 +0100\r\n" +
          "Message-Id: <message@test.eu>\r\n" +
          "To: one.two@test.fr, two.three@test.fr\r\n" +
          "Subject: =?ISO-8859-1?Q?##ALLCAPS##123456## - ?=\r\n" +
          "        =?ISO-8859-1?Q?[ALERT][P3][ONE.TWO.FR] ?=\r\n" +
          "        =?ISO-8859-1?Q?Some Subject Line \"D:\\\"?=\r\n" +
          "From: \"Test Account (Rltvty L)\" <account@test.com>\r\n" +
          "Reply-to: account@test.com\r\n" +
          "\r\n" +
          "Bonjour, redacted, adios"}
      ]
    }
  },
  debug: false
});

imap_server.listen(1143);


//use npm imap to connect to IMAP and fetch the envelope
var imap = new Imap({ user: 'testuser', password: 'testpass', host: 'localhost', port: 1143, tls: false });

imap.once('ready', function () {
  imap.openBox('INBOX', true, function () {
    var f = imap.seq.fetch('1', { envelope: true });
    f.once('end', function () {
      imap.end();
      imap_server.close();
    });
  });
});
imap.connect();

Produces this unhandled exception:

TypeError: Object ALERT has no method 'toLowerCase'
    at Parser.parseFetch (./node_modules/imap/lib/Parser.js:430:19)
    at Parser._resUntagged (./node_modules/imap/lib/Parser.js:256:24)
    at Parser._parse (./node_modules/imap/lib/Parser.js:136:16)
    at Parser._tryread (./node_modules/imap/lib/Parser.js:82:15)
    at Socket.Parser._cbReadable (./node_modules/imap/lib/Parser.js:53:12)
    at Socket.EventEmitter.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:408:10)
    at emitReadable (_stream_readable.js:404:5)
    at readableAddChunk (_stream_readable.js:165:9)
    at Socket.Readable.push (_stream_readable.js:127:10)
@mscdex
Owner
mscdex commented Jul 4, 2014

@rltvty Thanks for the test, I was able to reproduce it as well. It should now be fixed in master, please give it a try, @mhahn @florian-rossiaud too.

@rltvty
rltvty commented Jul 4, 2014

@mscdex, it seems like the issue is resolved in master. Thanks for the quick fix.

Can you release a new version with this fix so I can use it in production?

@mscdex
Owner
mscdex commented Jul 4, 2014

@rltvty Done.

@rltvty
rltvty commented Jul 4, 2014

@mscdex Awesome. Thanks again!

@aenario
aenario commented Aug 29, 2014

I have been doing some testing and the error still seems to be present in 0.8.13
My usecase : I download ~200mails from a gmail account.
The behaviour is the same as @Nathanaela : if I .pipe the stream into mailparser, It fail randomly (~1/2 the time), with either a hang + body in log as described in #379 or a "has no method .toLowerCase"as described here. The fail can occurs on any email, so its more likely a race condition. If I buffer it into memory, then apply mailparser on the 'end' event, the error seem to disappear (~12 runs & counting)

@mscdex
Owner
mscdex commented Aug 29, 2014

@aenario Can you provide a minimal test case that reproduces this bug, even if it's only half the time as you said? I've struggled to reproduce the problem and I'm eager to get it fixed.

@aenario
aenario commented Aug 29, 2014

Arch, my code is -shame on me- pretty tightly coupled, I will try to write up a minimal repro.
In #411, you proposed a modified Parser with more logging output.
I made a pastebin using this parser in my app (http://pastebin.com/zHyFiAAt), if it helps.

@aenario
aenario commented Aug 29, 2014

PS: log is limited to 80 char length lines as it was unreadable otherwise, let me know if you need more

@aenario
aenario commented Aug 29, 2014

Reproduction code is pretty standard https://gist.github.com/aenario/b466f9f1cc6dd301e36d
Note that if we change Mailparser thing by the commented lines it work.

@mscdex
Owner
mscdex commented Aug 31, 2014

@aenario Ok, I think I may have a general idea of what is happening, but I need a little more debug output to confirm my suspicions and to help create a test case. Please replace your parser.js with this new debug version and post your resulting output again.

@mscdex mscdex added a commit that closed this issue Aug 31, 2014
@mscdex Parser: ensure no socket read if push() calls _read() during body finish
This fixes an edge case where the following happens:

 * a body stream had data pushed to it such that the highWaterMark was reached
 * when the body stream buffer dips below highWaterMark, _read() is called
 * _read() tries to read more data from the socket, which pushes the last part of the body
 * the last body part push() again calls _read() which in turn reads more data from the socket
 * at this point the parser state is not stable because _body._read and _body are not reset yet, this causes the parser to potentially try to start reading the beginning of a response in the middle of the data for another fetch result for example

Fixes #345
8376f21
@mscdex mscdex closed this in 8376f21 Aug 31, 2014
@mscdex
Owner
mscdex commented Aug 31, 2014

@florian-rossiaud @aenario @mhahn @bblack Can all of you please try the master branch now? I was able to reproduce the issue and I just pushed a fix.

@bblack
bblack commented Sep 1, 2014

Today I tried reproducing the error I described in #411 - both with 0.8.13 (the version I was seeing the issue with), and with latest master - and I haven't seen it on either. So, I can neither confirm nor deny that the problem still exists.

@aenario
aenario commented Sep 8, 2014

hi @mscdex, I was in vacation, will try today and let you know.

@WimAtIHomer

Hi @mscdex, I am having similar problems with the latest release 0.8.13. The latest master solves my problems. Can you make a new release please?

@mscdex
Owner
mscdex commented Sep 22, 2014

@WimAtIHomer Done.

@WimAtIHomer

Thanx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.