Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

write ECONNRESET / write EPIPE / API error with HTTP Code: 400 /Error: API error with HTTP Code: 505 #39

Open
japrescott opened this Issue · 8 comments

3 participants

@japrescott

Hey mr SaltwaterC!

Since recent releases of node.js (0.6.17 and 18) I've noticed some strange errors while exchanging data between node.js and S3. They don't happen everytime, so its very hard for me to reproduce and from my perception, it seems they happen at random and the ECONNRESET as well as EPIPE errors happen the most. Since your "mr insight" to me, I wanted to drop a line here and see if you had to battle against this yourself in recent times...

The errors occur while using node 0.6.18 and aws2js 0.6.15 (and I tried to log as much about the error as possible);

ECONNRESET;

write ECONNRESET
{"code":"ECONNRESET","errno":"ECONNRESET","syscall":"write"}
{ [Error: write ECONNRESET]
  [stack]: [Getter/Setter],
  [arguments]: undefined,
  syscall: 'write',
  errno: 'ECONNRESET',
  [message]: 'write ECONNRESET',
  code: 'ECONNRESET',
  [type]: undefined }
Error: write ECONNRESET
    at errnoException (net.js:670:11)
    at Object.afterWrite [as oncomplete] (net.js:503:19)
{"code":"ECONNRESET","errno":"ECONNRESET","syscall":"write"}

    

EPIPE;

write EPIPE
{"code":"EPIPE","errno":"EPIPE","syscall":"write"}
{ [Error: write EPIPE]
  [stack]: [Getter/Setter],
  [arguments]: undefined,
  syscall: 'write',
  errno: 'EPIPE',
  [message]: 'write EPIPE',
  code: 'EPIPE',
  [type]: undefined }
 Error: write EPIPE
    at errnoException (net.js:670:11)
    at Object.afterWrite [as oncomplete] (net.js:503:19)

400;

API error with HTTP Code: 400
 {"headers":{"transfer-encoding":"chunked","date":"Mon, 21 May 2012 16:13:05                 GMT","connection":"close","server":"AmazonS3"},"code":400}
 { [Error: API error with HTTP Code: 400]
  [stack]: [Getter/Setter],
  [arguments]: undefined,
  [message]: 'API error with HTTP Code: 400',
  code: 400,
  [type]: undefined,
  headers: 
   { 'transfer-encoding': 'chunked',
     date: 'Mon, 21 May 2012 16:13:05 GMT',
     connection: 'close',
     server: 'AmazonS3' } }
Error: API error with HTTP Code: 400
    at /home/ubuntu/node_modules/aws2js/lib/internals.js:130:14
    at /home/ubuntu/node_modules/aws2js/node_modules/libxml-to-js/lib/libxml-to-js.js:180:3
    at /home/ubuntu/node_modules/aws2js/lib/internals.js:128:4
    at IncomingMessage.<anonymous> (/home/ubuntu/node_modules/aws2js/lib/internals.js:237:10)
    at IncomingMessage.emit (events.js:88:20)
    at HTTPParser.parserOnMessageComplete [as onMessageComplete] (http.js:130:23)
    at CleartextStream.socketOnData [as ondata] (http.js:1288:20)
    at CleartextStream._push (tls.js:375:27)
    at SecurePair.cycle (tls.js:734:20)
    at EncryptedStream.write (tls.js:130:13)

    

ECONNRESET

socket hang up
 {"code":"ECONNRESET"}
 { [Error: socket hang up]
  [stack]: [Getter/Setter],
  [arguments]: undefined,
  [message]: 'socket hang up',
  code: 'ECONNRESET',
  [type]: undefined }
 Error: socket hang up
    at createHangUpError (http.js:1124:15)
    at CleartextStream.socketCloseListener (http.js:1173:23)
    at CleartextStream.emit (events.js:88:20)
    at Array.2 (tls.js:792:22)
    at EventEmitter._tickCallback (node.js:190:38)

    

505;

API error with HTTP Code: 505
 {"headers":{"date":"Mon, 21 May 2012 16:13:05 GMT","connection":"close","server":"AmazonS3"},"code":505}
 { [Error: API error with HTTP Code: 505]
  [stack]: [Getter/Setter],
  [arguments]: undefined,
  [message]: 'API error with HTTP Code: 505',
  code: 505,
  [type]: undefined,
  headers: 
   { connection: 'close',
     server: 'AmazonS3',
     date: 'Mon, 21 May 2012 16:13:05 GMT' } }
 Error: API error with HTTP Code: 505
    at /home/ubuntu/node_modules/aws2js/lib/internals.js:130:14
    at /home/ubuntu/node_modules/aws2js/node_modules/libxml-to-js/lib/libxml-to-js.js:180:3
    at /home/ubuntu/node_modules/aws2js/lib/internals.js:128:4
    at IncomingMessage.<anonymous> (/home/ubuntu/node_modules/aws2js/lib/internals.js:237:10)
    at IncomingMessage.emit (events.js:88:20)
    at CleartextStream.socketCloseListener (http.js:1166:15)
    at CleartextStream.emit (events.js:88:20)
    at Array.0 (tls.js:792:22)
    at EventEmitter._tickCallback (node.js:190:38)

505;

The server prematurely closed the connection and there was no data.
 {"headers":{"date":"Mon, 21 May 2012 16:13:05 GMT","connection":"close","server":"AmazonS3"},"code":505}
 { [Error: The server prematurely closed the connection and there was no data.]
  [stack]: [Getter/Setter],
  [arguments]: undefined,
  [message]: 'The server prematurely closed the connection and there was no data.',
  code: 505,
  [type]: undefined,
  headers: 
   { connection: 'close',
     server: 'AmazonS3',
     date: 'Mon, 21 May 2012 16:13:05 GMT' } }
 Error: The server prematurely closed the connection and there was no data.
    at IncomingMessage.<anonymous> (/home/ubuntu/node_modules/aws2js/lib/internals.js:257:19)
    at IncomingMessage.emit (events.js:64:17)
    at CleartextStream.socketCloseListener (http.js:1168:13)
    at CleartextStream.emit (events.js:88:20)
    at Array.0 (tls.js:792:22)
    at EventEmitter._tickCallback (node.js:190:38)

I hope you have some sort of direction you could point me towards :)

@SaltwaterC
Owner

Found this on AWS forums: https://forums.aws.amazon.com/message.jspa?messageID=39255 - however, I need to properly reproduce it. Escaping the path for AWS is a bitch.

@japrescott

Hey Stefan

Thank you for getting back to me so fast!

I've checked my code for global variables and have now escaped the urls (which doesn't change the urls -> nothing to escape)
I can't get this error to happen while uploading one file (and creating 4thumbs of it, utilizing GraphicsMagick) and streaming it to s3. It seems to only occurs with multiple file uploads (but not everytime) and when it happens, it still successfully puts 2 or 3 of the thumbs on s3.

this url worked;
/tmp/9/4/4/9/a/f/9449afbe420ce5e8cebbb81327c07ef8/1318159759670.jpg/0/thumb.png
while in a parallel async loop I also try to put this url, which fails;
/tmp/9/4/4/9/a/f/9449afbe420ce5e8cebbb81327c07ef8/1318159759670.jpg/0/normal.png

thumb.png exists on s3

I'll try to nail down this error or try to setup a git where you could reproduce this error...

on a side note; do you know how to force HTTP1.0, as it is stated further down in your link?

@SaltwaterC
Owner

Unfortunately node.js supports only HTTP 1.1. Anyway, S3 doesn't support chunked transfers for PUT's, therefore the requests are pretty much backward compatible with 1.0. The core library has HTTP/1.1 hardcoded all over the place.

Since you mentioned a loop, by any chance do you try to access variables from the scope of the loop within the callback itself? Here's an example:

for (var i = 0; i < 10; i++) {
    process.nextTick(function () {
        console.log(i);
    });
}
@johanbaath

Status on this issue? I get random EPIPE errors as well...

@japrescott

@johanbaath sadly, I haven't been able to figure out what causes this. I've been tinkerin arround with nginx conf and rewriting part of my node code, but haven't catched the error or was able to eliminate it.
Isaac Schleuter somewhere writes about the EPIPE error, that you need to expect one end of an http connection to suddenly stop. In my case, that would be AWS disconnecting while receiving a file.
Maybe the solution is simply just to retry a write to aws if you get a error from aws2js (which I doubt is the "real" solution...)

@SaltwaterC
Owner

Unfortunately the AWS APIs are hardly the most stable. That's the reason most of the work went on detecting the service side errors, at the expense of letting the user to properly use the provided functionality. I could implement some sort of mechanism for recovering the temporary errors of an AWS API, but I need to be sure that the error is triggered by the API vs an error triggered by the user code. EPIPE, ECONNRESET, 505 - may fit this description. 400 (aka bad request) usually means that either the HTTP request is poorly formatted (some edge cases?) or the request is missing some data that the API expects (user error).

In order to improve something in my side, I also need some method to reproduce the issue. Unfortunately, the unit tests didn't reproduce it. They do sometimes fail with random errors on various query APIs.

@SaltwaterC
Owner

Found the EPIPE issue. Managed to reproduce it in a short timeframe with s3.put(). The HTTP request is issued twice, while the callback is called twice. The first request fails with EPIPE, but the second is successful. The object is on S3 like nothing happened.

The only place where a request is issued multiple times is for 307 redirects. There's the race condition that triggers the double callback. I assume that the pipe call to the write stream continues even though 307 is returned, but S3 reverted to normal before I could actually test that hypothesis. I updated the unit test to catch these cases when they happen, but I still have to patch the code to avoid this. I'll keep you posted.

@SaltwaterC
Owner

v0.7.11 handles things differently. All the S3 requests with HTTP request bodies (PUT / POST) use the Expect: 100-continue header, while the body is written to the socket only when the request object fires the continue event. Should mitigate or remove the EPIPE issues, maybe the others too. Can't tell exactly since I didn't find a way to reliably reproduce this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.