Adds reliability and simplifies, finalizes v1 HTTP API

[johnnyman727]

Previously, the HTTP API required headers to indicate the name of the compiled binary and the name of the project folder. This PR removes those headers and instead deduces the name of the project after unpacking it and grabs the binary name from the Cargo.toml.

Additionally, this PR adds JSON support to the response which not only makes the error handling easier for the client but allows the server to send down compilation logs.

This server should be all set for 1.0.0 release.

[cflewis]

Nit: space missing at end of Cargo.toml

[johnnyman727]

@cflewis updated with your comments.

[cflewis]

LGTM

[johnnyman727]

Thanks! We'll merge whenever we merge the CLI branch just in case there are some more changes to be made.

[johnnyman727]

I'm in the process of deploying this Dockerfile to Digital Ocean. Unfortunately, during testing, I find that the server fails at the point of sending the binary back down to the client with:

  server output of compilation - error: null, stdout:    Compiling hello v0.1.0 (file:///tmp/tessel-rust-compile116613-271-qr3udd/hello)
, stderr:  +495ms
  server Finished cross compilation job. +533ms
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: write after end
    at ServerResponse.OutgoingMessage.write (_http_outgoing.js:430:15)
    at Base64Encode.ondata (_stream_readable.js:542:20)
    at emitOne (events.js:77:13)
    at Base64Encode.emit (events.js:169:7)
    at Base64Encode.Readable.read (_stream_readable.js:368:10)
    at flow (_stream_readable.js:759:26)
    at ServerResponse.<anonymous> (_stream_readable.js:616:7)
    at emitNone (events.js:67:13)
    at ServerResponse.emit (events.js:166:7)
    at Socket.ondrain (_http_common.js:215:44)

It looks to me like the packingStream.once('finish') event is being fired before the packingStream actually stops finishes writing to the res. Then, we write the final bracket to res, packingStream tries to write to it again, and the error is thrown. @LinusU any ideas why finish isn't being thrown at the proper time? It's also interesting that it works fine locally but fails when deployed remotely to Digital Ocean.

[johnnyman727]

It might also be helpful to know that if I replace packingStream.once('finish', ... with a setTimeout(..., 2000), it completes successfully. This confirms to me that the finish event is being fired too early.

[johnnyman727]

It might also be helpful to note that this hack seems to un-break it:

var tarStream = tar.pack(binaryPath, {entries: [binaryName]});
var encodeStream = base64.encode();

tarStream.on('data', (d) => encodeStream.write(d));

tarStream.once('end', () => encodeStream.end());

encodeStream.on('data', (d) => res.write(d));

encodeStream.once('finish', () => {
   res.write("}");
});

[LinusU]

I think the problem is that the finish event if for a writeable stream to indicate that all data has been written to it (i.e. it has been ended). My guess to way this happened to work locally for you is that it will actually work about 25% of the times because of how base64 works (see here).

I think that the fix is as easy as changing .once('finish') into .once('end'). 👍

The end is event is a readable stream event that says that the stream has ended. Since this is a duplex stream it will emit both the writeable events and the readable events.

[johnnyman727]

👍 confirmed fix!

[johnnyman727]

This was deployed on Digital Ocean with:

docker pull johnnyman727/rust-compilation-server:v1
docker run -p 49160:8080 johnnyman727/rust-compilation-server:v1