--slim by default. Fixes gh-412, Fixes gh-418

[rwaldron]

No description provided.

[HipsterBrown]

Nice catch. :)

[HipsterBrown]

Great fixes and refactor. 👍

[rwaldron]

@HipsterBrown thanks for the review :D

[HipsterBrown]

I missed this before. Should this option, and making --slim the default action, be added to the push command as well?

[johnnyman727]

We may want to make this more explicit: "Bundle all files in project including those not used by the program".

[rwaldron]

"Bundle all files in project including those not used by the program, but excluding any files matched by non-negated rules in .tesselignore"

[HipsterBrown]

From @LinusU in another issue:

Preferably don't touch the filesystem at all, aka. get a stream from browserify and pipe that straight to tar.pack. If that isn't possible (I think it is thought), let the temporary file live in the OS temporary directory, using e.g. fs-temp.

Thoughts on this approach @rwaldron?

[rwaldron]

@HipsterBrown that's out of scope for this changeset. I don't want to pig-pile a bunch of changes into one PR.

[HipsterBrown]

@rwaldron That works for me. What about having a better namespaced path for the generated bundle?

[rwaldron]

path for the generated bundle?

Yeah, that should be addressed here, update to follow

[rwaldron]

@HipsterBrown

that's out of scope for this changeset. I don't want to pig-pile a bunch of changes into one PR.

After re-reviewing the conversation, I decided that you and @LinusU were right and this needed to be addressed.

The slim path now...

• bundles with browserify (as before)
• optimizes if one file
• creates an "in-memory" directory using @LinusU's fs-temp
• writes the bundle to that temp dir
• tars the temp dir and resolves with the buffer (as before)
• cleans up the temp dir

No there is no chance of paving over user project files! Thanks for highlighting this issue :)

[LinusU]

Just to clarify about fs-temp, it doesn't create the directory in-memory, it will reside in the os temporary directory (e.g. /tmp).

I think that this is the minimum we should do thought, I'm pro this change.

However, what I really wanted was something like this:

const source = browserify(...).bundle()
const gzip = zlib.createGzipStream()

source.pipe(gzip).pipe(/* to the tessel */)

It's unnecessary to use tar since we are only sending one file over. I also feel it's a bit overkill to create a temporary directory, stream the entire file to disk, use a new stream that iterates directories, then create a tar stream out of that and then pipe that to the tessel. In reality, the only thing that happens is that a few bytes gets prepended to the stream before the file. tar is a very simple format.

But as I said, this is absolutely good for now!

[rwaldron]

Just to clarify about fs-temp, it doesn't create the directory in-memory, it will reside in the os temporary directory (e.g. /tmp).

Yep, I was monitoring the actual creation of dirs—easier to think about an ephemeral concept as "in-memory"

const source = browserify(...).bundle()
const gzip = zlib.createGzipStream()
source.pipe(gzip).pipe(/* to the tessel */)

It's unnecessary to use tar since we are only sending one file over.

This was brought up in another thread and I held off responding because I thought @HipsterBrown's mention of the --full path was sufficient—I based the changes in this patch on your response to that. If we went this route, it would require having a different remote process command to handle extracting the gzip contents, with potentially different outcomes that would have to be handled. The approach here works exactly as it should for both a directory with a single file and a directory with arbitrary number of files and subdirectories (as in the --full flag case). In both cases the result is either resolving with the bundle buffer, or rejecting with some error, and then displaying some useful information.

I also feel it's a bit overkill to create a temporary directory, stream the entire file to disk, use a new stream that iterates directories, then create a tar stream out of that and then pipe that to the tessel.

Skipping the phase where this occurs means eliminating the opportunity to optimize the resulting bundle; in cases where browserify has only bundled a single file, the original file (minified) will always be a smaller payload because it will have the same contents as the browserify result, but will not have the module loading/dependency boilerplate. Also, the same argument I gave above applies to this phase.

Do you know of a way to simplify the operation that preserves the all of these semantics?

[rwaldron]

Thanks for the reviews!

[LinusU]

I just realized that the tar header needs to contain the length of the file :(

I thought that I was so smart thinking there was a way to just append the header ourselves but that won't work. I can totally see the benefit of having the same receiving mechanism for both a single file and a directory of files.

A thing that I would like slightly more, mostly because it skips the Sync part of fs-temp, would be to stream it to a file and skip the directory. Than roll the tar files using tar-stream instead of tar. I need to check my sources on this but I remember tar-stream being significantly faster than tar as well. The downside to this is that we would bring in both tar and tar-stream which feels a bit redundant.

const fs = require('fs')
const tar = require('tar-stream')
const temp = require('fs-temp')
const browserify = require('browserify')

const file = temp.createWriteStream()
const source = browserify(...).bundle()

source.pipe(file).on('finish', function () {
  const pack = tar.pack()
  const infile = fs.createReadStream(file.path)

  const header = { name: 'index.js', size: file.bytesWritten }
  const entry = pack.entry(header, function (err) {
    if (err) throw err

    pack.finialize()
  })

  infile.pipe(entry)

  pack.pipe(/* To the Tessel */)
})

I don't know if this feels like extra work, but on the other hand, getting the deployment to go as fast as possible would be very nice. It would be interesting to see if actually yields any difference though. It feels cleaner since it doesn't use Sync methods and reduces the amount of "unnecessary" work.