New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate nodestream #46

thejmazz opened this Issue Apr 9, 2017 · 2 comments


2 participants

thejmazz commented Apr 9, 2017

nodestream - Storage-agnostic streaming library for binary data transfers

This is beneficial because it can move the "tee to a file" work off of us perhaps, but more importantly, agnostically provides transfer to various cloud services. For now, can just use it for local filesystem.

Also need to consider how tasks produce output. If it is a program that takes an outputFile as params for example, do we need to create a readstream on that file as it is created to produce an outgoing stream of it?

Not sure if enhancement or feature, it is a bit of both - not entirely necessary for MVP - but very useful to have.


This comment has been minimized.


bmpvieira commented Apr 10, 2017

Thanks @thejmazz, seems like an interesting project worth watching! Yes, we could use it to add more features (unified cloud storage) or enhance existing code (local file storage, transforms). So if we move forward I think we'll need more specific issues, but for now I think this is just a discussion.
Maybe some of that discussion can happen on the Gitter channel, but at a first glance, these are the question I have:

  • How easy would it be to combine nodestream with other modules that are just regular Streams?
  • The examples shown seem to use promises, would that be a problem (e.g., make our code less Streamable)?

This comment has been minimized.


thejmazz commented Apr 10, 2017

Ah, did not realize that. If this is used just for uploading/downloading files before/after tasks, having a Promise API is not a problem (it can be like a stream that emits one chunk and finishes), but the underlying stream is there (with transforms) - which would be nice to have access to, and we can handle the finish ourselves (though also looks like there is some transformer based results.

@bmpvieira bmpvieira added the mozsprint label Apr 24, 2017

@bmpvieira bmpvieira referenced this issue May 24, 2017


Mozilla's Global Sprint (June 1st and 2nd 2017) #44

7 of 17 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment