New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port away from event based stream #24

Closed
thejmazz opened this Issue Mar 15, 2016 · 1 comment

Comments

2 participants
@thejmazz
Member

thejmazz commented Mar 15, 2016

No way to tell when an event based stream ends. So essentially need to port into a transform stream that takes obo and emits parsed [Term]s. Perhaps throws you header with .on('header') or something.

Best way to do this?

What happens now:

line events emitted, then lines are added into string buffers using some flags and buffers. Checking what the current line is, emit header once, and emit stanza which is piped through stanzaParser, filtered, ndjsonified, etc.

Events was a nice way to separate concerns. So it was good for prototyping. But the output stream not having a defined end is not good. So some way to either force in the end event, or move out of using events. A little bit stumped, so any suggestions would be awesome. Perhaps I should move back to "vanilla" streams instead of wrapping with highland, and build buffers based on current character and flags. If anyone knows any good examples of streaming parsers, that would be good to see too.

@thejmazz thejmazz self-assigned this Mar 15, 2016

@thejmazz thejmazz added this to the 0.2.0 milestone Mar 15, 2016

@bmpvieira

This comment has been minimized.

Member

bmpvieira commented Mar 18, 2016

@thejmazz did you have a look at https://github.com/bionode/bionode-fasta?
or https://github.com/mafintosh/csv-parser and https://github.com/maxogden/csv-write-stream @mafintosh and @maxogden are stream experts, maybe ask them on #dat IRC or http://gitter.im/datproject/discussions

However, maybe the problem is the wrapping with highlandjs, I remember having some issues with it and ended up not using it. Instead, I made a quick and dirty module for some stream operations https://github.com/bmpvieira/tool-stream

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment