Several bug fixes + performance improvements (HOLD FOR NOW) #9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
November 25, 2014 11:09
…property to help speed up tokenization for huge files where there is distinct breaks between tokens (e.g. CVS file).
|
Hello, What I could also do is merge your pull requests as well as #11 into a 1.2 branch and publish that one to npm. |
|
Thanks @Floby. Let's hold off on the merge until I can be certain is solves my scenario (you can find it here: https://github.com/joshjung/slim-to-jade. The fixes in this merge request took care of issue #8 but I still found other issues as I continued, primarily with how the chunks are being broken up and how process.nextTick was being called. It made me wonder if either I was not using the tokenizer properly or if some underlying structure needs to change. |
joshjung
changed the title
added 9 commits
December 3, 2014 11:34
…d the split token is encountered.
…get the 'end' event to fire and it just won't.
…g issues when trying to listen to when tokenizing was done.
…he split occurs so that there is some indicator the split is there.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix for #8.
Adding unit test for special scenario that was failing for me. Also adding options.split to improve performance on predictable streams of data with boundaries across which tokens will not ever pass (e.g. CSV file)