Skip to content
This repository has been archived by the owner on Sep 12, 2018. It is now read-only.

[tx] Speed up transaction parser #436

Closed
bgrins opened this issue Apr 27, 2017 · 2 comments
Closed

[tx] Speed up transaction parser #436

bgrins opened this issue Apr 27, 2017 · 2 comments
Assignees
Labels

Comments

@bgrins
Copy link
Member

bgrins commented Apr 27, 2017

When importing a map structure via transactions that contains around 1000 lines I'm getting around 3 seconds on the parse

mentat-profile

@bgrins
Copy link
Member Author

bgrins commented Apr 27, 2017

From slack:

Nice thing is it's trivially parallel
Mmap the file, segment lines, pass offsets to parser, preprocess in parallel, queue for final write

@rnewman
Copy link
Collaborator

rnewman commented Apr 27, 2017

Though I should add: it's theoretically possible for us to parse this data with zero string copying, and it should be way faster than this, even serially. It might be a lot of work to get there, but it seems wiser to do some of it than to immediately throw cores at the problem!

@ncalexan ncalexan changed the title Speed up transaction parser [tx] Speed up transaction parser May 5, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

4 participants