New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement fast in-memory database with JSON storage #1963

Open
faustbrian opened this Issue Jan 11, 2019 · 2 comments

Comments

Projects
None yet
3 participants
@faustbrian
Copy link
Collaborator

faustbrian commented Jan 11, 2019

Things like the transaction pool could greatly benefit from a fast in-memory storage that could fallback to store everything in a JSON file for which we then could use fast.js to do very fast queries on objects rather then a database as the pool itself doesn't contain any relational data.

Something like https://github.com/typicode/lowdb but a faster implementation because lowdb uses lodash which is rather fat and slow.

@faustbrian faustbrian self-assigned this Jan 11, 2019

@faustbrian faustbrian changed the title Implement fast in-memory storage with JSON storage Implement fast in-memory database with JSON storage Jan 11, 2019

@vasild

This comment has been minimized.

Copy link
Contributor

vasild commented Jan 11, 2019

Currently we store the transactions in the transaction pool in memory in a quite efficient way. It is more involved than the following, but to illustrate, we use:

byId = {}
byId[tx1id] = tx1
byId[tx2id] = tx2
...
// and to retrieve:
byId[someId]

This has O(1) complexity. I can't imagine something more efficient without implementing it in native code (C++).

Our on-disk storage (which is engaged occasionally) is ACID compliant. Or in lame's terms, it is crash safe.

That said, there is always room for improvement (for example https://github.com/ArkEcosystem/core/blob/develop/packages/core-transaction-pool/src/mem.ts#L156), but any changes would better be well justified and tested in order to avoid performance regressions.

@fix

This comment has been minimized.

Copy link
Member

fix commented Jan 15, 2019

We could store faster in-memory with reduced memory footprint by using aip11 and having getter/setter directly in the aip11 buffer.

For instance:

tx = new AIP11Transaction()
tx.setType(0)
tx.setRecipient(...)

The functions handling directly the data in buffer

This way we could store buffer in DB as well and

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment