Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Implement fast in-memory database with JSON storage #1963
Things like the transaction pool could greatly benefit from a fast in-memory storage that could fallback to store everything in a JSON file for which we then could use fast.js to do very fast queries on objects rather then a database as the pool itself doesn't contain any relational data.
Something like https://github.com/typicode/lowdb but a faster implementation because lowdb uses lodash which is rather fat and slow.
Currently we store the transactions in the transaction pool in memory in a quite efficient way. It is more involved than the following, but to illustrate, we use:
Our on-disk storage (which is engaged occasionally) is ACID compliant. Or in lame's terms, it is crash safe.
That said, there is always room for improvement (for example https://github.com/ArkEcosystem/core/blob/develop/packages/core-transaction-pool/src/mem.ts#L156), but any changes would better be well justified and tested in order to avoid performance regressions.
We could store faster in-memory with reduced memory footprint by using aip11 and having getter/setter directly in the aip11 buffer.
The functions handling directly the data in buffer
This way we could store buffer in DB as well and