Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move inputs from ETS to disk #684

Merged

Conversation

bchamagne
Copy link
Member

@bchamagne bchamagne commented Nov 14, 2022

Description

Once a UTXO is spent, we move it to disk to reduce memory footprint.

Fixes #624

Type of change

  • New feature

How Has This Been Tested?

This is tested in the integration tests.
I did not do any BEFORE/AFTER metrics, should I do some benchmark?

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

Bastien CHAMAGNE added 5 commits November 14, 2022 16:29
there is one file per address
there is one file per address
we need the prefix to avoid one ledger overriding the content of the other
@bchamagne
Copy link
Member Author

IMPORTANT NOTE: we loose millisecond precision due to serialization using DateTime.to_unix/1

@samuelmanzanera samuelmanzanera added feature New feature request DB Involve database labels Nov 14, 2022
Copy link
Member

@samuelmanzanera samuelmanzanera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, I wrote you some comments to improve it in terms of performances and maintainability.

lib/archethic/account/mem_tables/token_ledger.ex Outdated Show resolved Hide resolved
lib/archethic/account/mem_tables/token_ledger.ex Outdated Show resolved Hide resolved
lib/archethic/db/embedded_impl/inputs.ex Outdated Show resolved Hide resolved
@samuelmanzanera
Copy link
Member

IMPORTANT NOTE: we loose millisecond precision due to serialization using DateTime.to_unix/1

If we would like to preserve it, we can use DateTime.to_unix(date, :millisecond)

@samuelmanzanera
Copy link
Member

After restart of the node, the inputs are duplicated with different state

image

To reproduce it, start a node, do 2 faucet requests, stop the node, and restart it. Then go to the previous addresses generated, you will see the double inputs, one spent and one unspent, while they are the same.

@bchamagne
Copy link
Member Author

After restart of the node, the inputs are duplicated with different state

image

To reproduce it, start a node, do 2 faucet requests, stop the node, and restart it. Then go to the previous addresses generated, you will see the double inputs, one spent and one unspent, while they are the same.

This happened because the transactions history was processed in inverse-chronological order. Resulting in spend_all_transactions(previous_address) not doing anything because the previous was not existing yet in memloader. To fix the issue, we reversed the processing of transactions history to be chronological.

lib/archethic/account/mem_tables/token_ledger.ex Outdated Show resolved Hide resolved
lib/archethic/account/mem_tables/uco_ledger.ex Outdated Show resolved Hide resolved
lib/archethic/db/embedded_impl/inputs_reader.ex Outdated Show resolved Hide resolved
lib/archethic/db/embedded_impl/inputs_reader.ex Outdated Show resolved Hide resolved
UCO are either all spent or all unspent, so no need to read from disk
if the ETS table contains spent uco
Tokens are either all spent or all unspent, so no need to read from disk
if the ETS table contains spent tokens
@samuelmanzanera samuelmanzanera merged commit 123a7b3 into archethic-foundation:develop Nov 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DB Involve database feature New feature request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Purge Account UTXO table once an input is spent
3 participants