Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ledger: generic kv trackers backend implementation #5488

Merged
merged 3 commits into from Jul 25, 2023

Conversation

icorderi
Copy link
Contributor

Summary

We are breaking down #5212 in several smaller PRs to make it more digestible.

This is PRs "numero 3" on the KV epic.

Content:

  • A generic KV implementation with behavioral parity with the SQLite implementation
  • A testsuite to run tests across multiple db backends

Test Plan

Adding cross-backend db testing.

@codecov
Copy link

codecov bot commented Jun 22, 2023

Codecov Report

Merging #5488 (d30b4b7) into master (b06de44) will decrease coverage by 0.54%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #5488      +/-   ##
==========================================
- Coverage   55.41%   54.87%   -0.54%     
==========================================
  Files         446      460      +14     
  Lines       63289    64231     +942     
==========================================
+ Hits        35069    35245     +176     
- Misses      25826    26610     +784     
+ Partials     2394     2376      -18     
Impacted Files Coverage Δ
...r/store/trackerdb/generickv/accounts_ext_reader.go 0.00% <0.00%> (ø)
...r/store/trackerdb/generickv/accounts_ext_writer.go 0.00% <0.00%> (ø)
...edger/store/trackerdb/generickv/accounts_reader.go 0.00% <0.00%> (ø)
...edger/store/trackerdb/generickv/accounts_writer.go 0.00% <0.00%> (ø)
ledger/store/trackerdb/generickv/catchpoint.go 0.00% <0.00%> (ø)
ledger/store/trackerdb/generickv/init_accounts.go 0.00% <0.00%> (ø)
ledger/store/trackerdb/generickv/migrations.go 0.00% <0.00%> (ø)
...store/trackerdb/generickv/onlineaccounts_reader.go 0.00% <0.00%> (ø)
...store/trackerdb/generickv/onlineaccounts_writer.go 0.00% <0.00%> (ø)
ledger/store/trackerdb/generickv/reader.go 0.00% <0.00%> (ø)
... and 4 more

... and 36 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@algorandskiy algorandskiy changed the title feat: generic kv implementation ledger: generic kv trackers backend implementation Jun 23, 2023
ledger/store/trackerdb/generickv/schema.go Show resolved Hide resolved
ledger/store/trackerdb/generickv/schema.go Show resolved Hide resolved
ledger/store/trackerdb/generickv/schema.go Show resolved Hide resolved
return ret
}

// resourceKey: 4-byte prefix + 32-byte address + 8-byte big-endian uint64
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these 32 bytes per resource looks as a huge waste

}

// TODO: lifted from sql.go, we might want to refactor it
func keyPrefixIntervalPreprocessing(prefix []byte) ([]byte, []byte) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it has to be a shared function, not a copy

}

// RunMigrations runs the migrations on the store up to the target version.
func RunMigrations(ctx context.Context, db dbForMigrations, params trackerdb.Params, targetVersion int32) (mgr trackerdb.InitParams, err error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider rewriting by using utils/db's migration/init code - see and staterproof as an example

require.Equal(t, trackerdb.ErrNotFound, err)
}

func CustomTestOnlineAccountsDelete(t *customT) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a bigger test TestOnlineAccountsDeletion and I do think the new implementation have to at least pass it but ideally extend it to 3+ accounts and more rounds.

Copy link
Contributor

@algorandskiy algorandskiy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good but there are couple primary concerns:

  1. new OnlineAccountsDelete needs more test scenarios. The original SQL code was written for continuous address sequence like AAAABBBCCCC and not ABCABAACC.
  2. keys creation has unneeded allocation overhead, need to fix.

Copy link
Contributor

@AlgoAxel AlgoAxel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I have used/been exposed to this code previously, I:

  • Read over existing comments
  • Skimmed over all
  • Read closely sections I was not already familiar with (stateproof)

As I already use this code (in a slightly different format), I am comfortable approving.

@algorandskiy
Copy link
Contributor

Going to merge after the 3.16.3 cut and have follow up ticket for tasks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants