Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delegation backend: new storage structure #9712

Merged
merged 11 commits into from
Apr 6, 2022

Conversation

georgeee
Copy link
Member

@georgeee georgeee commented Oct 27, 2021

Upon request from OnTab, a few changes were introduced to Delegation backend. A different directory structure was agreed upon to ease implementation of Ontab's software.

Explain your changes:

  • User submissions are saved by path <submited_date>/<submitted_time>-<submitter>.json
  • Fields created_at, submitter and block_hash are added to the JSON file
  • (Minor) encoding of submitter field is fixed to match Mina's standard Pk encoding (prefix was added)
  • Migration tool to migrate existing submissions to the new format was implemented

Explain how you tested your changes:

  • Existing tests were tested accordingly

Checklist:

  • Modified the current draft of release notes with details on what is completed or incomplete within this project
  • Document code purpose, how to use it
    • Mention expected invariants, implicit constraints
  • Tests were added for the new behavior
    • Document test purpose, significance of failures
    • Test names should reflect their purpose
  • All tests pass (CI will check this if you didn't)
  • Serialized types are in stable-versioned modules
  • Does this close issues?

@ghost-not-in-the-shell ghost-not-in-the-shell added the ci-build-me Add this label to trigger a circle+buildkite build for this branch label Oct 27, 2021
@georgeee georgeee force-pushed the georgeee/delegation-backend-new-structure branch from ed1d9bf to 9f0993d Compare November 15, 2021 10:08
@georgeee georgeee force-pushed the georgeee/delegation-backend-new-structure branch from 9f0993d to b5e2326 Compare November 22, 2021 11:04
@georgeee georgeee force-pushed the georgeee/delegation-backend-new-structure branch from 5325131 to 23ddde4 Compare December 15, 2021 17:34
@georgeee georgeee force-pushed the georgeee/delegation-backend-new-structure branch from a85f884 to a194967 Compare February 7, 2022 13:09
@georgeee georgeee requested a review from a team as a code owner February 7, 2022 16:39
@georgeee georgeee force-pushed the georgeee/delegation-backend-new-structure branch from df34a13 to a194967 Compare February 7, 2022 16:40
@georgeee georgeee force-pushed the georgeee/delegation-backend-new-structure branch from a194967 to ceb1421 Compare March 2, 2022 21:49
@georgeee georgeee force-pushed the georgeee/delegation-backend-new-structure branch from 2ec96ce to 3ea6f55 Compare April 5, 2022 10:50
* Store files at new paths
* Implement a migration tool
Encode public key in correct way
Problem: migration tool is slow.

Solution: try to improve the speed by allowing
up to 100 concurrent migrations.
Problem: blocks are being overwritten again and again.
This puts a substantial burden upon backend.

Solution: do not overwrite existing objects on the bucket.
In case of a block, that would result into block being uploaded
only once.
@georgeee georgeee force-pushed the georgeee/delegation-backend-new-structure branch from 3ea6f55 to 9b806a3 Compare April 6, 2022 12:03

const NETWORK_ID = 1 // mainnet
const PK_LENGTH = 33 // one field element (32B) + 1 bit (encoded as full byte)
const SIG_LENGTH = 64 // one field element (32B) and one scalar (32B)

// we use state hash code here, although it's not state hash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use a different code here? The idea behind having separate ones was so that they were easy to distinguish / hard to confuse.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's complicated: it would require another round of data migration and some more action on Ontab's side. We might do it if we'll be doing such a pass for other reasons in future though.

@mrmr1993 mrmr1993 merged commit b593af8 into compatible Apr 6, 2022
@mrmr1993 mrmr1993 deleted the georgeee/delegation-backend-new-structure branch April 6, 2022 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-build-me Add this label to trigger a circle+buildkite build for this branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants