Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hip30 balance migration with fix. #4502

Merged

Conversation

Frozen
Copy link
Contributor

@Frozen Frozen commented Sep 7, 2023

Issue

Test

Unit Test Coverage

Before:

<!-- copy/paste 'go test -cover' result in the directory you made change -->

After:

<!-- copy/paste 'go test -cover' result in the directory you made change -->

Test/Run Logs

Operational Checklist

  1. Does this PR introduce backward-incompatible changes to the on-disk data structure and/or the over-the-wire protocol?. (If no, skip to question 8.)

    YES|NO

  2. Describe the migration plan.. For each flag epoch, describe what changes take place at the flag epoch, the anticipated interactions between upgraded/non-upgraded nodes, and any special operational considerations for the migration.

  3. Describe how the plan was tested.

  4. How much minimum baking period after the last flag epoch should we allow on Pangaea before promotion onto mainnet?

  5. What are the planned flag epoch numbers and their ETAs on Pangaea?

  6. What are the planned flag epoch numbers and their ETAs on mainnet?

    Note that this must be enough to cover baking period on Pangaea.

  7. What should node operators know about this planned change?

  8. Does this PR introduce backward-incompatible changes NOT related to on-disk data structure and/or over-the-wire protocol? (If no, continue to question 11.)

    YES|NO

  9. Does the existing node.sh continue to work with this change?

  10. What should node operators know about this change?

  11. Does this PR introduce significant changes to the operational requirements of the node software, such as >20% increase in CPU, memory, and/or disk usage?

TODO

Comment on lines +34 to +35
localnetBlocksPerEpoch = 8
localnetBlocksPerEpochV2 = 8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reducing to 8 will impact the external external validator

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

16 ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on testing for external it had to be 64

@Frozen Frozen added the HIP-30 label Sep 11, 2023
@ONECasey ONECasey changed the base branch from dev to hip30/testing September 11, 2023 18:59
@Frozen Frozen force-pushed the hip30-balance-migration-fixed-stuck branch from e8d3fa4 to cb9b156 Compare September 12, 2023 01:04
@Frozen Frozen force-pushed the hip30-balance-migration-fixed-stuck branch from edc3ca1 to 073e16f Compare September 12, 2023 01:43
@ONECasey
Copy link
Contributor

@diego1q2w same with these now.

@diego1q2w
Copy link
Contributor

@diego1q2w same with these now.

done! i also deleted a few api methods I added for testing that are no longer necessary.

@ONECasey ONECasey marked this pull request as ready for review September 18, 2023 15:00
@ONECasey ONECasey merged commit c2bf8de into harmony-one:hip30/testing Sep 18, 2023
2 of 3 checks passed
Frozen added a commit to Frozen/harmony that referenced this pull request Sep 19, 2023
* flags: set up preimage flags

* hip30: set up preimage import, export, api

* preimages: re-generate them using CLI

* move from rpc to core

* migrate balance uring epoch T - 1

highly untested code. also missing is the ability to generate a
pre-migration report for future verification.

* test account migration in localnet

* enable preimages on the whitelist

* add the generate method

* fix cropping log

* fix cropping log

* cropping startpoint when bigger than endpoint

* add support for the rpcblocknumer type

* enable import api

* Fixed stuck.

* Additional logs.

* Cleanup.

* Rebased on harmony-one:hip30/testing.

* Removed code duplicate.

* Fixed stuck.

* IsOneEpochBeforeHIP30 for only 1 epoch.

---------

Co-authored-by: MaxMustermann2 <82761650+MaxMustermann2@users.noreply.github.com>
Co-authored-by: Nita Neou (Soph) <soph@harmony.one>
Co-authored-by: Diego Nava <diego.nava77@hotmail.com>
Co-authored-by: Diego Nava <8563843+diego1q2w@users.noreply.github.com>
sophoah added a commit that referenced this pull request Oct 18, 2023
* Fixed debug run for mac. (#4484)
* Fix debug run for mac.
* Next validator in view change.  (#4492)
* NthNextValidatorHmy
* Fixed func usage.
* Additional checks.
* HIP-30 Boilerplate (#4495)
* HIP-30: sharding configuration boilerplate
* update test
* update comment
---------

Co-authored-by: Nita Neou (Soph) <soph@harmony.one>
* HIP-30: minimum validator commission of 7% (#4496)
* HIP-30: sharding configuration boilerplate
* update comments
* goimports
* HIP-30: minimum validator commission of 7%

Based on #4495, which must be merged before this PR. This PR should be
rebased with dev after #4495 is merged to retain atomicity of changes by
pull request.
* goimports
* update test
* update test

---------

Co-authored-by: Casey Gardiner <117784577+ONECasey@users.noreply.github.com>

* HIP-30: Emission split (#4497)
* HIP-30: sharding configuration boilerplate
* update comments
* goimports
* HIP-30: minimum validator commission of 7%

Based on #4495, which must be merged before this PR. This PR should be
rebased with dev after #4495 is merged to retain atomicity of changes by
pull request.

* goimports
* HIP-30: Emission split implementation

Note that the allocated split of the emission goes directly to the
recipient (and not via the Reward). This is because rewards are indexed
by validator and not by delegator, and the recipient may/may not have
any delegations which we can reward. Even if one was guaranteed to
exist, it would mess up the math of the validator.

* set up mainnet recipient of emission split
* HIP-30: Emission split addresses for non mainnet
* update test
* Update mainnet.go

---------

Co-authored-by: Casey Gardiner <117784577+ONECasey@users.noreply.github.com>

* HIP-30: Set up pre-image generation, recording, export and import (#4494)
* flags: set up preimage flags
* hip30: set up preimage import, export, api
* save pre-images by default
* add pre images api
* goimports
* commit rpc preimages file
* preimages: re-generate them using CLI
* add metrics and numbers for pre-images
* automate generation after import
* move from rpc to core
* add back core/preimages.go file
* export prometheus metric when no error importing preimage
* add preimages flags to rootflags

---------

Co-authored-by: Nita Neou (Soph) <soph@harmony.one>

* HIP-30: Shard reduction (#4498)
* HIP-30: sharding configuration boilerplate
* update comments
* goimports
* HIP-30: minimum validator commission of 7%

Based on #4495, which must be merged before this PR. This PR should be
rebased with dev after #4495 is merged to retain atomicity of changes by
pull request.

* goimports
* HIP-30: Emission split implementation

Note that the allocated split of the emission goes directly to the
recipient (and not via the Reward). This is because rewards are indexed
by validator and not by delegator, and the recipient may/may not have
any delegations which we can reward. Even if one was guaranteed to
exist, it would mess up the math of the validator.

* set up mainnet recipient of emission split
* HIP-30: Emission split addresses for non mainnet
* HIP-30: deactivate shard 2 and 3 validators
* update test
* shard reduction: update block reward

---------

Co-authored-by: Casey Gardiner <117784577+ONECasey@users.noreply.github.com>

* Fix for index. (#4504)
* Small improvements. (#4477)
* HIP-30: Balance migration (#4499)
* flags: set up preimage flags
* hip30: set up preimage import, export, api
* save pre-images by default
* add pre images api
* goimports
* commit rpc preimages file
* preimages: re-generate them using CLI
* add metrics and numbers for pre-images
* automate generation after import
* move from rpc to core
* add back core/preimages.go file
* HIP-30: sharding configuration boilerplate
* update comments
* HIP-30: minimum validator commission of 7%

Based on #4495, which must be merged before this PR. This PR should be
rebased with dev after #4495 is merged to retain atomicity of changes by
pull request.

* goimports
* HIP-30: Emission split implementation

Note that the allocated split of the emission goes directly to the
recipient (and not via the Reward). This is because rewards are indexed
by validator and not by delegator, and the recipient may/may not have
any delegations which we can reward. Even if one was guaranteed to
exist, it would mess up the math of the validator.

* set up mainnet recipient of emission split
* HIP-30: Emission split addresses for non mainnet
* HIP-30: deactivate shard 2 and 3 validators
* goimports
* update test
* migrate balance uring epoch T - 1

highly untested code. also missing is the ability to generate a
pre-migration report for future verification.

* update test
* export prometheus metric when no error importing preimage
* add comment
* test account migration in localnet
* add preimages flags to rootflags
* enable preimages on the whitelist
* add the generate method
* fix cropping log
* cropping startpoint when bigger than endpoint
* add support for the rpcblocknumer type
* enable import api
* use earlies block
* fix error catching
* make end optional for the comand line
* fix cropping logic
* improve error when apply message fails
* add balance on the error
* fix importing
* remove unused imports

---------

Co-authored-by: Nita Neou (Soph) <soph@harmony.one>
Co-authored-by: Soph <35721420+sophoah@users.noreply.github.com>
Co-authored-by: Diego Nava <diego.nava77@hotmail.com>
Co-authored-by: Diego Nava <8563843+diego1q2w@users.noreply.github.com>

* Hip30 balance migration with fix. (#4502)
* flags: set up preimage flags
* hip30: set up preimage import, export, api
* preimages: re-generate them using CLI
* move from rpc to core
* migrate balance uring epoch T - 1
* test account migration in localnet
* enable preimages on the whitelist
* add the generate method
* fix cropping log
* cropping startpoint when bigger than endpoint
* add support for the rpcblocknumer type
* enable import api
* Fixed stuck.
* Additional logs.
* Rebased on harmony-one:hip30/testing.
* Removed code duplicate.
* Fixed stuck.
* IsOneEpochBeforeHIP30 for only 1 epoch.

---------

Co-authored-by: MaxMustermann2 <82761650+MaxMustermann2@users.noreply.github.com>
Co-authored-by: Nita Neou (Soph) <soph@harmony.one>
Co-authored-by: Diego Nava <diego.nava77@hotmail.com>
Co-authored-by: Diego Nava <8563843+diego1q2w@users.noreply.github.com>

* remove double import
* rename variable
* remove unused fmt
* Fixed imports. (#4507)
* Block gas 30m. (#4501)
* BlockGas30M renamed to BlockGas30MEpoch.
* Removed redundant code.
* Hip30 : localnet account migration fix (#4508)
* flags: set up preimage flags
* hip30: set up preimage import, export, api
* save pre-images by default
* add pre images api
* goimports
* commit rpc preimages file
* preimages: re-generate them using CLI
* add metrics and numbers for pre-images
* automate generation after import
* add back core/preimages.go file
* HIP-30: sharding configuration boilerplate
* update comments
* goimports

* HIP-30: minimum validator commission of 7%

Based on #4495, which must be merged before this PR. This PR should be
rebased with dev after #4495 is merged to retain atomicity of changes by
pull request.

* goimports

* HIP-30: Emission split implementation

Note that the allocated split of the emission goes directly to the
recipient (and not via the Reward). This is because rewards are indexed
by validator and not by delegator, and the recipient may/may not have
any delegations which we can reward. Even if one was guaranteed to
exist, it would mess up the math of the validator.

* set up mainnet recipient of emission split
* HIP-30: Emission split addresses for non mainnet
* HIP-30: deactivate shard 2 and 3 validators
* goimports
* update test
* goimports
* migrate balance uring epoch T - 1
highly untested code. also missing is the ability to generate a
pre-migration report for future verification.
* update test
* export prometheus metric when no error importing preimage
* test account migration in localnet
* add preimages flags to rootflags
* enable preimages on the whitelist
* add the generate method
* fix cropping log
* cropping startpoint when bigger than endpoint
* add support for the rpcblocknumer type
* enable import api
* use earlies block
* debug logs
* fix error catching
* fix error catching
* make end optional for the comand line
* fix cropping logic
* improve error when apply message fails
* add balance on the error
* fix importing
* remove unused imports
* create preimage for genesis block
* fix consensus with No Migration Possible
* use correct header for migration
* process all tx in all block for non shard 0

---------

Co-authored-by: MaxMustermann2 <82761650+MaxMustermann2@users.noreply.github.com>
Co-authored-by: Diego Nava <diego.nava77@hotmail.com>
Co-authored-by: Diego Nava <8563843+diego1q2w@users.noreply.github.com>

* create the sate as the insertchain does
* roll back changes
* use the updated state in case there is one
* use the updated state in case there is one
* add testing fmt
* fix getReceipts rpc issue (#4511)
* pass the correct config
* Fixes.
* reduce the block number count
* add verify preimages rpc method
* write preimages on process
* commit preimages
* commit preimages
* verify root hashes after commit
* send metrics on node start
* send the verified preimages
* correct the starting block
* register the verified address
* flush the db every export and verify
* add shard label
* aggregate the recovery multisig reward (#4514)

* 1) Removed unused worker (#4512)
2) Proper error checking
3) Tests for gas 30m

* Improvements of streamsync to deploy on mainnet (#4493)
* add faultRecoveryThreshold to reset stream failures
* increase MaxStreamFailures to let stream be longer in the list
* set Concurrency to 2 for devnet to be same as MinStreams, otherwise it will rewrite MinStreams
* stream sync loop checks for ErrNotEnoughStreamsand waits for enough streams in case there are not enough connected streams in list
* fix fault recovery issue
* improve checkPrerequisites to be able to continue with minimum streams
* refactor fixValues function, put priority on MinStreams rather than Concurrency
* drop remote peer if sending empty blocks array
* goimports to fix build issue
* fix getReceipts array assignments
* fix getReceipts and add tests for it
* fix duplicate function def
* reset devnet and set 30M epoch for all network except mainnet/testnet (#4517)
* enable on devnet hip28 and hip30 together at hip30
* enable 30M epoch for all except mainnet/testnet, update devnet for restart
* remove unused var
* update partner/devnet feature activation
* Remove old devnet/partner instance config
* reduce the epoch time for devnet to 30 min (#4522)
* add GetNodeData tests for stream client, increase nodes and receipts cap (#4519)
* add tests for GetNodeData in stream protocol
* increase client cap for nodes and receipts requests
* use new(big.Int) so we don't modify the epoch value (#4523)
* add hip30 testing for devnet/partner network (#4525)
* enable hip30 epoch for testnet (#4526)
* enable hip30 epoch for testnet
* fix date comment
* set blockgas30M epoch
* enable hip30 and gas30m epoch for mainnet (#4528)
* fix preimage import bugs (#4529)
* fix preimage import bugs
* Fixed lru cache size. (#4535)
* fix decryptRaw issue for nil/empty data (#4532)
* update deprecated ioutil, improve local accounts (#4527)
* make peer connected/disconnected debug log level (#4537)
* Revert improvements. (#4520)
* Updated go lib p2p deps. (#4538)
* Flush data. (#4536)
* Rotation fix and update. (#4516)

---------

Co-authored-by: Konstantin <355847+Frozen@users.noreply.github.com>
Co-authored-by: Max <82761650+MaxMustermann2@users.noreply.github.com>
Co-authored-by: Nita Neou (Soph) <soph@harmony.one>
Co-authored-by: Soph <35721420+sophoah@users.noreply.github.com>
Co-authored-by: Diego Nava <diego.nava77@hotmail.com>
Co-authored-by: Diego Nava <8563843+diego1q2w@users.noreply.github.com>
Co-authored-by: Gheis Mohammadi <Gheis.Mohammadi@gmail.com>
Co-authored-by: “GheisMohammadi” <36589218+GheisMohammadi@users.noreply.github.com>
Co-authored-by: Adam Androulidakis <37982984+adsorptionenthalpy@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants