Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change replay percentage to total block size processed #1289

Closed
abitmore opened this issue Aug 27, 2018 · 12 comments

Comments

Projects
6 participants
@abitmore
Copy link
Member

commented Aug 27, 2018

User Story
As a node admin, during replaying, I want to know percentage of total size of blocks that have been processed, rather than count of blocks, so that I can estimate more accurately when it will complete.

CORE TEAM TASK LIST

  • Evaluate / Prioritize Feature Request
  • Refine User Stories / Requirements
  • Define Test Cases
    • Assigned: @cogutvalera
    • Estimated: 2 hours
    • Remitted: Week 39
  • Design / Develop Solution
  • Perform QA/Testing
  • Update Documentation
@oxarbitrage

This comment has been minimized.

Copy link
Member

commented Aug 27, 2018

i don't think block % to total will allow you to estimate time. count or % of operations will do better.

@abitmore

This comment has been minimized.

Copy link
Member Author

commented Aug 27, 2018

We don't have the number of total operations before replay. For this issue I mean we can use total_processes_block_size / total_block_size which should have been obtained from block log already when replaying.

@bangzi1001

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2018

Is that possible to add "Estimate Completion Time" beside the Percentage? If can, then node admin no need to estimate completion time manually.

@pmconrad

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2018

For this issue I mean we can use total_processes_block_size / total_block_size which should have been obtained from block log already when replaying.

I believe using the file size of the block log is an inaccurate measure of the total block size to apply. The reason is that the block log is only ever appended to. For example, when switching between forks, the blocks that are re-applied will be appended to the block log again. Similarly, the last 50 blocks of a replay are appended again.

I believe the error is small for a typical node, but can be significant for a node that is restarted often.

@abitmore

This comment has been minimized.

Copy link
Member Author

commented Aug 29, 2018

the last 50 blocks of a replay are appended again.

This doesn't sound ideal.

I believe the error is small for a typical node, but can be significant for a node that is restarted often.

I guess it won't be too significant, since we don't reapply too old blocks anyway. IMHO, even it's not accurate, it's still better for estimation than block counts.

@cogutvalera

This comment has been minimized.

Copy link
Member

commented Sep 10, 2018

@ryanRfox assign this issue to me please so I will continue my researchers after #946 issue.

Thanks !

@ryanRfox ryanRfox removed this from New -Awaiting Core Team Evaluation in Project Backlog Sep 10, 2018

@ryanRfox ryanRfox added this to In Development in Community Claims Sep 10, 2018

@ryanRfox

This comment has been minimized.

Copy link
Member

commented Sep 11, 2018

Assigned @cogutvalera and awaiting his estimate for @bitshares/core-dev review. Thanks

@cogutvalera

This comment has been minimized.

Copy link
Member

commented Sep 11, 2018

@ryanRfox my estimation for this issue is approximately 2 hours

@cogutvalera

This comment has been minimized.

Copy link
Member

commented Sep 19, 2018

PR is ready #1335

Thanks !

@cogutvalera cogutvalera referenced this issue Sep 21, 2018

Closed

Log console output during replay to file #985

5 of 9 tasks complete

@abitmore abitmore added this to To do in Feature release (201810) via automation Oct 1, 2018

@abitmore abitmore moved this from To do to In progress in Feature release (201810) Oct 1, 2018

pmconrad added a commit that referenced this issue Oct 2, 2018

Merge pull request #1335 from cogutvalera/issue_1289
Change replay percentage to total block size processed #1289
@pmconrad

This comment has been minimized.

Copy link
Contributor

commented Oct 2, 2018

Resolved in #1335

@pmconrad pmconrad closed this Oct 2, 2018

Feature release (201810) automation moved this from In progress to Done Oct 2, 2018

@ryanRfox

This comment has been minimized.

Copy link
Member

commented Oct 14, 2018

I understand this is a closed Issue, so perhaps I need to open a separate one. I feel the output could be improved. Currently is looks like this:

1944127ms th_a       db_management.cpp:103         reindex              ]    [by size: 2.37002290685776762%   821428243 of 34659084544]   [by num: 17.34458437871986192%   5430000 of 31306602]

Is it trivial to make it look something like:

1944127ms th_a       db_management.cpp:103         reindex              ]    [by size: 2.37%   821.42M of 34.66G]   [by blocks: 17.345%   5.43M of 31.31M]

Round to two decimal places and convert to K, M, G notation.

@abitmore

This comment has been minimized.

Copy link
Member Author

commented Oct 14, 2018

I more like accurate numbers, at least for blocks ;) I agree that it's better if the decimal places of percentages are fewer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.