Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block production summary in 100 node #1876

Merged
merged 55 commits into from Feb 24, 2020
Merged

Conversation

@ailisp
Copy link
Member

ailisp commented Dec 18, 2019

image

Output is roughly as above. Each line in each txt above is in format:
block_hash block_height approvals is_chunk_included_all_in_current_height chunk_included_height

@ailisp ailisp requested a review from SkidanovAlex Dec 18, 2019
@codecov

This comment has been minimized.

Copy link

codecov bot commented Dec 18, 2019

Codecov Report

❗️ No coverage uploaded for pull request base (staging@c94c555). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##             staging    #1876   +/-   ##
==========================================
  Coverage           ?   86.76%           
==========================================
  Files              ?      176           
  Lines              ?    34153           
  Branches           ?        0           
==========================================
  Hits               ?    29634           
  Misses             ?     4519           
  Partials           ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c94c555...1c15651. Read the comment docs.

@ailisp

This comment has been minimized.

Copy link
Member Author

ailisp commented Dec 22, 2019

Test result so far:

block production time 10s, fork after about 4 hours
when check after 6 hours, all node agree validator is ['node17']

block production time 100s: not fork after 12 hours. cpu 100% though

block production time 55s:
doesn't fork after 8 hours, also 100% cpu

block production time 30s:
fork after 6 hours :(

block production time 40s:
fork after 8 hours (probably i should retest 50s or 55s or 60s one, giving it a longer time to see if it forks)

@SkidanovAlex

@bowenwang1996

This comment has been minimized.

Copy link
Member

bowenwang1996 commented Dec 22, 2019

Maybe we are doomed. Which one do you refer to when you talk about block production time? Is it min_block_production_delay, max_block_production_delay, or max_block_wait_delay?

@ailisp

This comment has been minimized.

Copy link
Member Author

ailisp commented Dec 22, 2019

@bowenwang1996

This comment has been minimized.

Copy link
Member

bowenwang1996 commented Dec 22, 2019

So min_block_production_delay is 10s, max_block_production_delay is 20s and max_block_wait_delay is 60s?

@ailisp

This comment has been minimized.

Copy link
Member Author

ailisp commented Dec 22, 2019

@bowenwang1996

This comment has been minimized.

Copy link
Member

bowenwang1996 commented Dec 22, 2019

That's unfortunate. Is memory usage normal?

@ailisp

This comment has been minimized.

Copy link
Member Author

ailisp commented Dec 22, 2019

@ailisp

This comment has been minimized.

Copy link
Member Author

ailisp commented Dec 23, 2019

@bowenwang1996

This comment has been minimized.

Copy link
Member

bowenwang1996 commented Dec 24, 2019

yeah there seems to be some new memory leak issue. Will investigate separately.

@ailisp ailisp requested a review from bowenwang1996 as a code owner Jan 9, 2020
ailisp added 3 commits Feb 4, 2020
ailisp added 3 commits Feb 10, 2020
ailisp added 2 commits Feb 22, 2020
@ailisp

This comment has been minimized.

Copy link
Member Author

ailisp commented Feb 22, 2020

@bowenwang1996 @mfornet
I removed all rust logging part and plan to merge this PR so we can run newer tests from new branches from staging.

@ailisp ailisp requested review from SkidanovAlex and mfornet Feb 22, 2020
* docs: Improve README for one hundred nodes

* test: Enable diagnostic in one hundred nodes test

* doc: Add more instructions for the 100 nodes test
@ailisp ailisp requested review from evgenykuzyakov and nearmax as code owners Feb 24, 2020
Copy link
Member

bowenwang1996 left a comment

I think we should clean up the analysis scripts a bit.

node_blocks = [0] * 100
block_by = {}
for i in range(100):
with open(f'pytest-node-{i}.txt') as f:

This comment has been minimized.

Copy link
@bowenwang1996

bowenwang1996 Feb 24, 2020

Member

can we clean up this a bit? It looks outdated.

This comment has been minimized.

Copy link
@ailisp

ailisp Feb 24, 2020

Author Member

Sure. The problem is I don't exactly remember which was outdated 😂 but I guess you and Marcelo has already keep one or two useful, up to date scripts, So I can simply delete all analyze scripts here?

This comment has been minimized.

Copy link
@bowenwang1996

bowenwang1996 Feb 24, 2020

Member

That's also fine I guess

This comment has been minimized.

Copy link
@ailisp

ailisp Feb 24, 2020

Author Member

Deleted. If you feel some of your analyze script help, please add to this branch

pytest/tests/stress/hundred_nodes/collect_logs.py Outdated Show resolved Hide resolved
ailisp added 2 commits Feb 24, 2020
@ailisp ailisp added the automerge label Feb 24, 2020
@nearprotocol-bulldozer nearprotocol-bulldozer bot merged commit 60f2da2 into staging Feb 24, 2020
2 checks passed
2 checks passed
gitlab-ci
Details
reallyfastci
Details
@nearprotocol-bulldozer nearprotocol-bulldozer bot deleted the block-prod-100 branch Feb 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.