Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IBD: max blocks in transit per peer (cached PoW) #135

Merged
merged 2 commits into from
May 15, 2020
Merged

IBD: max blocks in transit per peer (cached PoW) #135

merged 2 commits into from
May 15, 2020

Conversation

decryp2kanon
Copy link
Contributor

@decryp2kanon decryp2kanon commented May 12, 2020 •
edited
Loading

#22 (comment)

MicroBitcoinOrg/MicroBitcoin#19

Background

#22

solardiz commented

Also relevant is ResistancePlatform/resistance-core@4efff30 which sets MAX_BLOCKS_IN_TRANSIT_PER_PEER to be same as MAX_HEADERS_RESULTS. Without this change, at least in Resistance the block headers download would get unnecessarily far ahead of the full blocks download, resulting in more work lost and redone in case the initial blocks download is interrupted and continued. The work loss is because we do not yet store cached PoWs on disk.

Don't change MAX_HEADERS_RESULTS. Just set MAX_BLOCKS_IN_TRANSIT_PER_PEER to some value from e.g. 200 to 2000 - experiment with this.

@decryp2kanon decryp2kanon changed the title IBD: same as MAX_HEADERS_RESULTS to match buffersize IBD: MAX_BLOCKS_IN_TRANSIT_PER_PEER = MAX_HEADERS_RESULTS May 12, 2020
@decryp2kanon
Copy link
Contributor Author

decryp2kanon commented May 13, 2020 •
edited
Loading

  • 7% less traffic.
    image

  • little faster
    image

very good

@decryp2kanon decryp2kanon changed the title IBD: MAX_BLOCKS_IN_TRANSIT_PER_PEER = MAX_HEADERS_RESULTS IBD: amount of max blocks in transit per peer May 13, 2020
@decryp2kanon decryp2kanon changed the title IBD: amount of max blocks in transit per peer IBD: max blocks in transit per peer for GetPoWHash_cached May 13, 2020
@decryp2kanon decryp2kanon changed the title IBD: max blocks in transit per peer for GetPoWHash_cached IBD: max blocks in transit per peer (cached PoW) May 13, 2020
cryptozeny
cryptozeny approved these changes May 15, 2020
View reviewed changes
Copy link
Member

@cryptozeny cryptozeny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested ACK a7b1cca

@cryptozeny cryptozeny merged commit eff20b3 into master-v0.16.3 May 15, 2020
@cryptozeny cryptozeny deleted the fix_MAX_BLOCKS_IN_TRANSIT_PER_PEER branch May 15, 2020 06:26
@decryp2kanon decryp2kanon linked an issue May 15, 2020 that may be closed by this pull request
[PoW cache] wallet.dat recognize issue on Win64 after unexpected system shut down (ie: power outage) #46
Closed
@decryp2kanon decryp2kanon mentioned this pull request May 15, 2020
Merged
decryp2kanon added a commit that referenced this pull request May 15, 2020
@decryp2kanon
bump v0.16.3.32rc2 + manpage (#137)
5ddf16f 
**Changes: v0.16.3.32rc2**

- Major
  * Fix: IBD `30%` faster, and `60%` reduced data traffic #122 
  * Add: new option `-prunedebuglogfile`: limit filesize of debug.log #110  
  * Add: when IBD, print blockheader count on debug.log #128 
  * Update: checkpoints (mainnet) #132 
  * Fix: IBD optimizing #135 

- Minor
  * GUI: display size in MB (was GB) #125 
  * Revert: IBD settings back to BTC original #124 #126
This was referenced May 17, 2020
Closed
Merged
decryp2kanon added a commit that referenced this pull request May 23, 2020
@decryp2kanon
bump v0.16.3.33rc3 + manpage (#152)
ebfd740 
**Changes: v33rc3**

- Major
  * Fix: IBD `30%` faster, and `60%` reduced data traffic #122 
  * Add: new option `-prunedebuglogfile`: limit filesize of debug.log #110  
  * Add: when IBD, print blockheader count on debug.log #128 
  * Update: checkpoints (mainnet) #132 
  * Fix: IBD optimizing #135 
  * Update: seed list #144 
  * Remove: BCLog::POW (-debug=pow) #142 

- Minor
  * GUI: display size in MB (was GB) #125 
  * Revert: IBD settings back to BTC original #124 
  * Add: bootstrap height at 4421701 #143 
  * Fix: travis pathlib2 #148
@decryp2kanon decryp2kanon mentioned this pull request Jun 6, 2020
Merged
decryp2kanon added a commit that referenced this pull request Jun 6, 2020
@decryp2kanon
bump: v0.16.3.34-starboy + manpage (#161)
88fe002 
**Changes: v34-starboy (same as v33)**

- Major
  * Fix: IBD `30%` faster, and `60%` reduced data traffic #122 
  * Add: new option `-prunedebuglogfile`: limit filesize of debug.log #110  
  * Add: when IBD, print blockheader count on debug.log #128 
  * Update: checkpoints (mainnet) #132 
  * Fix: IBD optimizing #135 
  * Update: seed list #144 
  * Remove: BCLog::POW (-debug=pow) #142 

- Minor
  * GUI: display size in MB (was GB) #125 
  * Revert: IBD settings back to BTC original #124 
  * Add: bootstrap height at 4421701 #143 
  * Fix: travis pathlib2 #148
@decryp2kanon decryp2kanon mentioned this pull request Jun 9, 2020
Closed
decryp2kanon added a commit that referenced this pull request Jun 14, 2020
@decryp2kanon
IBD: max blocks in transit per peer (cached PoW) (#135)
28239d0
decryp2kanon added a commit that referenced this pull request Jun 14, 2020
@decryp2kanon
bump v0.16.3.32rc2 + manpage (#137)
3b0e8b2 
**Changes: v0.16.3.32rc2**

- Major
  * Fix: IBD `30%` faster, and `60%` reduced data traffic #122 
  * Add: new option `-prunedebuglogfile`: limit filesize of debug.log #110  
  * Add: when IBD, print blockheader count on debug.log #128 
  * Update: checkpoints (mainnet) #132 
  * Fix: IBD optimizing #135 

- Minor
  * GUI: display size in MB (was GB) #125 
  * Revert: IBD settings back to BTC original #124 #126
decryp2kanon added a commit that referenced this pull request Jun 14, 2020
@decryp2kanon
bump v0.16.3.33rc3 + manpage (#152)
4808337 
**Changes: v33rc3**

- Major
  * Fix: IBD `30%` faster, and `60%` reduced data traffic #122 
  * Add: new option `-prunedebuglogfile`: limit filesize of debug.log #110  
  * Add: when IBD, print blockheader count on debug.log #128 
  * Update: checkpoints (mainnet) #132 
  * Fix: IBD optimizing #135 
  * Update: seed list #144 
  * Remove: BCLog::POW (-debug=pow) #142 

- Minor
  * GUI: display size in MB (was GB) #125 
  * Revert: IBD settings back to BTC original #124 
  * Add: bootstrap height at 4421701 #143 
  * Fix: travis pathlib2 #148
decryp2kanon added a commit that referenced this pull request Jun 14, 2020
@decryp2kanon
bump: v0.16.3.34-starboy + manpage (#161)
333ba6a 
**Changes: v34-starboy (same as v33)**

- Major
  * Fix: IBD `30%` faster, and `60%` reduced data traffic #122 
  * Add: new option `-prunedebuglogfile`: limit filesize of debug.log #110  
  * Add: when IBD, print blockheader count on debug.log #128 
  * Update: checkpoints (mainnet) #132 
  * Fix: IBD optimizing #135 
  * Update: seed list #144 
  * Remove: BCLog::POW (-debug=pow) #142 

- Minor
  * GUI: display size in MB (was GB) #125 
  * Revert: IBD settings back to BTC original #124 
  * Add: bootstrap height at 4421701 #143 
  * Fix: travis pathlib2 #148
decryp2kanon added a commit that referenced this pull request Jun 14, 2020
@decryp2kanon
[backport] API server v34 starboy (88fe002) (#165)
c4e813b 
* add: option "-prunedebuglogfile": limit filesize of debug.log (#110)

The idea is `ShrinkDebugFile` in realtime.

- AIM:
to prevent disk is filling full up with log file.

- DEBUG:
If `debug.log` is over 10 MB (`10*1000*1000`), shrink to 1 MB (`1*1000*1000`). 10x smaller
  * `watch -n1 ls -lh debug.log`
  * `watch -n5 ps -p "$(cat sugarchaind.pid)" -o %cpu,%mem,cmd`

- RUN: 
check logging speed and filesize
  * `sugarchaind -prunedebuglogfile -reindex-chainstate`

- PERIOD:
a cycle took around `4:30` when `-reindex-chainstate`
```
2020-04-20 23:18:55 DEBUG.LOG PRUNED at 10000071
2020-04-20 23:24:33 DEBUG.LOG PRUNED at 10000014
2020-04-20 23:30:11 DEBUG.LOG PRUNED at 10000018
2020-04-20 23:35:45 DEBUG.LOG PRUNED at 10000186
```

* IBD: do not check PoW (Yespower) during downloading headers (#122)

* IBD: do not check PoW (Yespower) during downloading headers
However this means checking PoW during IBD is, not actually skipped, but still checking in another places. This makes IBD much faster.
* remove: debug printf
* adding comment by volbil

* revert: (#80) do not disconnect whitelisted peers during IBD (#124)

* GUI: do not display in GB, but in MB (#125)

* revert (#78) & fix: disabled more getheaders (#126)

* revert&fix: disabled more getheaders
* remove: printf

* IBD: Print blockheader count on debug.log (#128)

* revert: MINIMUM_CONNECT_TIME (#129)

https://github.com/bitcoin/bitcoin/blob/f56c00b2345cd2e392ade4733e2ca9cb9b0af623/src/net_processing.h#L36

* Revert "revert: MINIMUM_CONNECT_TIME (#129)" (#130)

This reverts commit e37dfec.

* update: checkpoint (#132)

* bump 0.16.3.31rc1 + manpage (#133)

**Changes: v0.16.3.31rc1**

- Major
  * Fix: IBD `30%` faster, and `60%` reduced data traffic #122 
  * Add: new option `-prunedebuglogfile`: limit filesize of debug.log #110  
  * Add: when IBD, print blockheader count on debug.log #128 
  * Update: checkpoints (mainnet) #132 

- Minor
  * GUI: display size in MB (was GB) #125 
  * Revert: IBD settings back to BTC original #124 #126

* IBD: max blocks in transit per peer (cached PoW) (#135)

* bump v0.16.3.32rc2 + manpage (#137)

**Changes: v0.16.3.32rc2**

- Major
  * Fix: IBD `30%` faster, and `60%` reduced data traffic #122 
  * Add: new option `-prunedebuglogfile`: limit filesize of debug.log #110  
  * Add: when IBD, print blockheader count on debug.log #128 
  * Update: checkpoints (mainnet) #132 
  * Fix: IBD optimizing #135 

- Minor
  * GUI: display size in MB (was GB) #125 
  * Revert: IBD settings back to BTC original #124 #126

* fix: daemon Killed on ARM during IBD (out-of-memory) (#140)

* Revert "revert (#78) & fix: disabled more getheaders (#126)"

This reverts commit 7c45e62.

* comment

* remove: BCLog::POW (-debug=pow) (#142)

* add: bootstrap height at 4421701 (#143)

* scripts: In linearize, search for next position of magic bytes rather than fail
bitcoin/bitcoin#16802

* add: bootstrap height at 4421701

* update: seeds 2020-05-19 KST (#144)

* fix: travis: pathlib2 (#148)

ImportError: No module named 'pathlib2'

* doc (#151)

* bump v0.16.3.33rc3 + manpage (#152)

**Changes: v33rc3**

- Major
  * Fix: IBD `30%` faster, and `60%` reduced data traffic #122 
  * Add: new option `-prunedebuglogfile`: limit filesize of debug.log #110  
  * Add: when IBD, print blockheader count on debug.log #128 
  * Update: checkpoints (mainnet) #132 
  * Fix: IBD optimizing #135 
  * Update: seed list #144 
  * Remove: BCLog::POW (-debug=pow) #142 

- Minor
  * GUI: display size in MB (was GB) #125 
  * Revert: IBD settings back to BTC original #124 
  * Add: bootstrap height at 4421701 #143 
  * Fix: travis pathlib2 #148

* update: blockchain size as 3GB (#153)

* cleanup (#154)

* comment
* seeds version checker

* doc: release note v34 starboy (#160)

* doc: release note: starboy
* fix: known issue

* bump: v0.16.3.34-starboy + manpage (#161)

**Changes: v34-starboy (same as v33)**

- Major
  * Fix: IBD `30%` faster, and `60%` reduced data traffic #122 
  * Add: new option `-prunedebuglogfile`: limit filesize of debug.log #110  
  * Add: when IBD, print blockheader count on debug.log #128 
  * Update: checkpoints (mainnet) #132 
  * Fix: IBD optimizing #135 
  * Update: seed list #144 
  * Remove: BCLog::POW (-debug=pow) #142 

- Minor
  * GUI: display size in MB (was GB) #125 
  * Revert: IBD settings back to BTC original #124 
  * Add: bootstrap height at 4421701 #143 
  * Fix: travis pathlib2 #148
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cryptozeny cryptozeny cryptozeny approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[PoW cache] wallet.dat recognize issue on Win64 after unexpected system shut down (ie: power outage)
2 participants
@decryp2kanon @cryptozeny