Skip to content
This repository has been archived by the owner on Jun 11, 2024. It is now read-only.

Blockchain synchronisation slow down with progress #147

Closed
ghost opened this issue May 24, 2016 · 11 comments
Closed

Blockchain synchronisation slow down with progress #147

ghost opened this issue May 24, 2016 · 11 comments

Comments

@ghost
Copy link

ghost commented May 24, 2016

With latest release of testnet v0.2.3 blockchain synchronisation slow down with progress so after 24h my node can't catch up with network since interval between loading each block is much higher than actual network block time.

At beginning it's as fast as it should i guess.

info 2016-05-24 07:07:55 Block 16027920132576797141 loaded from 51.255.86.161:7000 at 207
info 2016-05-24 07:07:55 Block 10971402408485944279 loaded from 51.255.86.161:7000 at 208
info 2016-05-24 07:07:55 Block 14290578737735551437 loaded from 51.255.86.161:7000 at 209
info 2016-05-24 07:07:55 Block 8967653460925932409 loaded from 51.255.86.161:7000 at 210
info 2016-05-24 07:07:55 Block 12861445348496127252 loaded from 51.255.86.161:7000 at 211
info 2016-05-24 07:07:55 Block 14082482237090135045 loaded from 51.255.86.161:7000 at 212
info 2016-05-24 07:07:55 Block 8524510368941954595 loaded from 51.255.86.161:7000 at 213
info 2016-05-24 07:07:55 Block 6011172009682647946 loaded from 51.255.86.161:7000 at 214
info 2016-05-24 07:07:55 Block 4949396090849470976 loaded from 51.255.86.161:7000 at 215
info 2016-05-24 07:07:55 Block 9616531363559409411 loaded from 51.255.86.161:7000 at 216
info 2016-05-24 07:07:55 Block 12847142259814052877 loaded from 51.255.86.161:7000 at 217
info 2016-05-24 07:07:55 Block 2527104611989216463 loaded from 51.255.86.161:7000 at 218
info 2016-05-24 07:07:55 Block 4899261458032572340 loaded from 51.255.86.161:7000 at 219
info 2016-05-24 07:07:55 Block 8313503740366176501 loaded from 51.255.86.161:7000 at 220
info 2016-05-24 07:07:55 Block 116739694398215277 loaded from 51.255.86.161:7000 at 221
info 2016-05-24 07:07:55 Block 17729957199777519798 loaded from 51.255.86.161:7000 at 222
info 2016-05-24 07:07:55 Block 10061484172125794469 loaded from 51.255.86.161:7000 at 223
info 2016-05-24 07:07:55 Block 2893039422167403417 loaded from 51.255.86.161:7000 at 224
info 2016-05-24 07:07:55 Block 7304162042938245683 loaded from 51.255.86.161:7000 at 225
info 2016-05-24 07:07:55 Block 13152027098765385988 loaded from 51.255.86.161:7000 at 226
info 2016-05-24 07:07:55 Block 14301456006470555680 loaded from 51.255.86.161:7000 at 227
info 2016-05-24 07:07:55 Block 3709565955703322623 loaded from 51.255.86.161:7000 at 228
info 2016-05-24 07:07:55 Block 14354439903929585508 loaded from 51.255.86.161:7000 at 229
info 2016-05-24 07:07:55 Block 3045067056866389161 loaded from 51.255.86.161:7000 at 230
info 2016-05-24 07:07:55 Block 16933407687401574287 loaded from 51.255.86.161:7000 at 231
info 2016-05-24 07:07:55 Block 14396574498644348396 loaded from 51.255.86.161:7000 at 232
info 2016-05-24 07:07:55 Block 3416927168971869147 loaded from 51.255.86.161:7000 at 233
info 2016-05-24 07:07:55 Block 11543581272405744012 loaded from 51.255.86.161:7000 at 234
info 2016-05-24 07:07:55 Block 10747794576225700626 loaded from 51.255.86.161:7000 at 235
info 2016-05-24 07:07:55 Block 11506628357165352442 loaded from 51.255.86.161:7000 at 236
info 2016-05-24 07:07:55 Block 4416381309806323378 loaded from 51.255.86.161:7000 at 237
info 2016-05-24 07:07:55 Block 1293740756410494420 loaded from 51.255.86.161:7000 at 238
info 2016-05-24 07:07:55 Block 17403166990745697479 loaded from 51.255.86.161:7000 at 239
info 2016-05-24 07:07:55 Block 24327239611821086 loaded from 51.255.86.161:7000 at 240
info 2016-05-24 07:07:55 Block 1570043680878273544 loaded from 51.255.86.161:7000 at 241
info 2016-05-24 07:07:55 Block 15311251797584680702 loaded from 51.255.86.161:7000 at 242
info 2016-05-24 07:07:55 Block 14545331096425639798 loaded from 51.255.86.161:7000 at 243
info 2016-05-24 07:07:55 Block 6791471016011267892 loaded from 51.255.86.161:7000 at 244
info 2016-05-24 07:07:55 Block 6989913920757805857 loaded from 51.255.86.161:7000 at 245
info 2016-05-24 07:07:56 Block 10666441902367199611 loaded from 51.255.86.161:7000 at 246
info 2016-05-24 07:07:56 Block 13255664351737072037 loaded from 51.255.86.161:7000 at 247

after not more than 15minutes synchronisation process become very slow, so after 24h my 2 nodes couldn't catch up with network. i think it needs to be fixed soon, before release of mainnet.

info 2016-05-24 07:20:59 Block 3991217456860976736 loaded from 51.255.86.161:7000 at 7346
info 2016-05-24 07:21:01 Block 6384380097413846322 loaded from 51.255.86.161:7000 at 7347
log 2016-05-24 07:21:02 POST /peer/blocks from 5.189.153.97 
log 2016-05-24 07:21:02 POST /peer/blocks from 108.61.211.199 
log 2016-05-24 07:21:02 POST /peer/blocks from 45.32.7.191 
log 2016-05-24 07:21:02 POST /peer/blocks from 185.92.221.6 
log 2016-05-24 07:21:02 GET /api/blocks/getHeight from 192.99.58.223 
log 2016-05-24 07:21:02 GET /peer/height from 81.7.8.206 
log 2016-05-24 07:21:02 POST /peer/blocks from 104.236.179.172 
log 2016-05-24 07:21:02 POST /peer/blocks from 104.236.16.207 
log 2016-05-24 07:21:02 POST /peer/blocks from 133.130.101.73 
log 2016-05-24 07:21:02 POST /peer/blocks from 163.44.169.182 
log 2016-05-24 07:21:02 POST /peer/blocks from 45.32.74.240 
log 2016-05-24 07:21:02 POST /peer/blocks from 159.203.36.201 
log 2016-05-24 07:21:03 POST /peer/blocks from 188.166.235.73 
log 2016-05-24 07:21:03 POST /peer/blocks from 51.255.87.64 
log 2016-05-24 07:21:03 POST /peer/blocks from 149.56.126.216 
log 2016-05-24 07:21:03 POST /peer/blocks from 45.32.186.206 
log 2016-05-24 07:21:03 POST /peer/blocks from 62.138.6.4 
log 2016-05-24 07:21:03 POST /peer/blocks from 104.238.162.113 
log 2016-05-24 07:21:03 GET /api/loader/status/sync from 10.0.0.2 
log 2016-05-24 07:21:03 POST /peer/blocks from 192.99.58.223 
log 2016-05-24 07:21:04 POST /peer/blocks from 128.199.117.5 
info 2016-05-24 07:22:04 Block 17751953443219194156 loaded from 51.255.86.161:7000 at 7348
log 2016-05-24 07:22:05 POST /peer/blocks from 91.121.64.132 
log 2016-05-24 07:22:05 POST /peer/blocks from 5.1.88.172 

It's not networking issue since both nodes are located in different locations. Both have pretty good and stable connection. As i have noticed further synchronisation goes process becomes slower.

@karmacoma karmacoma self-assigned this May 24, 2016
@karmacoma
Copy link
Contributor

@karek314 I have encountered the same issue, will look to fix asap.

@karmacoma karmacoma assigned fix and unassigned karmacoma May 24, 2016
@fix
Copy link
Contributor

fix commented May 24, 2016

testing with current development is ok

@ghost
Copy link
Author

ghost commented May 24, 2016

I have noticed that it's not related strictly to progress. Process randomly slows down and can't recover into normal speed, but actually restarting node fix issue for random time.

Edit 1
Im still not able to sync node. Even when blocks are loading correctly node load corrupted block as described in issue #82 at random height, sometimes ~16k, 20k, 24k and so on.

Edit 2
I would like to confirm that issue still occurs in 0.3.0

@fix
Copy link
Contributor

fix commented May 26, 2016

Yes it is better but still happening. I think now this is because of overal network load.
We are still working on several sides of this issue

@mrv777
Copy link
Contributor

mrv777 commented May 26, 2016

Not sure if it helps, but it seemed to load better if I blocked port 8000 during the rebuild and when it gets close to being caught up, I stop the rebuild, open port 8000/tcp again, and have it finish the rebuild

@fix
Copy link
Contributor

fix commented May 30, 2016

the last version has helped (with the noise reduced over the network), but not solved the issue completely. One possibility would be to load blocks by batch of 1000s. also querying several different clients.

@karmacoma karmacoma added this to the Version 0.3.2 milestone Jun 8, 2016
@ByronAP
Copy link

ByronAP commented Jun 10, 2016

client should be pulling from more than 1 server or at least round robin through a batch of servers, I have noticed that trying to do a full sync from just one server makes the server tired (so to speak) and may come to a crawl.

@fix
Copy link
Contributor

fix commented Jun 11, 2016

yes this is already foreseen in the next version.

see https://github.com/fix/lisk/commit/574f88df80f4adbe9b6ffe6712399cd09afbad66

@ByronAP
Copy link

ByronAP commented Jun 11, 2016

@fix the pull looks good but I think it should have a limit say 1k block then it rechecks and uses a new peer from the good peer pool

karmacoma pushed a commit that referenced this issue Jun 21, 2016
- Fixing error messages in logic/transaction.js.
- Blocks#processBlock: Separating block verification and application logic. Checking block and transactions validity before any database write.
- Refactoring Blocks#loadBlocksOffset to use new applyBlock().
- Documenting code with comments.
@karmacoma karmacoma modified the milestone: Mainchain Stabilisation Aug 31, 2016
@karmacoma karmacoma reopened this Sep 22, 2016
@karmacoma
Copy link
Contributor

In the latest release candidates 0.4.0a and 0.4.0b, block synchronization remains unstable.

This is being caused by a memory leak in the 3rd party library z-schema. This should be resolved in the next release candidate 0.4.0c.

@karmacoma
Copy link
Contributor

I've found another memory leak in the http request module when processing large response bodies, such as loading blocks. Again, it will be fixed in 0.4.0c.

karmacoma pushed a commit that referenced this issue Sep 24, 2016
Defining functions on constructor prototype.
karmacoma pushed a commit that referenced this issue Sep 24, 2016
Defining transaction schemas on constructor prototype.
karmacoma pushed a commit that referenced this issue Sep 24, 2016
- Adding unique ids to schemas.
- Moving schemas into separate modules.
- Do not create query/body variables from req.body.
- Removing duplicate schema validations.
- Standardising variable names.
- Updating z-schema dependency.
karmacoma pushed a commit that referenced this issue Sep 24, 2016
- Controlling flow using async waterfall.
- Removing async.whilst count < 30.
shuse2 pushed a commit that referenced this issue Jan 19, 2022
Add size check when adding to transactionPool - Closes #146
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants