Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Claimtrie progress logging. #57

Merged
merged 3 commits into from
Jun 14, 2022
Merged

Conversation

moodyjon
Copy link
Collaborator

@moodyjon moodyjon commented Jun 14, 2022

Fixes: #50

I copied claimtrie/logger.go from netsync/blocklogger.go. Copyright comment from there left intact. Let me know if there is a better way to handle derivative works like this.

I did not find a way to handle percentage progress... The pebble stats API stuff seems to be private, so I did not try to use that to estimate the number of records.

Example output:

2022-06-14 13:32:48.121 [INF] MAIN: Version 0.0.0-local.0+67ce441247ecc148a9f21ef1798bb11269103c72-dirty
2022-06-14 13:32:48.121 [INF] MAIN: Loading block database from '/Users/swdev1/Library/Application Support/Lbcd/data/mainnet/blocks_ffldb'
2022-06-14 13:32:48.470 [INF] MAIN: Block database loaded
2022-06-14 13:32:51.472 [INF] SRVR: Can't discover UPnP-enabled device: write udp4 0.0.0.0:54154->239.255.255.250:1900: i/o timeout
2022-06-14 13:32:51.479 [INF] INDX: Transaction index is enabled
2022-06-14 13:32:51.480 [INF] INDX: Committed filter index is enabled
2022-06-14 13:32:52.000 [INF] LBRY: Normalizing strings via Go. Casefold and NFD table versions: 11.0.0 (from ICU 63.2)
2022-06-14 13:32:52.001 [INF] LBRY: Building the entire claim trie in RAM...
2022-06-14 13:32:56.030 [INF] LBRY: Spending claim but missing existing claim with TXO bd183d94dc0b2c38d500838f4ab7c93b643123f46e87214d3e37f5e9e0d4baf5:0, Name: 1791l-trumpcnn, ID: 6a28cbbe64c3c54113770705c928e80a138ddae9
2022-06-14 13:33:02.000 [INF] LBRY: Processed 1915480 claim names in the last 10s (total 1915480)
2022-06-14 13:33:12.001 [INF] LBRY: Processed 2113173 claim names in the last 10s (total 4028653)
2022-06-14 13:33:22.001 [INF] LBRY: Processed 1787925 claim names in the last 10s (total 5816578)
2022-06-14 13:33:28.681 [INF] MAIN: RAM: using 1.3 GB with 3.0 available, DISK: using 123.0 GB with 1164.9 available
2022-06-14 13:33:32.001 [INF] LBRY: Processed 1596516 claim names in the last 10s (total 7413094)
2022-06-14 13:33:42.001 [INF] LBRY: Processed 1519922 claim names in the last 10s (total 8933016)
2022-06-14 13:33:52.004 [INF] LBRY: Processed 1667066 claim names in the last 10s (total 10600082)
2022-06-14 13:33:52.753 [INF] LBRY: Spending claim but missing existing claim with TXO 546a1b99218f5935d6d11d6a5632ba199b24d6dbb6c07e75bea1089eee23987d:0, Name: ggminingpool-xzc, ID: 0b9e8ea179d6f68d3b37ab0e636af6ab1b93b52e
2022-06-14 13:33:52.753 [INF] LBRY: Spending claim but missing existing claim with TXO 5dff52209029a63547f8805c3e92fce87e3744a15f6539a12011fe0c421ca132:0, Name: ggminingpoolxzc, ID: 3fa8d7113bd4873e79ec02019bea3c7840f0deee
2022-06-14 13:34:02.004 [INF] LBRY: Processed 1486449 claim names in the last 10s (total 12086531)
2022-06-14 13:34:08.598 [INF] MAIN: RAM: using 2.3 GB with 2.7 available, DISK: using 123.0 GB with 1164.9 available
2022-06-14 13:34:12.004 [INF] LBRY: Processed 1337525 claim names in the last 10s (total 13424056)
2022-06-14 13:34:22.005 [INF] LBRY: Processed 1376697 claim names in the last 10s (total 14800753)
2022-06-14 13:34:29.657 [INF] LBRY: Spending claim but missing existing claim with TXO 022a25b42e0327d6bf7a741cd3bae34404453549635d916e5b05eb15edb05019:1, Name: redbullfourthphase, ID: f054e085e8d38c0a0e63a472b033e42d437a4014
2022-06-14 13:34:32.005 [INF] LBRY: Processed 1455552 claim names in the last 10s (total 16256305)
2022-06-14 13:34:42.005 [INF] LBRY: Processed 1426823 claim names in the last 10s (total 17683128)
2022-06-14 13:34:45.046 [INF] LBRY: Spending claim but missing existing claim with TXO d549351f731019573d921f5db8e2844449edf3011073b3e5f4ce9b0d9aee5490:1, Name: timcast-sweden-day1, ID: 247895d4b9389b74038ea5a4fd807cafc97ede53
2022-06-14 13:34:48.822 [INF] MAIN: RAM: using 2.8 GB with 2.7 available, DISK: using 123.0 GB with 1164.9 available
2022-06-14 13:34:48.850 [INF] LBRY: Spending claim but missing existing claim with TXO 36a719a156a1df178531f3c712b8b37f8e7cc3b36eea532df961229d936272a1:0, Name: two, ID: bb6168c9a56286999c7c938c7b8c5afb221d38d0
2022-06-14 13:34:52.005 [INF] LBRY: Processed 1134572 claim names in the last 10s (total 18817700)
2022-06-14 13:35:02.005 [INF] LBRY: Processed 1028133 claim names in the last 10s (total 19845833)
2022-06-14 13:35:23.789 [INF] CHAN: Loading block index...
2022-06-14 13:35:28.856 [INF] MAIN: RAM: using 2.8 GB with 2.7 available, DISK: using 123.0 GB with 1162.8 available
2022-06-14 13:35:38.269 [INF] CHAN: Chain state (height 1176375, hash 47098eddcdd048c47830428603983b73646c3dcd019c33ba0595a63aa84d45fc, totaltx 80627449, work 44778323276798910409405)

@coveralls
Copy link

coveralls commented Jun 14, 2022

Pull Request Test Coverage Report for Build 2497077944

  • 25 of 47 (53.19%) changed or added relevant lines in 3 files are covered.
  • 8 unchanged lines in 1 file lost coverage.
  • Overall coverage decreased (-0.01%) to 51.805%

Changes Missing Coverage Covered Lines Changed/Added Lines %
claimtrie/node/log.go 0 6 0.0%
claimtrie/logger.go 19 35 54.29%
Files with Coverage Reduction New Missed Lines %
peer/peer.go 8 75.92%
Totals Coverage Status
Change from base Build 2450492318: -0.01%
Covered Lines: 22471
Relevant Lines: 43376

💛 - Coveralls

@roylee17
Copy link
Collaborator

Though we don't track total number of claims at the moment, this looks like a nice improvement already.

  1. Preserving copyright comment is fine.
  2. I like how you group changes into logical commits.
  3. Another nice to have thing is prefix a category tag in the commit header.
    For example, this can be prefixed with "claimtrie: xxx"
    (I understand not every commits can be categorized cleanly though).

@roylee17 roylee17 merged commit ca9b4e5 into lbryio:master Jun 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Display progress during RamTrie rebuilding at (re)startup
3 participants