Add lotus-shed command for backfilling txhash.db #10932

fridrik01 · 2023-05-30T20:06:13Z

Context

This PR adds a new lotus-shed command for backfilling the txhash.db.

It also moves existing command related to managing sqlite indices into a indexes subcategory.

New help format:

lotus-shed indexes
NAME:
   lotus-shed indexes - Commands related to managing sqlite indexes

USAGE:
   lotus-shed indexes command [command options] [arguments...]

COMMANDS:
   MSGINDEX:
     backfill-msgindex  Backfill the msgindex.db for a number of epochs starting from a specified height
     prune-msgindex     Prune the msgindex.db for messages included before a given epoch
   TXHASH:
     backfill-txhash  Backfills the txhash.db for a number of epochs starting from a specified height

OPTIONS:
   --help, -h  show help (default: false)

Test Plan

Compile:

make lotus-shed

Run backfilling (on local node with 2k blocks):

./lotus-shed indexes backfill-txhash   --epochs=1000
2023-05-30T20:03:03.958Z    INFO    lotus-shed	lotus-shed/indexes.go:98       0/1000 processing epoch:2905526
2023-05-30T20:03:05.232Z    INFO    lotus-shed	lotus-shed/indexes.go:98       100/1000 processing epoch:2905426
2023-05-30T20:03:06.937Z    INFO    lotus-shed	lotus-shed/indexes.go:98       200/1000 processing epoch:2905326
2023-05-30T20:03:08.221Z    INFO    lotus-shed	lotus-shed/indexes.go:98       300/1000 processing epoch:2905226
2023-05-30T20:03:09.458Z    INFO    lotus-shed	lotus-shed/indexes.go:98       400/1000 processing epoch:2905126
2023-05-30T20:03:11.006Z    INFO    lotus-shed	lotus-shed/indexes.go:98       500/1000 processing epoch:2905026
2023-05-30T20:03:12.623Z    INFO    lotus-shed	lotus-shed/indexes.go:98       600/1000 processing epoch:2904926
2023-05-30T20:03:14.289Z    INFO    lotus-shed	lotus-shed/indexes.go:98       700/1000 processing epoch:2904826
2023-05-30T20:03:15.987Z    INFO    lotus-shed	lotus-shed/indexes.go:98       800/1000 processing epoch:2904726
2023-05-30T20:03:17.490Z    INFO    lotus-shed	lotus-shed/indexes.go:98       900/1000 processing epoch:2904626
2023-05-30T20:03:18.837Z    INFO    lotus-shed	lotus-shed/indexes.go:131      Done, inserted 1 missing txhashes

Run backfilling on fevm archival node:

time sudo ./lotus-shed indexes backfill-txhash --repo=/mnt/raid/
...
2023-06-01T03:20:06.027Z    INFO    lotus-shed	lotus-shed/indexes.go:98        139300/200000 processing epoch:2642256
2023-06-01T03:20:08.799Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0xecbd3cb0694099536b7144487edb066f047022c220b922517783821a4fce1a19, cid: bafy2bzacebschxegwkxsx36swhi3kc4itkslteugdwbzfs4ejoavcuzcw3tgk at epoch: 2642245
2023-06-01T03:20:10.438Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x5df09354a8bf8af464aa70762ba5a07474720be0c8917282b5bbd085f49a3ac3, cid: bafy2bzacedmivmrkqmwm77cakvugdb2igepmxv77tqxm2dxhepquxgwglovt6 at epoch: 2642238
2023-06-01T03:20:15.674Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x9a5e09e0125c5441b9d39ce56d058dbc233954676d6d7d9eabfb860608096003, cid: bafy2bzaceaqtkhcrstr3w2apstz4gyijbonr25cn4bwaxgj7nubgzfkge2qru at epoch: 2642215
2023-06-01T03:20:20.671Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x3e7e73a8d4d9a74be98518f4e4a2158369b7cf874bcfdae91f58f71402434d41, cid: bafy2bzacebwn4k6nbmtarlpejcqsqdpqmhlulpzc5meql2jewdgfzjalpbqn4 at epoch: 2642195
2023-06-01T03:20:21.969Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x9021ccf82ca858ab4ea82dda0c1424bdc3e7d02265ed64118fedabbd4c0f4764, cid: bafy2bzacebmvz533pvhbzt4ks6w2ughvmfzplpiisjyieabcvzesxulnvggqs at epoch: 2642189
2023-06-01T03:20:22.727Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x316baa31410890112a8e127f565382112aa18739cc83c5a1ed8e0392b235cc0e, cid: bafy2bzacedkclgnnmlj3ws4jxw2qa6in7t4277ijwyfdmrsbuyk5avd33wafe at epoch: 2642186
2023-06-01T03:20:22.761Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x463651d219704bcbbc7ebb6e231e833216d37ba4e7a790ba588509041edbe8aa, cid: bafy2bzacedyi36chiva4t5expx3xzzxud37r7eagmummo754ngvyrehehmapa at epoch: 2642186
2023-06-01T03:20:24.176Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x05e3c33c66bd8fa6a218d38e39da2e32591a6e0602fd451926a03a07e57c5390, cid: bafy2bzacebvkqrnehtompd2v5h6o26qtzs6lj645iuqc7kyjcrrey7n4kt7k4 at epoch: 2642179
2023-06-01T03:20:24.862Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0xf60232bcdc5b2b32f466b8dff18f71d293ed790cd4359776c4fc0d9c80d2055b, cid: bafy2bzaceavqsnqf5qsulycfxb6o6yjln34xrthejfscfvddcukogl3tstlmy at epoch: 2642176
2023-06-01T03:20:25.139Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0xddd98be3ce7db50ab17c809ee3275ac618747c6ef3914d6e55f3e7f63200c5a8, cid: bafy2bzacea2mskctt3o2eqowfxl3kn5jxyzpkiw7u6mshdwxekhmk534tx3ra at epoch: 2642174
2023-06-01T03:20:25.570Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x44ef42b94517c7ca60191625bd7e876368cb6aebb8d70501e787630a46764e89, cid: bafy2bzaceatnqoxnv5ijz3uuh425jxxtwsqynr4zti6a5ajfnnm5wjhwyxfui at epoch: 2642173
2023-06-01T03:20:25.771Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x84f8012760446e4b631b97e88c58c95469f5d80549fa3e4b7b3fa696641039b2, cid: bafy2bzaceaa22blnxoqiwq7rfrrbbmas2oyu7iw4eynv7lnlxlkormclynp6k at epoch: 2642172
2023-06-01T03:20:26.646Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0xc93ca2cfa4bec624304722c064a2b94e41827b1e61a9a25dddfa6631fac2a43c, cid: bafy2bzacebzn5ac7i6zsppbz4vbrtrnx3cd3lru7fgwy2cw4prcghh7w3gxwa at epoch: 2642169
2023-06-01T03:20:27.253Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x07aabb1659ec7e3bffc7bf11d8ec8db9d78bf1e538b8721b872a0b6c76d8e43f, cid: bafy2bzacedgttrj7pamyj2o34c7ud6bbyneaso6kfclpypslqmv5ainnkckfu at epoch: 2642166
2023-06-01T03:20:29.052Z    INFO    lotus-shed	lotus-shed/indexes.go:98        139400/200000 processing epoch:2642156
2023-06-01T03:20:29.216Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x000c23f24345cf47c071d9fb2f58a4daa2bd9bfe96fd0902b7c33ef930e9bb4c, cid: bafy2bzacebzff62fwuxoyicwtwvbjpxtbpcupbs3wqmks45xslxn7rdc64elk at epoch: 2642156
2023-06-01T03:20:29.220Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x00d4ce4903b96cab724e258523965a6c62a44aa066985a865196e56212113cf3, cid: bafy2bzaceakxapabrajvpzh5zj2qdrmsdq6gjcdqdfa27tnur4sqemy5yvt3i at epoch: 2642156
2023-06-01T03:20:54.061Z    INFO    lotus-shed	lotus-shed/indexes.go:98        139500/200000 processing epoch:2642056
2023-06-01T03:21:06.465Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x6cb08bd0c9b5f499a5d0892d745e55854b628100e5d9b027943a50e3c516284b, cid: bafy2bzacebi7pcuk5kf3gnlrcxv7ia5pddwkbrvj7axsjlwgrztko4ubkoays at epoch: 2642009
2023-06-01T03:21:19.410Z    INFO    lotus-shed	lotus-shed/indexes.go:98        139600/200000 processing epoch:2641956
2023-06-01T03:21:22.628Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0xe368fd0c150a8c709b51abd908594b0025bfdbd4b9d5e07d9fa088f4c659a8d7, cid: bafy2bzacebjnxg4meo6qwsttu7bcomeaq5ogpk5jecj2zokvdgrrdg7zb65d2 at epoch: 2641944
2023-06-01T03:21:23.287Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0x4db1435665f842cba7d4991e88e4d5da8364f50f1f3e65c3345cc54b47d11f7c, cid: bafy2bzacedpmdqm7rowlw2o2dithvnqmatwaslkqgawdlbxg6ue7a6sj2ubwy at epoch: 2641941
2023-06-01T03:21:23.986Z    INFO    lotus-shed	lotus-shed/indexes.go:321       Inserted txhash 0xd2f9b9731a55225dc6c5b0a5857970b3d5ed0f38ce068e8d22e19f1d17d4cd8e, cid: bafy2bzaceb3e6nrzybeq2p3jdwgsu2yhfyzilu36rtrhdlha3k4v6jk4swwbo at epoch: 2641938
2023-06-01T03:21:35.351Z    INFO    lotus-shed	lotus-shed/indexes.go:290       Could not get block messages at epoch: 2641891, stopping walking up the chain
2023-06-01T03:21:35.352Z    INFO    lotus-shed	lotus-shed/indexes.go:331       Done, inserted 449 missing txhashes

real    387m41.284s
user    41m32.460s
sys     2m59.541s

jennijuju · 2023-06-01T02:47:10Z

I would suggest to move this to ./lotus evm

magik6k

Seems reasonable, two small comments

Did we test that it works on mainnet/devnets?

magik6k · 2023-06-01T15:07:56Z

cli/evm.go

+
+var EvmBackfillTxHashCmd = &cli.Command{
+	Name:        "backfill-txhash",
+	Description: "Backfills the txhash.db for a number of epochs starting from a specified height",


Should be Usage

magik6k · 2023-06-01T15:08:27Z

cli/evm.go

+		&cli.StringFlag{
+			Name:  "repo",
+			Value: "~/.lotus",
+			Usage: "path to the repo",
+		},


That is already defined as a global flag, shouldn't be needed

@magik6k I think its needed as EvmCmd does not have this as a Flag, at least skipping it produces an error.

If we move this back to lotus-shed, you shouldn't need this flag.

fridrik01 · 2023-06-01T15:40:53Z

Seems reasonable, two small comments

Did we test that it works on mainnet/devnets?

I tested both locally on mainet using lightweight snapshot, and also on fevm arvhival node (see test plan). I did not test on devnet though.

jennijuju · 2023-06-01T18:30:39Z

Seems reasonable, two small comments

Did we test that it works on mainnet/devnets?

i did for calibration.

arajasek

Code LGTM, but I think we need to move this back to lotus-shed. The Lotus CLI is meant to be run against a live node, so we can't have commands that require locking the db.

I'd suggest adding a subcategory under shed for indexes, and include this along with the msgindex command and others.

fridrik01 · 2023-06-03T17:23:37Z

@arajasek moved back to lotus-shed and introduced a new indexes subcategory

fridrik01 force-pushed the backfill-txhash branch from 307bcb3 to d0cbd9b Compare May 31, 2023 16:53

fridrik01 marked this pull request as ready for review May 31, 2023 17:15

fridrik01 requested a review from a team as a code owner May 31, 2023 17:15

fridrik01 force-pushed the backfill-txhash branch from d0cbd9b to f6b0b36 Compare June 1, 2023 11:10

fridrik01 changed the title ~~Add lotus-shed command for backfilling txhash.db~~ Add lotus evm command for backfilling txhash.db Jun 1, 2023

magik6k reviewed Jun 1, 2023

View reviewed changes

arajasek reviewed Jun 1, 2023

View reviewed changes

fridrik01 added 3 commits June 3, 2023 11:20

Add lotus evm command for backfilling txhash.db

c38193a

Run docsgen-cli

4e55147

Address review comments

e414acb

fridrik01 force-pushed the backfill-txhash branch 2 times, most recently from d2b8c9c to 61ec334 Compare June 3, 2023 17:16

fridrik01 requested a review from a team June 3, 2023 17:16

Move back to lotus-shed and add subcategory for indexes

26b4866

fridrik01 force-pushed the backfill-txhash branch from 61ec334 to 26b4866 Compare June 3, 2023 17:22

fridrik01 changed the title ~~Add lotus evm command for backfilling txhash.db~~ Add lotus-shed command for backfilling txhash.db Jun 3, 2023

arajasek approved these changes Jun 6, 2023

View reviewed changes

arajasek merged commit 424bab3 into master Jun 6, 2023
93 checks passed

arajasek deleted the backfill-txhash branch June 6, 2023 16:17

fridrik01 mentioned this pull request Jun 23, 2023

Add support to backfill actor events + resuming when left off #10939

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lotus-shed command for backfilling txhash.db #10932

Add lotus-shed command for backfilling txhash.db #10932

fridrik01 commented May 30, 2023 •

edited

jennijuju commented Jun 1, 2023

magik6k left a comment

magik6k Jun 1, 2023

magik6k Jun 1, 2023

fridrik01 Jun 1, 2023

arajasek Jun 1, 2023

fridrik01 commented Jun 1, 2023

jennijuju commented Jun 1, 2023

arajasek left a comment

fridrik01 commented Jun 3, 2023

Add lotus-shed command for backfilling txhash.db #10932

Add lotus-shed command for backfilling txhash.db #10932

Conversation

fridrik01 commented May 30, 2023 • edited

jennijuju commented Jun 1, 2023

magik6k left a comment

Choose a reason for hiding this comment

magik6k Jun 1, 2023

Choose a reason for hiding this comment

magik6k Jun 1, 2023

Choose a reason for hiding this comment

fridrik01 Jun 1, 2023

Choose a reason for hiding this comment

arajasek Jun 1, 2023

Choose a reason for hiding this comment

fridrik01 commented Jun 1, 2023

jennijuju commented Jun 1, 2023

arajasek left a comment

Choose a reason for hiding this comment

fridrik01 commented Jun 3, 2023

fridrik01 commented May 30, 2023 •

edited