Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Lotus nv22 integration and testing #11584

Closed
6 tasks done
rjan90 opened this issue Jan 17, 2024 · 6 comments
Closed
6 tasks done

[WIP] Lotus nv22 integration and testing #11584

rjan90 opened this issue Jan 17, 2024 · 6 comments

Comments

@rjan90
Copy link
Contributor

rjan90 commented Jan 17, 2024 •
edited
Loading

We are preparing to ship Filecoin nv22 on April 2nd. Check the NV22 logistics track board for other key dates.

In order to prepare for the first Lotus v1.26.0-rc, we need to:

Tasks
Beta Give feedback
  1. Benchmark the nv22/DDO-migration #11593
    4 of 4
@rjan90
Copy link
Contributor Author

rjan90 commented Jan 17, 2024 •
edited
Loading

Testing plan: Switch Drand to Quicknet

Core Implementation

  • Confirm the beacon value used is first available exactly 30 seconds before the corresponding epoch time
  • Confirm that afternull rounds, the "missed" beacon values are NOT included, only one entry is included
In a 2k-network, observed null round:

2024-02-08T15:05:48.890Z	INFO	miner	miner/miner.go:505	completed mineOne	{"tookMilliseconds": 690, "forRound": 676, "baseEpoch": 674, "baseDeltaSeconds": 5, "nullRounds": 1, "lateStart": false, "beaconEpoch": 4867127, "lookbackEpochs": 900, "networkPowerAtLookback": "40960", "minerPowerAtLookback": "40960", "isEligible": true, "isWinner": true, "error": null}

----
674: (Feb  8 15:05:43) [ bafy2bzacedazttlbf54kt2w3vdhq2wei4ualc4rjnad3pfc22lffh6tubb2gm: t01000, ]
676: (Feb  8 15:05:51) [ bafy2bzacec7aomytnfyu3u7nn6dhvbn75knzjivrrxrfhaxpfy2wlrut6pas6: t01000, ]
---
root@387f81a45cbc:/lotus# lotus chain get-block bafy2bzacec7aomytnfyu3u7nn6dhvbn75knzjivrrxrfhaxpfy2wlrut6pas6
{
  "Miner": "t01000",
  "Ticket": {
    "VRFProof": "j6PTLiXCVCp2oiEoBNuuoVnxvopD/6eBnEHkOD/jK7XFffMMGpYJWAlIeYF3umEmCuhsFCxo3liL2Mo8cpJokcZxQrG1QjG3OD0pAb5tcqNaRk4W5r2+MAIZH4yqIdCt"
  },
  "ElectionProof": {
    "WinCount": 3,
    "VRFProof": "kKlgmvBTp4XHgOG/0BN3Besl+febZ5lGUlgPcmzbvLNa4lNXx8Pfh6n3DgpPhEaSF30YRdWZOLK4BfAIG8a61nB8gA5OVBCHTD0FKaua1P/tUP1GjNBRyB2OTTqbib4p"
  },
  "BeaconEntries": [
    {
      "Round": 4867126,
      "Data": "iyXSccL8jy3puxxz9F7vYeIToBJjWSWA4i30Zw1gtq44RCAldjAiHt3zh5ulpmK6"
    },
    {
      "Round": 4867127,
      "Data": "uJ278YQbnn6cmP/RvpdcQoqn9JiqhXdVRKdOCQYyDWrWN8YIuRhBnkrlQ9l1yawz"
    }
  ],
  •  If the exact upgrade epoch of the Drand switch is a null round. Confirm that 1) The first block after the null epoc should not include any beacons from mainnet. 2) Should include as many quicknet beacons as we've had null rounds, plus the beacon for the epoch in question itself (same as any other null round)

Switch Logic:

  • Test Drand switch when going from nv21 --> nv22 in a 2k-network with only a Lotus-node
**Upgrade height 200:**
2024-02-08T11:54:07.916+0100	INFO	consensus-common	consensus/compute_state.go:304	ApplyBlocks stats	{"early": 0.000002875, "earlyCronGas": 0, "vmMsg": 0.002520707, "msgGas": 0, "vmCron": 0.001092327, "cronGas": 15731091, "vmFlush": 0.022060526, "epoch": "200", "tsk": "{bafy2bzacecmywxwu6exjtlwt3vt3uvi2kdvwzi2sxytikjhaoynxztsoem6fg}"}
2024-02-08T11:54:09.065+0100	INFO	chain	chain/sync.go:625	block validation	{"took": 0.009393761, "height": "201", "age": 0.065922273}
2024-02-08T11:54:09.066+0100	INFO	chainstore	store/store.go:683	New heaviest tipset! [bafy2bzaceddvoqgsyipseky6ibf3zlr5lzqcagvw4skxccb64x7tucpkmdewo] (height=201)
2024-02-08T11:54:09.072+0100	INFO	chain	chain/sync_manager.go:234	selected sync target: [bafy2bzaceddvoqgsyipseky6ibf3zlr5lzqcagvw4skxccb64x7tucpkmdewo]
2024-02-08T11:54:09.072+0100	INFO	chain	chain/sync_manager.go:316	worker 200 syncing in [bafy2bzaceddvoqgsyipseky6ibf3zlr5lzqcagvw4skxccb64x7tucpkmdewo]
2024-02-08T11:54:09.072+0100	INFO	chain	chain/sync_manager.go:324	worker 200 done; took 2.685µs
2024-02-08T11:54:11.709+0100	WARN	statemgr	stmgr/forks.go:196	STARTING migration	{"height": "200", "from": "bafy2bzaceaoeushcv6ryvdwzlhuckwbx43oelzbdjjpomzghornghcd4bt7li"}
manifest cid: bafy2bzacec7m4klgyekusdfdfluyzudr247vquxayubhnddv3obexvpeg3jba
2024-02-08T11:54:11.709+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Started 9 workers
2024-02-08T11:54:11.709+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Creating migration jobs
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Result writer started
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Done creating 17 migration jobs after 81.253µs
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 8 done
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 4 done
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 6 done
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 7 done
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 3 done
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 2 done
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 0 done
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 5 done
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 1 done
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Result writer wrote 17 results to state tree after 202.562µs
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	All workers done after 200.358µs
2024-02-08T11:54:11.710+0100	INFO	fil-consensus	filcns/upgrades.go:2418	All 17 done after 233.43µs (72827/s)
2024-02-08T11:54:11.724+0100	WARN	statemgr	stmgr/forks.go:210	COMPLETED migration	{"height": "200", "from": "bafy2bzaceaoeushcv6ryvdwzlhuckwbx43oelzbdjjpomzghornghcd4bt7li", "to": "bafy2bzacecuw4rzvqccflkawdcjkp4veuxhy2et3v2eawq5gqlk3vzk26svyw", "duration": 0.014238705}
-----------
**Drand-switch at epoch 300:**
1259943, "cronGas": 15749626, "vmFlush": 0.022021152, "epoch": "300", "tsk": "{bafy2bzacecpfm25nhwjtrk3z7xnubhhj6rccnwivqfmz6fviro7vvxv34qc3q}"}
2024-02-08T12:00:49.043+0100	INFO	chain	chain/sync.go:625	block validation	{"took": 0.009254609, "height": "301", "age": 0.043214859}
2024-02-08T12:00:49.043+0100	INFO	chainstore	store/store.go:683	New heaviest tipset! [bafy2bzacebbfebzisraxruvk5qf4hh4suouwf2rvneppmfvbpxxvoa2imcp52] (height=301)
2024-02-08T12:00:49.049+0100	INFO	chain	chain/sync_manager.go:234	selected sync target: [bafy2bzacebbfebzisraxruvk5qf4hh4suouwf2rvneppmfvbpxxvoa2imcp52]
2024-02-08T12:00:49.049+0100	INFO	chain	chain/sync_manager.go:316	worker 299 syncing in [bafy2bzacebbfebzisraxruvk5qf4hh4suouwf2rvneppmfvbpxxvoa2imcp52]
2024-02-08T12:00:49.049+0100	INFO	chain	chain/sync_manager.go:324	worker 299 done; took 2.995µs
[DEBUG]<StorageMiner::1000> storage provder f01000 burning 0.0
[DEBUG]<StoragePower::4> ProofValidationBatch was nil, quitting verification
  • Test Drand switch when going from nv21 --> nv22 in a 2k-network with multiple clients (Lotus and Forest)
  • Test Drand switch when going from nv21 --> nv22 in the butterfly network

After reset of the butterfly-network:

lotus state actor-cids
Network Version: 21
Actor Version: 12
Manifest CID: bafy2bzacectxvbk77ntedhztd6sszp2btrtvsmy7lp2ypnrk6yl74zb34t2cq

Upgrade height 200:

2024-02-28T19:39:00.114+0100	WARN	statemgr	stmgr/forks.go:196	STARTING migration	{"height": "200", "from": "bafy2bzacea44wjrtpm2qmynntxrnwtoofbmy4nhdjhakfmnrq4rtaweh7am7u"}
manifest cid: bafy2bzacebk6jok5fx77hg4fx65evyomz7odpums6si47oasexgyenid6dphw
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Started 9 workers
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Result writer started
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Creating migration jobs
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Done creating 24 migration jobs after 165.943µs
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 6 done
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 0 done
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 2 done
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 8 done
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 7 done
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 5 done
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 4 done
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 3 done
2024-02-28T19:39:00.114+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Worker 1 done
2024-02-28T19:39:00.115+0100	INFO	fil-consensus	filcns/upgrades.go:2418	All workers done after 368.614µs
2024-02-28T19:39:00.115+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Result writer wrote 23 results to state tree after 405.915µs
2024-02-28T19:39:00.115+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Running deferred migrations
2024-02-28T19:39:00.115+0100	INFO	fil-consensus	filcns/upgrades.go:2418	Running deferred migration for t05
2024-02-28T19:39:00.116+0100	INFO	fil-consensus	filcns/upgrades.go:2418	All 24 done after 2.103141ms (11412/s)
2024-02-28T19:39:00.138+0100	INFO	chain	chain/sync_manager.go:234	selected sync target: [bafy2bzaceaq6fnuktl2rnhdw22ndggijzmgiryfzupzqreknkoo6qttoerof6]
2024-02-28T19:39:00.138+0100	INFO	chain	chain/sync_manager.go:316	worker 379 syncing in [bafy2bzaceaq6fnuktl2rnhdw22ndggijzmgiryfzupzqreknkoo6qttoerof6]
2024-02-28T19:39:00.144+0100	WARN	statemgr	stmgr/forks.go:206	COMPLETED migration	{"height": "200", "from": "bafy2bzacea44wjrtpm2qmynntxrnwtoofbmy4nhdjhakfmnrq4rtaweh7am7u", "to": "bafy2bzacebsftuvukiaekppqejivhqdqdxoepiamtqgnnligyfl2naj7kmq6y", "duration": 0.030165002}
----------------------------------
lotus state actor-cids
Network Version: 22
Actor Version: 13
Manifest CID: bafy2bzacebk6jok5fx77hg4fx65evyomz7odpums6si47oasexgyenid6dphw

Drand-switch at epoch 320:

2024-02-28T20:38:30.145+0100	INFO	consensus-common	consensus/compute_state.go:304	ApplyBlocks stats	{"early": 0.000002895, "earlyCronGas": 0, "vmMsg": 0.027843265, "msgGas": 0, "vmCron": 0.003234563, "cronGas": 47009891, "vmFlush": 0.005857653, "epoch": "320", "tsk": "{bafy2bzaceatqoo6qm6zetk3ykeu3ffmyyitqeebi5unb6kzeu2pfkxga63ivm,bafy2bzacedrt67cockn6w2lofods3kzu6yg7kdmkxlcauqusibbwgvxma77be}"}
2024-02-28T20:38:30.150+0100	INFO	chain	chain/sync.go:625	block validation	{"took": 0.019671653, "height": "321", "age": 0.15075138}
2024-02-28T20:38:30.151+0100	INFO	chainstore	store/store.go:683	New heaviest tipset! [bafy2bzacectoalcsj4lrh6odouryeyz4aea3w7jxonc5bcbb6jahvzvho6frw] (height=321)
2024-02-28T20:38:30.156+0100	INFO	chain	chain/sync.go:625	block validation	{"took": 0.048458696, "height": "321", "age": 0.156141984}
2024-02-28T20:38:30.156+0100	INFO	chain	chain/sync.go:625	block validation	{"took": 0.03069714, "height": "321", "age": 0.156133117}

General:

  • Sync calibration network with the new Drand code
    • Confirm WinningPoSt is ✅
  • Sync mainnet with the new Drand code
Mainnet is syncing fine with the release/v1.26.0 branch
  • Confirm WinningPoSt is ✅

Snapshot:

  • After the switch to Quicknet - import and sync a node with a snapshot taken before nv22.
    In the Butterfly-network, @ height 60 export a snapshot.car
lotus chain export --recent-stateroots=900 --skip-old-msgs /root/snapshot.car

Importing the snapshot:

2024-02-28T20:48:14.902+0100	WARN	chainstore	store/snapshot.go:160	Only able to load the last 58 tipsets
 9.39 MiB / 9.39 MiB [=================================================================================================================================================] 100.00% 95.80 MiB/s 0s
2024-02-28T20:48:14.910+0100	INFO	chainstore	store/store.go:724	clearing block validation cache...
2024-02-28T20:48:14.925+0100	INFO	chainstore	store/store.go:763	826 block validation entries cleared.
2024-02-28T20:48:14.925+0100	INFO	main	lotus/daemon.go:599	setting genesis
2024-02-28T20:48:14.926+0100	INFO	chainstore	store/store.go:683	New heaviest tipset! [bafy2bzacedfeatx2nkgsfk5j2njre62t62x6ahvadtllxburyaunxdp33dgju] (height=0)
2024-02-28T20:48:14.938+0100	WARN	chainstore	store/store.go:711	no previous heaviest tipset found, using [bafy2bzacedfeatx2nkgsfk5j2njre62t62x6ahvadtllxburyaunxdp33dgju]
2024-02-28T20:48:14.943+0100	INFO	main	lotus/daemon.go:622	accepting [bafy2bzaceccrjxrqotyb2llljm7cbizhepnduel65wlxwv35t7eh2ah5ocwiq bafy2bzacebjadsgmu7us2mzrosb4quzgx4gtu63i443rmkuvnq3negcoipxwq] as new head
2024-02-28T20:48:14.943+0100	WARN	chainstore	store/store.go:575	(!!!) forcing a new head silently; new head: [bafy2bzaceccrjxrqotyb2llljm7cbizhepnduel65wlxwv35t7eh2ah5ocwiq bafy2bzacebjadsgmu7us2mzrosb4quzgx4gtu63i443rmkuvnq3negcoipxwq]
2024-02-28T20:48:14.955+0100	WARN	chainstore	store/store.go:668	reorgWorker quit
2024-02-28T20:48:14.966+0100	INFO	badgerbs	v2@v2.2007.4/db.go:1027	Storing value log head: {Fid:0 Len:30 Offset:10211543}

2024-02-28T20:48:14.988+0100	INFO	badgerbs	v2@v2.2007.4/levels.go:1000	[Compactor: 173] Running compaction: {level:0 score:1.73 dropPrefixes:[]} for level: 0

2024-02-28T20:48:15.013+0100	INFO	badgerbs	v2@v2.2007.4/levels.go:962	LOG Compact 0->1, del 1 tables, add 1 tables, took 24.919742ms

2024-02-28T20:48:15.013+0100	INFO	badgerbs	v2@v2.2007.4/levels.go:1010	[Compactor: 173] Compaction for level: 0 DONE
2024-02-28T20:48:15.013+0100	INFO	badgerbs	v2@v2.2007.4/db.go:550	Force compaction on level 0 done
Chain import complete, halting as requested...

Hit a bug during the migration - has been fixed in this PR: #11663. Non-critical for the network upgrade (bug that has been around for a 1year), and not Drand related, but nice to backport.

Migration worked fine after cherry-picking that PR.

  • After the switch to Quicknet - export and sync a node with a snapshot taken after nv22.

Exporting a snapshot @ height 1739:

1739: (Feb 29 08:27:30) [ bafy2bzacebee44veb7d56j77rslq6wuu45ltmfr6pexoux6bxr3stzxnjtli6: t01002,bafy2bzaceavd4dcv6bldgcb65cc7s3hazzpeaqjqnpdek36qokfcymovmeu5q: t01001,bafy2bzacead4b6aide7iuc6ra6msjptxrbauvzgmanzntcfeuhbgmv2dlivti: t01000, ]
lotus chain export --recent-stateroots=900 --skip-old-msgs snapshot2.car

Importing snapshot:

2024-02-29T08:30:48.771+0100    INFO    main    lotus/daemon.go:559     importing chain from /root/snapshot2.car...
 37.28 MiB / 37.28 MiB [===============================================================================================================================] 100.00% 185.75 MiB/s2024-02-29T08:30:49.096+0100    WARN    chainstore      store/snapshot.go:160   Only able to load the last 1726 tipsets
 37.28 MiB / 37.28 MiB [=============================================================================================================================] 100.00% 97.28 MiB/s 0s
2024-02-29T08:30:49.155+0100    INFO    chainstore      store/store.go:724      clearing block validation cache...
2024-02-29T08:30:49.165+0100    INFO    chainstore      store/store.go:763      4117 block validation entries cleared.
2024-02-29T08:30:49.165+0100    INFO    main    lotus/daemon.go:599     setting genesis
2024-02-29T08:30:49.165+0100    INFO    chainstore      store/store.go:683      New heaviest tipset! [bafy2bzacedfeatx2nkgsfk5j2njre62t62x6ahvadtllxburyaunxdp33dgju] (height=0)
2024-02-29T08:30:49.171+0100    WARN    chainstore      store/store.go:711      no previous heaviest tipset found, using [bafy2bzacedfeatx2nkgsfk5j2njre62t62x6ahvadtllxburyaunxdp33dgju]
2024-02-29T08:30:49.176+0100    INFO    main    lotus/daemon.go:622     accepting [bafy2bzacebd4xxomoifjphocoxka36wqpir26gf62gbmxcsp5x5stxlfszcbu bafy2bzacebbeixdcpyguxgzs7c4bqdqqzo7vwte3rntduapukhr3tjaczg7jk] as new head
2024-02-29T08:30:49.176+0100    WARN    chainstore      store/store.go:575      (!!!) forcing a new head silently; new head: [bafy2bzacebd4xxomoifjphocoxka36wqpir26gf62gbmxcsp5x5stxlfszcbu bafy2bzacebbeixdcpyguxgzs7c4bqdqqzo7vwte3rntduapukhr3tjaczg7jk]
2024-02-29T08:30:49.188+0100    WARN    chainstore      store/store.go:668      reorgWorker quit
2024-02-29T08:30:49.200+0100    INFO    badgerbs        v2@v2.2007.4/db.go:1027 Storing value log head: {Fid:0 Len:30 Offset:40413715}
2024-02-29T08:30:49.228+0100    INFO    badgerbs        v2@v2.2007.4/levels.go:1000     [Compactor: 173] Running compaction: {level:0 score:1.73 dropPrefixes:[]} for level: 0
2024-02-29T08:30:49.262+0100    INFO    badgerbs        v2@v2.2007.4/levels.go:962      LOG Compact 0->1, del 1 tables, add 1 tables, took 34.187645ms
2024-02-29T08:30:49.262+0100    INFO    badgerbs        v2@v2.2007.4/levels.go:1010     [Compactor: 173] Compaction for level: 0 DONE
2024-02-29T08:30:49.262+0100    INFO    badgerbs        v2@v2.2007.4/db.go:550  Force compaction on level 0 done
Chain import complete, halting as requested...

Start the daemon:

2024-02-29T08:32:01.408+0100    INFO    chain   chain/sync.go:790       Got blocks: 1747 6
2024-02-29T08:32:05.186+0100    INFO    basichost       basic/natmgr.go:112     DiscoverNAT error:no NAT found
2024-02-29T08:32:05.500+0100    INFO    consensus-common        consensus/compute_state.go:304  ApplyBlocks stats       {"early": 3.978643838, "earlyCronGas": 19596344, "vmM
sg": 0.006167246, "msgGas": 0, "vmCron": 0.001899919, "cronGas": 19808533, "vmFlush": 0.005644402, "epoch": "1741", "tsk": "{bafy2bzacebd4xxomoifjphocoxka36wqpir26gf62gbmxcs
p5x5stxlfszcbu,bafy2bzacebbeixdcpyguxgzs7c4bqdqqzo7vwte3rntduapukhr3tjaczg7jk}"}
2024-02-29T08:32:05.506+0100    INFO    chain   chain/sync.go:625       block validation        {"took": 4.077961521, "height": "1742", "age": 185.506231967}
2024-02-29T08:32:05.506+0100    INFO    chain   chain/sync.go:625       block validation        {"took": 4.077923679, "height": "1742", "age": 185.506240984}
2024-02-29T08:32:05.511+0100    INFO    chain   chain/sync.go:625       block validation        {"took": 4.083248038, "height": "1742", "age": 185.511520609}
2024-02-29T08:32:05.511+0100    INFO    chain   chain/sync.go:1118      fetching messages at 0
2024-02-29T08:32:05.631+0100    INFO    chain   chain/sync.go:1160      fetching messages for 6 tipsets at 0 done; took 119.439485ms

@rjan90
Copy link
Contributor Author

rjan90 commented Jan 17, 2024 •
edited
Loading

Testing plan: Add built-in Actor events in the Verified Registry, Miner and Market Actors

See testing coverage in: #11673

@rjan90 rjan90 pinned this issue Jan 17, 2024
@Reiers
Copy link

Reiers commented Jan 24, 2024

Testing plan: DDO - Direct Data Onboarding

General

  •  Daemon stays in sync
  •  Onboard a regular deal sector via Boost check that events gets emitted.
  •  Onboard a FIL+ sector via Boost check that events gets emitted.
  • Snap up a sector, and check that events gets emitted.

Events for miner:

  • Sector update (SnapDeals / ProveCommit)
  • Sector update batching (ProveReplicaUpdates3)
  • Sector activation

Events for market:

  • Deal published
  • Deal activated

@magik6k
Copy link
Contributor

magik6k commented Jan 24, 2024 •
edited by Reiers
Loading

Testing plan: DDO Integration tests

General

  • Test a sector with Fil+ DDO piece
  • Test a sector with a (fil+) DDO piece + custom market actor??
  • Test the above with a piece filling half of a sector, and CC padding
  • Test the above with two pieces and inter-piece padding (p1-PP-p2-p2)
  • Test the above with snapdeals sector

@magik6k magik6k mentioned this issue Jan 24, 2024
Merged
7 tasks
@jennijuju
Copy link
Member

jennijuju commented Jan 30, 2024 •
edited
Loading

@Reiers
we will cover event testing including the DDO case!

Test a sector with a (fil+) DDO piece + custom market actor??

The custom market actor isn't supported yet fyi

@jennijuju
Copy link
Member

jennijuju commented Jan 30, 2024 •
edited by Reiers
Loading

@magik6k @Reiers a couple DDO things to test and confirm

  • one f05 and one DDO data ends up in the same sector, and the f05 deal can be found in f05 state and is fully activated
  • one can aggregate multiple DDO sectors
  • test a sector with multiple DDO pieces
  • 🔴 test a sector with multiple DDO pieces, some with data cap and some without
    ( Boost doesn't support DDO without DC )
  • test a couple fil+ deals (non f05) arriving at a similar time to lotus miner, one has a allocation duration of 180days and the other has 3.5 years
  • same as above but using snap
  • the existing deal making pathway still works

DDO Deals are not going into sectors;

lotus filplus list-claims t01000
ClaimID  Client  Provider  Data                                                              Size     TermMin  TermMax  TermStart  
18       1006    1000      baga6ea4seaqietozq5nuxycauasf24tghksdgmug5t7lbceqvd63xldfdnxg2jq  262144   3153600  5256000  2766       
  Sector: 53
3        1006    1000      baga6ea4seaqpzbckilvcjc4irekbibas3y4244wef6sjsqunc7dek65pzfip6la  8388608  518400   777600   800        
  Sector: 4
1        102     1000      baga6ea4seaqhrgxzwxr6paq6puxgpoj5hkcfnd2jy6i7odsercvarupxj7qecdq  8388608  3675537  3934737  0          
  Sector: 0
17       1006    1000      baga6ea4seaqietozq5nuxycauasf24tghksdgmug5t7lbceqvd63xldfdnxg2jq  262144   518400   5256000  2766       
  Sector: 53

ID   State      OnChain  Active  Expiration                    Deals  DealWeight  VerifiedPower  
0    Proving    YES      YES     3675537 (in 24 weeks 1 day)   CC                                
1    Proving    YES      YES     607911 (in 3 weeks 6 days)    1      2KiB                       
2    Proving    YES      YES     608396 (in 3 weeks 6 days)    CC         <-- DDO Deal                       
3    Proving    YES      YES     608611 (in 3 weeks 6 days)    1      2KiB                       
4    Proving    YES      YES     608606 (in 3 weeks 6 days)    1      8MiB        72MiB          
5    Proving    YES      YES     608760 (in 3 weeks 6 days)    CC                                
6    Proving    YES      YES     608771 (in 3 weeks 6 days)    CC <--- Short and Long Term DDO in the same sector
53   Proving    YES      YES     3156970 (in 20 weeks 5 days)  CC   <--- Snap DDO Deal                                                         

CreatedAt	2024-02-29 17:58:39.206 (3h ago)
Allocation ID	18
Client Address	t3u5q2eyhsl7b3665cqre3wz4sdluflgbmmff4fqzpyivrvlyuwqoi6wz6vhwgj3t6azngw7323czy4eouxw6q
Verified	No
Keep Unsealed Copy	Yes
Announce To IPNI	Yes
Piece CID	baga6ea4seaqietozq5nuxycauasf24tghksdgmug5t7lbceqvd63xldfdnxg2jq
Piece Size	256 KiB (262,144 bytes)
Current Epoch	5,316
Start Epoch	3,500 (2h)
End Epoch	3,157,100 (in 5 months)
Duration	3,153,600 (146 days)
Inbound File Path	/app/public/bafykbzacecxhno7yggojb57kee3otb5mvykpynucjidzm5a5zmbm72qsuuo2s.car
Delete After Add Piece	No
Sector ID	53

Screenshot 2024-02-29 at 21 44 13

@rjan90 rjan90 unpinned this issue Apr 25, 2024
@rjan90 rjan90 closed this as completed Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
FilOz
Status: ☑️Done (Archive)
Milestone
No milestone
Development

No branches or pull requests
5 participants
@magik6k @rjan90 @jennijuju @Reiers and others