feat: lotus-provider: SDR Sealing pipeline #11534

magik6k · 2023-12-17T11:50:30Z

Related Issues

Proposed Changes

Additional Info

This will be a longer-running branch for the SDR sealing pipeline in lotus-provider

TODO:

Checklist

Before you mark the PR ready for review, please make sure that:

Commits have a clear commit message.
PR title is in the form of of <PR type>: <area>: <change being made>
- example: fix: mempool: Introduce a cache for valid signatures
- PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
- area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
If the PR affects users (e.g., new feature, bug fix, system requirements change), update the CHANGELOG.md and add details to the UNRELEASED section.
New features have usage guidelines and / or documentation updates in
- Lotus Documentation
- Discussion Tutorials
Tests exist for new functionality or change in behavior
CI is green

lib/harmony/harmonydb/sql/20231217.sql

provider/lpseal/task_sdr.go

cmd/lotus-provider/pipeline.go

documentation/en/cli-lotus-provider.md

provider/lpffi/sdr_funcs.go

snadrus · 2024-01-16T22:46:44Z

provider/lpseal/task_porep.go

+	max int
+}
+
+func NewPoRepTask(db *harmonydb.DB, api PoRepAPI, sp *SealPoller, sc *lpffi.SealCalls, maxPoRep int) *PoRepTask {


Please comment what I need one of these for.

snadrus · 2024-01-16T22:54:17Z

provider/lpseal/task_submit_precommit.go

+}
+
+func (s *SubmitPrecommitTask) CanAccept(ids []harmonytask.TaskID, engine *harmonytask.TaskEngine) (*harmonytask.TaskID, error) {
+	id := ids[0]


These CanAccept()s are not picky enough to "enable everywhere and see what happens".
Can you add a bug+TODO on them?

It would be nice to default-enable most jobs in the future and have the system balance itself with CanAccept rather than hand-tuning.
We should be thinking about where on disk the various dependencies are if we want less data movement in a world with multiple sealing machines.

So only tasks which care about data locality are:

SDR, which actually doesn't care about where it is started, because it has no large input data

SDRTrees, which needs all the data from SDR, so that one ideally would be smart about where it runs; At minimum we should make it try to run on the same node that SDR ran on in the first place if that is possible, but even that imo shouldn't be a blocker for this PR landing

Finalize, which really needs to run on the same machine that SDRTrees ran on (it already does that)

MoveStorage, which needs to run on a node with long-term storage, ideally "close" to the node that SDRTrees ran on

Tasks like SubmitPrecommit and others don't even touch the data files, so they don't really need to run on any specific node.

That said it would be really nice to have some standard way in CanAccept to express some priority with which the machine "wants" to run a task, ideally without any waiting being involved.

Thanks.
In discussing an order of work assigned, the order of the tasks put into harmonytask.New() is the order the poller assigns the work. Therefore having very-cheap tasks first is good, but having heavier tasks in a last-to-first order ensures that work later in the pipeline gets picked-up before work earlier in the pipeline.

cmd/lotus-provider/tasks/tasks.go

snadrus · 2024-01-16T22:57:36Z

provider/lpseal/task_trees.go

+	max int
+}
+
+func NewTreesTask(sp *SealPoller, db *harmonydb.DB, sc *lpffi.SealCalls, maxTrees int) *TreesTask {


please comment what it does. Could these could link to a diagram?

provider/lpweb/hapi/routes.go

snadrus

a few little things

cmd/lotus-provider/cli.go

snadrus · 2024-02-21T01:27:28Z

cmd/lotus-provider/config_new.go

+
+var configNewCmd = &cli.Command{
+	Name:      "new-cluster",
+	Usage:     "Create new coniguration for a new cluster",


When do I use this? Where do I get SP actor addresses?

Right now the command to do that is lotus-shed miner create, I do want to create a separate tool for managing the miner actor, so then it will become sptool create

The commands that will live in sptool (vs curio / lotus-provider) would be the ones which interact with the miner actor, so things like create, managing balances, owner/control addresses, and other on-chain actions could be grouped in that binary.

(I just don't want to do all this in this PR, it's way too big already)

Lets refer to an open bug here with these goals in mind, that way it has a continue story for watchers.

snadrus · 2024-02-21T01:30:01Z

cmd/lotus-provider/config_new.go

+			}
+		}
+
+		if cctx.Bool("overwrite") {


if this is a new cluster, how about we encourage "base"?

No strong opinion on this one, want to handle this one given that you're also handling the other onboarding paths? (lets us not block this PR)

I'm handling this elsewhere. Just add a few comments on what you're expectations are so I can do the right thing when I integrate this code with the other migration.

cmd/lotus-provider/storage.go

snadrus · 2024-02-21T01:42:01Z

cmd/lotus-shed/actor.go

+
+var actorSetPeeridCmd = &cli.Command{
+	Name:      "set-peer-id",
+	Usage:     "set the peer id of your miner",


what's the peer ID used for? Should we mention that?

This is market node libp2p peer ID, which is where you send deal proposals and generally talk with the miner node.

Belongs in sptool is the future, when we build that tool (also imo this should just be automagically called by boost / market node on setup, but it's not..)

(This is just a basic lotus-shed tool, those are really only meant for developer use, this one I just needed for devnet boost testing)

provider/lpseal/task_sdr.go

provider/lpseal/task_trees.go

snadrus · 2024-02-21T03:49:30Z

provider/lpseal/task_trees.go

+
+	// If 'active' is nil, initiate the HTTP request
+	if u.active == nil {
+		resp, err := http.Get(u.Url)


should we have some kind of recovery here for over-the-Internet flakiness?

range request?

We can swap this for a more robust http requester later. I have one in RIBS, and we can borrow at least some of it.

That said currently the lotus-shed boost proxy command won't support interrupted streams (that is a problem and it definitely needs to be fixed, but may require boost modifications), and currently there isn't a good way to start deals with data fetched straight from the internet.

I wouldn't block this PR on this for now, but it should be fixed together with improved boost<->curio adapter code.

provider/lpweb/hapi/robust_rpc.go

snadrus · 2024-02-21T04:01:10Z

storage/paths/db_index.go

+		id)
+	if err != nil {
+		//return xerrors.Errorf("updating storage health in DB fails with err: %v", err)
+		if harmonydb.IsErrSerialization(err) {


is this replaced by the serialization option?

Yes, I wrote this code before adding the option; Will change

Ah, this is on db.Exec, not transaction; Should be done, but can be a follow up PR

magik6k commented Dec 17, 2023

View reviewed changes

lib/harmony/harmonydb/sql/20231217.sql Outdated Show resolved Hide resolved

magik6k force-pushed the feat/lp-seal branch 2 times, most recently from 5c2bb1a to a4435d2 Compare December 18, 2023 20:43

magik6k commented Dec 19, 2023

View reviewed changes

provider/lpseal/task_sdr.go Outdated Show resolved Hide resolved

magik6k force-pushed the feat/lp-seal branch 5 times, most recently from 6cd75fe to aa5d1ce Compare December 23, 2023 19:36

magik6k force-pushed the feat/lp-seal branch from 78eecea to 90cf182 Compare January 4, 2024 16:01

magik6k force-pushed the feat/lp-seal branch 3 times, most recently from e1fc241 to b471e16 Compare January 16, 2024 19:06

snadrus requested changes Jan 16, 2024

View reviewed changes

magik6k force-pushed the feat/lp-seal branch from 6f49143 to f3d604a Compare January 22, 2024 16:59

magik6k force-pushed the feat/lp-seal branch 2 times, most recently from 9e49e34 to e3d0b21 Compare February 16, 2024 19:38

magik6k added 13 commits February 16, 2024 21:20

begin drafting lotus-provider SDR pipeline

67fb3bd

lpseal: Sector number allocator, pipeline start cli

e89fff2

scayyold the SDR task

bb42a70

lpseal: Implement SDR for CC sectors

44b441d

lpseal: Fix sdr do query

4eb7bc9

lpseal: Wire up sdr task

5d26c3a

lpseal: Fix poller

04f5a59

lpseal: Working SDR!

e19b533

lpseal: store ticket after sdr

5eccc03

lpseal: scaffold sdr-trees task

d79bc0b

lpseal: Implement SDR-Trees

f74c3cc

storage: Fix dbindex heartbeat

47672e0

lpseal: Fix makePhase1Out properly

c63ecba

magik6k added 8 commits February 19, 2024 08:47

lotus-provider: Use timeouts from config

e0f8408

lpdeal: Make booster- retrievals work

648ac7a

harmony: BeginTransaction Retry Serialization option

94a31c6

lpdeal: Use serialization retry option

f2ebde4

lpweb: Cache api blockstore in sectors page

534db8d

lpmarket: Fix result output in case of tx retry

4808b49

lotus-provider: EnvVar for layers

8f29310

fix imports

df5ac9d

magik6k marked this pull request as ready for review February 20, 2024 16:07

magik6k requested a review from a team as a code owner February 20, 2024 16:07

magik6k requested review from snadrus and arajasek February 20, 2024 16:07

magik6k added 2 commits February 20, 2024 17:35

lpwindow: Fix the test command

859dbf5

lpseal: Correctly set expiration epochs when deals are present

f62dedc

snadrus approved these changes Feb 21, 2024

View reviewed changes

magik6k added 6 commits February 21, 2024 10:46

lpseal: Fix precommit poller hang

edbff6b

lp: Address review

91d3ede

lpproof treed: defer workWg.Done

3b51653

provider: Fix sql query capitalization

4ae73f0

provider: Expand provider subsystems config docs

2eaf34b

update docsgen

a586982

snadrus approved these changes Feb 21, 2024

View reviewed changes

magik6k mentioned this pull request Feb 22, 2024

feat: lotus-provider: Storage reservations in SDR #11643

Closed

8 tasks

magik6k changed the title ~~[wip] feat: lotus-provider: SDR Sealing pipeline~~ feat: lotus-provider: SDR Sealing pipeline Feb 22, 2024

magik6k added 3 commits February 22, 2024 11:39

Fix lint

0634dfc

Merge remote-tracking branch 'origin/master' into feat/lp-seal

848c132

fix mac build

5ab75d3

magik6k merged commit cb67eeb into master Feb 22, 2024
88 checks passed

magik6k deleted the feat/lp-seal branch February 22, 2024 16:10

magik6k mentioned this pull request Feb 28, 2024

feat: curio deal cache (Piece Park) #11664

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: lotus-provider: SDR Sealing pipeline #11534

feat: lotus-provider: SDR Sealing pipeline #11534

magik6k commented Dec 17, 2023 •

edited

snadrus Jan 16, 2024

snadrus Jan 16, 2024

magik6k Feb 12, 2024

snadrus Feb 21, 2024

snadrus Jan 16, 2024

snadrus left a comment

snadrus Feb 21, 2024

magik6k Feb 21, 2024

snadrus Feb 21, 2024

snadrus Feb 21, 2024

magik6k Feb 21, 2024

snadrus Feb 21, 2024

snadrus Feb 21, 2024

magik6k Feb 21, 2024

magik6k Feb 21, 2024

snadrus Feb 21, 2024

snadrus Feb 21, 2024

magik6k Feb 21, 2024

snadrus Feb 21, 2024

magik6k Feb 21, 2024

magik6k Feb 21, 2024

feat: lotus-provider: SDR Sealing pipeline #11534

feat: lotus-provider: SDR Sealing pipeline #11534

Conversation

magik6k commented Dec 17, 2023 • edited

Related Issues

Proposed Changes

Additional Info

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snadrus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magik6k commented Dec 17, 2023 •

edited