Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sealing data without a deal #3044

Closed
anorth opened this issue Jul 10, 2019 · 8 comments · Fixed by #3231 or #3263

Comments

@anorth
Copy link
Contributor

commented Jul 10, 2019

Description

The deal protocol is more or less independent of proving storage, and a miner logically can commit and prove storage without deals. Go-filecoin doesn't provide any interface to do that, so miners have to go through unnecessary self-dealing in order to prove storage.

We should add a CLI command that directly stages a piece of data for sealing, bypassing the deal flow.

Acceptance criteria

  • A miner can seal arbitrary miner-provided data without doing a storage deal
  • CLI commands enable the above

Risks + pitfalls

Where to begin

cc @acruikshank @laser

@anorth anorth added the A-storage label Jul 10, 2019
@steven004

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2019

As filecoin is to provide a decentralized storage market, and encourage people to make deals, that is the most important value of filecoin network, this feature is not strictly necessary.

However, since it is feasible based on the current implementation, this feature would be widely used by miners, especially in the early stage of filecoin network.

Based on the understanding, we could consider spending some time to contribute this by providing a save-data command while we have already done some study.

@anorth

This comment has been minimized.

Copy link
Contributor Author

commented Jul 17, 2019

Thanks @steven004, agreed. We'd welcome contributions from prospective miners for this.

@deltazxm

This comment has been minimized.

Copy link

commented Jul 19, 2019

I think this feature is very useful for early of filecoin network.

@anorth

This comment has been minimized.

Copy link
Contributor Author

commented Aug 15, 2019

#3231 supports manual trigger of sealing, creating a sector with a piece of dummy data if none exist. It doesn't support the operator choosing their own data, so I'm re-opening this to request that the sealing and piece-staging be exposed separately.

Not urgent, and would really welcome any contribution from the community.

@anorth anorth reopened this Aug 15, 2019
@acruikshank

This comment has been minimized.

Copy link
Contributor

commented Aug 15, 2019

@anorth Could you motivate the additional functionality?

We could add a mining add-piece command that would take the cid of an imported chunk of data and add it into a staged sector. This would allow a miner to add a data to their own storage. They wouldn't have the deal infrastructure to track the file, but as long as they track the CID separately, they could retrieve it.

It would probably be better if this command not automatically trigger sealing so that the miner could add more than one piece to try and fill the staged sector.

Is this what you're looking for?

@anorth

This comment has been minimized.

Copy link
Contributor Author

commented Aug 18, 2019

Yes, exactly.

My primary motivation is a general goal of providing orthogonal operations that a miner/operator can compose into their desired behaviour. To reduce the scope for go-filecoin, we are not attempting to implement every possible use case, but expect operators to write some code (either in go-filecoin or invoking go-filecoin) for all but the most basic. We have to implement the primitives for this to be feasible. This should result in less total work for go-filecoin. In this case, staging a piece and triggering sealing are orthogonal.

As a somewhat strained example, mining add-piece would support a miner obtaining and staging piece data from a client other than via bitswap, maybe because they have an agreement external to Filecoin, or just have negotiated a different exchange mechanism. And an idempotent seal-now that did not create and immediately seal an "empty" sector would be more useful for operators wishing to disable auto-seal and do their own scheduling, but maximise useful data stored.

I am not suggesting direct support for all use cases we can imagine: on the contrary I'm suggesting atomic primitive operations which others can compose into their use cases.

@acruikshank

This comment has been minimized.

Copy link
Contributor

commented Aug 19, 2019

Can we open another issue with that description then? I don't get why this one needs to be repurposed.

@ingar ingar self-assigned this Aug 19, 2019
@anorth

This comment has been minimized.

Copy link
Contributor Author

commented Aug 21, 2019

I think that was the original purpose! Just perhaps not clearly communicated.

A miner can seal arbitrary miner-provided data

@ingar ingar closed this in #3263 Sep 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.