Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slot reservations #162

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Slot reservations #162

wants to merge 6 commits into from

Conversation

markspanbroek
Copy link
Member

@markspanbroek markspanbroek commented May 11, 2023

Describes a mechanism that allows hosts to reserve a slot by depositing collateral early. The main advantage of this is that it eliminates duplicated effort in the network; instead of multiple hosts competing to provide a proof of storage, only a single host will download the content and calculate a proof of storage.

This mechanism has been discussed in the Codex team on various occasions. This is my attempt at capturing it in a design.

I'm not entirely convinced that we should adopt this design, however. My main doubts are around the added complexity in the smart contract (suggestions to simplify the design are very welcome!), and I'm not entirely convinced that the dispersal mechanism cannot be utilized to provide the same functionality. Thoughts?

Closes #158.


Slot reservations could however provide a relatively cheap way for malicious
hosts to sabotage storage requests, by reserving a slot and never filling it. To
avoid this we burn the collateral of all reserved slots when the request times
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens in the case of repair when there is no request expiry? Will the sliding Kademlia address window be enough of a deterrent?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question, I think we should add a maximum duration for repair. When the host fails to repair within that time, it also loses its collateral. What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point @emizzle, I would say @markspanbroek solution should work 👍 I guess we will make it as Contract's constructor parameter?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would the max duration be checked? It would require a transaction.

The max duration might need to be a function of the rate at which the sliding window expands (or inversely the rate which the sliding window expands might need to be a function of the max duration) because it feels like we should have allowed x% of the network to attempt to repair the slot before we close off the ability to repair the slot.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we will make it as Contract's constructor parameter?

The max duration for repair would depend on several factors, such as the data size, the amount of redundancy, the overall capacity in the network for repair, or even the sliding window as @emizzle mentioned. So I would add it as a parameter to the storage request, for the client to determine. Better to have that complexity in the node than in the smart contract.

How would the max duration be checked? It would require a transaction.

Good point. When the host has not sent in a storage proof within the maximum duration for repair, it won't be able to withdraw its collateral. Also, a new host is allowed to reserve the slot after the maximum duration, at which point the collateral of the previous host can be burned. The main drawback of this is that there won't be a nice SlotFreed event when the maximum duration is reached, which makes it harder for hosts to act on it.

Copy link
Member

@AuHau AuHau Jun 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The max duration for repair would depend on several factors, such as the data size, the amount of redundancy, the overall capacity in the network for repair, or even the sliding window as @emizzle mentioned. So I would add it as a parameter to the storage request, for the client to determine. Better to have that complexity in the node than in the smart contract.

I guess makes sense, although couldn't we just simply reuse the request expiry duration also for repair expiry? Maybe increased by some coefficient (which could be contract-wide) to offset potential additional recovery/recomputation of data? IMHO it could be used for now and if we find later on the need to have it separated into its own parameter than we can add it later on...

I mean the request expiry already should incorporate in its calculation variables like data size, redundancy etc...

The "overall capacity in the network for repair" is also IMHO something that is not possible to estimate prior to repair is really needed.

The main drawback of this is that there won't be a nice SlotFreed event when the maximum duration is reached, which makes it harder for hosts to act on it.

The SlotFreed is currently emitted thanks to Validators that check the proofs were submitted. Maybe they could also check if the reserved slot was indeed filled? Provided that we would offer them a cut on the burned collateral?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean the request expiry already should incorporate in its calculation variables like data size, redundancy etc...

Not exactly, when there is very little redundancy (n-k is small) then there are only a few hosts that have the data necessary for repair. This could slow down repairs. But the total amount of data is smaller, so starting the contract could be faster.

The SlotFreed is currently emitted thanks to Validators that check the proofs were submitted. Maybe they could also check if the reserved slot was indeed filled? Provided that we would offer them a cut on the burned collateral?

That is a good idea. In a way they'd be looking for additional missing proofs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The max duration for repair would depend on several factors, such as the data size, the amount of redundancy, the overall capacity in the network for repair, or even the sliding window as @emizzle mentioned. So I would add it as a parameter to the storage request, for the client to determine. Better to have that complexity in the node than in the smart contract.

This may be too detailed for now and a problem we can address later, but we should at least consider bounding the max duration parameter in the contract to prevent clients from being able to shoot themselves in the foot. Too small of a value could cause slots to go unrepaired and the contract to fail. Too large could cause squatting. Also, if the sliding window is contract-based, the max duration parameter could be bounded by the sliding window rate to provide opportunity to a minimum % of the hosts for repair (as mentioned previously).

design/marketplace.md Outdated Show resolved Hide resolved
design/marketplace.md Outdated Show resolved Hide resolved
design/marketplace.md Outdated Show resolved Hide resolved
design/marketplace.md Outdated Show resolved Hide resolved
Co-authored-by: Eric Mastro <github@egonat.me>
Copy link
Member

@AuHau AuHau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👍


Slot reservations could however provide a relatively cheap way for malicious
hosts to sabotage storage requests, by reserving a slot and never filling it. To
avoid this we burn the collateral of all reserved slots when the request times
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point @emizzle, I would say @markspanbroek solution should work 👍 I guess we will make it as Contract's constructor parameter?

---------------- time ---------------->


Slot reservations could however provide a relatively cheap way for malicious
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think overall, these approaches have severe UX drawbacks and they expose both the client and the providers to serious loss, both in monetary and resources. I did an analysis around the initial contract interaction. I believe approach 2 from my writeup is the most flexible and has the best UX characteristics, we should seriously re-evaluate this approach in the same context.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed write-up. I think you're right. Allowing a host to reserve a slot without collateral makes the whole design simpler. Squatting of slots, which is the main downside of this approach, is discouraged by both the gas fees, and the expanding window mechanism. I think those should be enough of a deterrent. I have updated the document to reflect this design.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not so convinced about approach 2 and the deterrents, more here: #166 (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adam raised some good points about the strength of the deterrents to prevent squatting. If we could disincentivise client withholding, then perhaps approach 3 would be the best. I outlined a suggestion in the discussion.

Copy link
Contributor

@dryajov dryajov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, LGTM - however as expressed in https://github.com/codex-storage/codex-research/pull/162/files#r1218365268, I'm concerned with the potential UX implications during contract start and I believe a better way of addressing it exists, specifically approach 2 with some small adjustments. Please take a look at this writeup - #166.

markspanbroek and others added 2 commits June 6, 2023 11:58
See #166

Co-Authored-By: Dmitriy Ryajov <dryajov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Claim slot before downloading
5 participants