Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pinned Content in Swarm #1274

Closed
nagydani opened this issue Feb 26, 2019 · 4 comments

Comments

@nagydani
Copy link

@nagydani nagydani commented Feb 26, 2019

Introduction

Nodes in Swarm may wish to retain certain chunks for various reasons.
For example, they might want to keep certain content available in Swarm
either because their operator owns it or because they have been paid to
do so. This ticket is for the design of an extension to Swarm's local database
that would facilitate the management of such content.

The problem with the naive implementation of simply flagging pinned content
is that it is not clear under what conditions can the flag be removed.
Instead of flags, we can use reference counters, whereby chunks with a
non-zero reference count can be considered pinned/flagged. Fortunately,
Swarm references, both encrypted and unencrypted are guaranteed to be
cycle-free. Thus, reference counters can be initialized by resetting all
to zero and then simply traversing references from a given root while
increasing the reference counter for each encountered chunk. Similarly,
dereferencing requires a decrement at each chunk encountered during a
traversal.

For practical reasons, I suggest that stickiness (the property of being pinned)
is defined by accessibility from a single root, stored locally. Thus, we can
effectively reuse already existing tools for managing sticky content.

Consistency

Unfinished traversals can leave the database in an inconsistent state.
However, since concurrent traversals do not result in race conditions,
if the state of all unfinished traversal is available, they can simply
be finished, thus restoring consistency. The state of a traversal is
described by a root reference and a path descriptor. In the path
descriptor we find ordinal numbers for references found in intermediate
chunks as well as those found in manifests. In both cases, in addition
to unambiguously marking the location of the reference, we need to
indicate whether or not that particular reference is encrypted or not.

While optimizations are obviously possible, the simplest way of updating
reference counters in case of change of root reference is increasing
counters in a tranversal from the new root and then dereferencing the
old root.

If the states of ongoing traversals is lost, consistency can be restored
by setting all reference counters to zero and traversing from the root.

In order not to require a full scan of all chunks, reference counters
can have a label identifying the consistency epoch. If a new epoch is
started, reference counters with the old label should be treated as zero.

@zelig zelig added this to Backlog in Swarm Core - Sprint planning via automation Feb 26, 2019
@zelig zelig moved this from Backlog to To do in Swarm Core - Sprint planning Feb 28, 2019
@acud acud closed this Apr 19, 2019
Swarm Core - Sprint planning automation moved this from To do to Done Apr 19, 2019
@acud acud reopened this Apr 19, 2019
Swarm Core - Sprint planning automation moved this from Done to To do Apr 19, 2019
@FantasticoFox FantasticoFox moved this from To do to Backlog in Swarm Core - Sprint planning May 23, 2019
@holisticode holisticode referenced this issue May 28, 2019
2 of 2 tasks complete
@acud acud moved this from Backlog to Meta in Swarm Core - Sprint planning May 28, 2019
@holisticode

This comment has been minimized.

Copy link
Contributor

@holisticode holisticode commented May 28, 2019

@homotopycolimit

This comment has been minimized.

Copy link

@homotopycolimit homotopycolimit commented Jun 3, 2019

Let us continue this discussion at swarmresear.ch
https://swarmresear.ch/t/pinned-content-in-swarm/18

@FantasticoFox FantasticoFox moved this from Epics to To do in Swarm Core - Sprint planning Jun 11, 2019
@jmozah jmozah referenced this issue Jun 23, 2019
@FantasticoFox

This comment has been minimized.

Copy link

@FantasticoFox FantasticoFox commented Jul 12, 2019

This should be a user-story which contains different issues. This was created as an issue. This should move into this board https://github.com/orgs/ethersphere/projects/8

@FantasticoFox FantasticoFox moved this from To do to In review (includes Documentation) in Swarm Core - Sprint planning Jul 19, 2019
acud added a commit that referenced this issue Aug 7, 2019
* add support for pinning content as per the specifications defined initially in #1274 and as further discussed in https://swarmresear.ch/t/pinned-content-in-swarm
@acud

This comment has been minimized.

Copy link
Contributor

@acud acud commented Aug 7, 2019

done by the one and only @jmozah

@acud acud closed this Aug 7, 2019
Swarm Core - Sprint planning automation moved this from In review (includes Documentation) to Done Aug 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
7 participants
You can’t perform that action at this time.