Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pins should support a replication factor #41

Closed
hsanjuan opened this issue Feb 7, 2017 · 9 comments
Closed

Pins should support a replication factor #41

hsanjuan opened this issue Feb 7, 2017 · 9 comments
Assignees
@hsanjuan
Labels
kind/enhancement A net-new feature or improvement to an existing feature
Milestone
Replication factor

Comments

@hsanjuan
Copy link
Collaborator

hsanjuan commented Feb 7, 2017

A Pin should not need to be pinned in every cluster member. We should be able to say that a pin needs to be pinned in 2, 3 cluster members.

We will start with a general replication factor for all pins, then maybe transition to replication factor per-pin.

These are thoughts for the first approach.

Replication factor -1 means Pin everywhere. If replication factor is larger than the number of clusters then it is assumed to be as large.

Pinning

We need a PeerMonitor component which is able to decide, when a pin request arrives, which peer comes next. The decision should be based on pluggable modules: for a start, we will start with one which attempts to evenly distribute the pins, although it should easily support other metrics like disk space etc.

Every commit log entry asking to Pin something must be tagged with the peers which are in charge. The Pin Tracker will receive the task and if it is tagged itself on a pin it will pin. Alternatively it will store the pin and mark it as remote.

If the PinTracker receives a Pin which is already known, it should unpin if it is no longer tagged among the hosts that are in charge of pinning. Somewhere in the pipeline we probably should detect re-pinnings and not change pinning peers stupidly.

Unpinning

Unpinning works as usual removing the pin only where it is pinned.

Re-pinning on peer failure

The peer monitor should detect hosts which are down (or hosts whose ipfs daemon is down).Upon a certain time threshold ( say 5 mins, configurable). It should grep the status for pins assigned to that host and re-pin them to new hosts.

The peer monitor should also receive updates from the peer manager and make sure that there are no pins assigned to hosts that are no longer in the cluster.

For the moment there is no re-rebalancing when a node comes back online.

This assumes there is a single peer monitor for the whole cluster. While monitoring the local ipfs daemon could be done by each peer (and triggering rebalances for that), if all nodes watch eachothers this will cause havoc when triggering rebalances. The Raft cluster leader should probably be in charge then. But this conflicts with being completely abstracted from the consensus algorithm below. If we had a non-leader-based consensus we could assume a distributed lottery to select someone. It makes no sense to re-implement code to choose a peer from the cluster when Raft has it all. Also, running the rebalance process in the Raft leader saves redirection for every new pin request.

UX

We need to attack ipfs-cluster-ctl to provide more human readable outputs as the API formats are more stable. status should probably show succinctly which pins are underreplicated or peers in error, 1 line per pin.
@hsanjuan hsanjuan added the kind/enhancement A net-new feature or improvement to an existing feature label Feb 7, 2017
@hsanjuan hsanjuan added this to the Replication factor milestone Feb 7, 2017
@hsanjuan hsanjuan self-assigned this Feb 7, 2017
@hsanjuan hsanjuan added the status/ready Ready to be worked label Feb 7, 2017
@meyerzinn
Copy link

I'd like to be clear. Is this pinning blocks or pinning merkel dags that describe files? I.e., say I use ipfs add shakespeare.txt and get a hash there representing the file--if I add that to the cluster, will the cluster distribute the file in the way you describe or distribute the individual blocks?

Also, can this support clusters where nodes have limits that vary?

@hsanjuan
Copy link
Collaborator Author

hsanjuan commented Feb 7, 2017 •
edited by Kubuxu
Loading

@20zinnm right now its not pinning blocsk individually. It's equivalent to ipfs pin add. It will distribute the file.

Also, can this support clusters where nodes have limits that vary?

yes, it should not be difficult to implement custom strategies for different limits.

// fixed formatting ~ Kubuxu

@hsanjuan hsanjuan added status/in-progress In progress and removed status/ready Ready to be worked labels Feb 8, 2017
@meyerzinn
Copy link

@hsanjuan I've been doing some reading concerning the Raft consensus algorithm you use in ipfs-cluster, and I wonder why a model like this wouldn't work:

  1. Leader receives request to pin IPFS file.
  2. Leader commits a message like 'ADD [hash]' or something.
  3. Followers receive the request and break it down to individual blocks.
  4. Followers then decide whether or not they have the capacity to keep specific blocks.
  5. Should a follower decide to keep a block, it proposes a change to the log with the block hash (and if in an untrusted environment a proof of retrievability 1).
  6. The leader will commit it (or, if in an untrusted environment, will commit it after verifying the proof).
  7. Followers prioritize blocks to keep based upon how many others are included in the state as keeping the block.

This would allow for fine-grained distribution and replication at the block level to maximize the utilization of peers and allow them the ability to chose whether or not to take blocks. This is advantageous because it allows for heterogeneous peers (my raspberry pi with a gigabyte and a rack in the cloud with a petabyte can participate on the same cluster).

@meyerzinn meyerzinn mentioned this issue Feb 8, 2017
Closed
@hsanjuan
Copy link
Collaborator Author

hsanjuan commented Feb 9, 2017

Hello @20zinnm ,

As it was pointed out on irc, while your proposal offloads the tracking decision to the nodes, it does not allow to provide balancing strategies where comparison between nodes are needed (i.e. geographically or network-topology based decisions).

Regarding block-level tracking, as I said, on this first approach we will not do it. A number of reasons were also provided on irc. I think you can accommodate your use case by sending the block hashes to cluster rather than the main hash, as cluster does not make distinctions. Would that work for you?

hsanjuan added a commit that referenced this issue Feb 13, 2017
@hsanjuan
WIP: Issue #41: Add PeerManager, Allocator, Informer components
2cbb243 
This adds the possibility of using a replication factor is now supported.

First, cluster peers collect and push metrics (Informer) to the Cluster
leader regularly. The Informer is an interface that can be implemented
in custom wayts to support custom metrics.

Second, on a pin operation, using the information from the collected metrics,
an Allocator can provide a list of preference as to where the new pin
should be assigned. The Allocator is an interface allowing to provide
different allocation strategies.

Both Allocator and Informer are Cluster Componenets, and have access
to the RPC API.

The allocations are kept in the shared state. This is still missing
re-allocations (although re-pinning something when a node is down -
metrics missing) should effectively re-allocate content.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
hsanjuan added a commit that referenced this issue Feb 13, 2017
@hsanjuan
WIP: Issue #41: Add PeerManager, Allocator, Informer components
d93e445 
This adds the possibility of using a replication factor is now supported.

First, cluster peers collect and push metrics (Informer) to the Cluster
leader regularly. The Informer is an interface that can be implemented
in custom wayts to support custom metrics.

Second, on a pin operation, using the information from the collected metrics,
an Allocator can provide a list of preference as to where the new pin
should be assigned. The Allocator is an interface allowing to provide
different allocation strategies.

Both Allocator and Informer are Cluster Componenets, and have access
to the RPC API.

The allocations are kept in the shared state. This is still missing
re-allocations (although re-pinning something when a node is down -
metrics missing) should effectively re-allocate content.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
hsanjuan added a commit that referenced this issue Feb 14, 2017
@hsanjuan
Issue #41: Add Replication factor
cf7b607 
New PeerManager, Allocator, Informer components have been added along
with a new "replication_factor" configuration option.

First, cluster peers collect and push metrics (Informer) to the Cluster
leader regularly. The Informer is an interface that can be implemented
in custom wayts to support custom metrics.

Second, on a pin operation, using the information from the collected metrics,
an Allocator can provide a list of preferences as to where the new pin
should be assigned. The Allocator is an interface allowing to provide
different allocation strategies.

Both Allocator and Informer are Cluster Componenets, and have access
to the RPC API.

The allocations are kept in the shared state. Cluster peer failure
detection is still missing and re-allocation is still missing, although
re-pinning something when a node is down/metrics missing does re-allocate
the pin somewhere else.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
hsanjuan added a commit that referenced this issue Feb 14, 2017
@hsanjuan
Issue #41: Add Replication factor
2512ecb 
New PeerManager, Allocator, Informer components have been added along
with a new "replication_factor" configuration option.

First, cluster peers collect and push metrics (Informer) to the Cluster
leader regularly. The Informer is an interface that can be implemented
in custom wayts to support custom metrics.

Second, on a pin operation, using the information from the collected metrics,
an Allocator can provide a list of preferences as to where the new pin
should be assigned. The Allocator is an interface allowing to provide
different allocation strategies.

Both Allocator and Informer are Cluster Componenets, and have access
to the RPC API.

The allocations are kept in the shared state. Cluster peer failure
detection is still missing and re-allocation is still missing, although
re-pinning something when a node is down/metrics missing does re-allocate
the pin somewhere else.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
@hsanjuan hsanjuan mentioned this issue Feb 14, 2017
Merged
@hsanjuan hsanjuan added need/review Needs a review and removed status/in-progress In progress labels Feb 14, 2017
@hsanjuan
Copy link
Collaborator Author

PR is in.The re-pinning on host failure has been moved to #45.

@hsanjuan hsanjuan removed the need/review Needs a review label Feb 15, 2017
@0zAND1z
Copy link

0zAND1z commented Feb 13, 2019

What is the default replication factor in an ipfs-cluster?

Would love to know that.

@lanzafame
Copy link
Contributor

lanzafame commented Feb 13, 2019 via email

@0zAND1z
Copy link

0zAND1z commented Feb 13, 2019

Thanks.

What's the range for these replication factor variables here?

I see the https://cluster.ipfs.io/documentation/configuration/#the-cluster-main-section describing the default values for replication factor, but not the range of values.

@lanzafame
Copy link
Contributor

lanzafame commented Feb 13, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hsanjuan hsanjuan

Labels
kind/enhancement A net-new feature or improvement to an existing feature
Projects
None yet
Milestone
Replication factor
Development

No branches or pull requests
4 participants
@hsanjuan @lanzafame @meyerzinn @0zAND1z