IPFS Node topology-based pinning strategy

[segator]

In some cases you would want to choice specific nodes to pin replication. Think the case you use JBOD and have 1 IPFS Daemon per disk, and also have multiple machines.
You would want to avoid ipfs cluster replicate a file in same machine diferent disks, you would want always to pin the file in diferent hosts and even maybe diferent region hosts.
I was looking into the documentation but I dind't found anything and could be awesome, because I am afraid to use a overlayFS over multiple drives as a single IPFS Repository, if one disk fail you could lose all the reopsitory located in the machine and you will need to redownload it again. In the case I say, you should only need to redownload the failed drive.

[hsanjuan]

Yes, currently we do not have a pinning strategy that provides this, but it's easy to do.

I mark this issue as ready so it can be worked on.

[hsanjuan]

Implementation hints:

• An new informer should provide a custom metric "group". Probably picked from config file.
• An allocator should prioritize hosts among different groups in the priority list. Discuss: should it fail when replication factor > different groups?
• When deciding which host to pick from a group, it might be worth to sort them by free-space
• This opens the question of sorting based on secondary metrics/allocator, or having metrics report multiple values. Might be worth to make Metric value an array of values to order.

[segator]

well reading the documentation, I see you could use a cluster as a daemon so you could group nodes creating diferent clusters.

1- Agree
2- Depends on the case, maybe you want prioritze replication to your group but if a node is offline on your group go to the next choice, I will implement something similar as use CEPH with CRUSH maps.
3-I will define diferents strategies, most-free, fill-up first with host weight or random. if you think about HDD power saving you always prefer to have some group of files in the same disk.(because de HDD spin down)
4-Complexity don't needed now but it's fine to think about.

[hsanjuan]

True, you can do cluster-composition, where one of the cluster peers of a cluster appears as an ipfs node to a different cluster. The higher cluster would use "pin everywhere" replication, and each sub-cluster would be a region (for example) with replication factor 1 (for example). This would make a pin to be replicated in every region, and for each region, one of the available nodes would be selected. It adds a layer of complexity though.

[hsanjuan]

@segator, most-free strategy is the default