Release Deployment with ipfs-cluster

[rvalle]

CI/CD with ipfs-cluster

As a decentralized app developer I would like to integrate ipfs-cluster in my continuous delivery process.

A release will generate a deliverable that translates into a CID and then be pinned.
It is then common practice to deploy deliverables to different environments (beta, stage, production). At the same time, deployments can be upgraded as stakeholders validate the deliverables during revisions, beta to stage, stage to production, etc.

It should also be easy to unpin previous deployments as they get overridden by new deliverables.

The current state

When trying to integrate a CD process with ipfs-cluster in its current state a number of issues arise:

The intuitive approach is to designate a PIN per environment. For example, a single pin will be used for each environment and will be NAMED as PROJECT-ENVIRONMENT, i.e. PINS: www.ipfs.io-beta www.ipfs.io-stage www.ipfs.io-production. Subsequent deployments will then update those PINs, cool!

This does not work for a number of issues:

• First is that Names are just a Label, would be nice to have the name behave like a primary key of the PIN, or introduce a PIN id/key.
• Another issue that arises is that it is not possible to PIN the same CID several times (already mentioned in add the ability pin the same CID multiple times (with different names) #1182) which is a common practice during a delivery revision process: i.e. Stage deployment is approved and then deployed to production.

Proposed enhancements

I think, that in order to get CD/CI releases to ipfs-cluster nicely integrated, the following would be required:

• add the ability pin the same CID multiple times (with different names) #1182 Allow multiple PINS to the same CID
• Introduce PIN id, and allow to ipfs-cluster-ctl pin update --id my-pin-id CID which also involves un-pinning the previous CID when this was the only pin on that CID

I think with those 2 changes everything would work.

[rvalle]

@hsanjuan In line with the required analysis:

Actually, I have run this workflow for some time now. As a workaround I did add a timestamp file with the deployment date, which assures all CIDs are different even when the same release is deployed on 2 different environments. This is a nasty workaround but allows me to test further.

I have also implemented some scripting to get pins by name, and unpins by name.

Un-pinning causes problems with DNS TTLs. DNS records need time to propagate and un-pinning at release time will break your deployment until DNS changes propagate across the internet.

I have introduced a workaround instead of un-pinning that could be called "forgetting". Basically before updating the pin I create another with the old CID named "PROJECT-ENVIRONMENT-deleting" and set a >TTL expiry time.

That would be an ideal modifier for the pin update command, like:

ipfs-cluster-ctl pin update --id my-pin-id --forget 1d CID

That would update a pin by its ID and then would keep the previous content pinned with an expiry date of 1 day.

[hsanjuan]

Have you considered co-locating several clusters peers, one for each environment you need?

Pinning something several times, with different names etc. is an architectural change that is not going to happen anytime soon.

Regarding your pin update proposal, I looks like the same is achieved with "pin update old new" + pin add old --expire 1d".

[rvalle]

@hsanjuan no, we have not considered duplicating clusters per project x deployment we would end up with 10s of clusters if not 100 or more. I don't think that is sustainable.

The bottom line of this issue is that it does not seem feasible to realistically manage pins with the current cluster functionality.

CIDs are random, and Names are only labels.

If you analyze ipfs.io website publishing pipeline you will see that pins are not been deleted as new releases are created. So, it is not only me, I think.

I am obviously not familiar with he current ipfs-cluster architecture, but something is missing. I guess there might be a simpler solution to the problem.

[hsanjuan]

realistically manage pins

"Managing pins" is something that is not very concrete. In your case, managing pins means tracking different outputs of different pipelines and make sure that some of them are pinned and some deleted. In a pinning provider, it may be tracking which users are pinning what. etc. Cluster's primary role in the "managing pins" story (as things are now) is to manage CID replication across daemons.

That doesn't mean I don't want to make your life managing pins easier, but I have to be careful about scope creep and complexity.

I do give a lot of thought to "how pinning the same cid multiple times" could work (just like I do to other things like access control etc), but I see this touches everything from APIs to pinset storage layer so it is no "quick fix" and I can only say up front that you should not expect this happening asap.

And sometimes what is going to solve the problem best is a custom layer running on top of cluster (that, for example, keeps the name-cid mappings as you more or less started doing).

[rvalle]

@hsanjuan yes, writing a custom layer to manage name-cid pairs is an obvious choice for the current state, I agree.

I am not seeking for a solution ASAP, but to contribute to IPFS in the long term. My current workarounds do for now.

Perhaps it is not pinning the same CID multiple times what we need but being able to associate one or more unique IDs with a PIN. Perhaps a solution can be provided at the pin metadata level.

Honestly (the grace of not being familiar with the codebase) I did not think that this feature request would have significant impact... :)