Pinning Service API

[lanzafame]

A Pinning Service is a service that accepts hashes from a user and will host the associated hash, i.e. Pinata, Infura, et al.

The rationale behind defining a pinning service API is to have a baseline functionality and interface that can be provided by these services so that tools can be built on top of a common base of functionality.

Draft pinning service api: https://app.swaggerhub.com/apis/lanzafame/ipfs-pinning-service/v0.0.0

Latest draft: https://app.swaggerhub.com/apis/lanzafame/ipfs-pinning-service/0.0.1

//cc @obo20 @MichaelMure

Anyone who knows someone running an IPFS pinning service, please tag them here, thanks. @parkan

[agentofuser]

It would be very helpful for ipfs-deploy if pinning services had a common API.

My impression is that this was going to be sort of standardized by https://github.com/ipfs-shipyard/cube, but I don't know what's the status on that project.

[lanzafame]

@agentofuser Yes, ipfs-deploy is a perfect example of the kind of tooling that would greatly benefit from this common API definition 👍.

The Cube project is building a pinning service that is easy to deploy for you and your friends/family. It has it's own pinning service API definition and is another potential adopter of this common API.

[lanzafame]

Sidenote: Utilizing DID in the security/user-id component of the API would be ideal.

[parkan]

➡️ a huge 👍 to this in general

➡️ a lot of object store services to date have taken to being S3 api compatible, we probably can't follow this 1:1 but I wonder if being nearly-compatible or at least inspired by would help folks migrate? in particular, stuff like legal holds probably encodes enough real-world experience to be worth looking at

➡️ this may be out of scope for this issue, but an alternate "API" I envision is a "re-pin everything I pin" service: one way this could work would be following a pubsub channel that announces new items

[lanzafame]

this may be out of scope for this issue, but an alternate "API" I envision is a "re-pin everything I pin" service: one way this could work would be following a pubsub channel that announces new items

This is something that Cluster is looking to provide but it is beyond what I envision being provided by a straight IPFS pinning service. It also has a lot of technical and security challenges.

a lot of object store services to date have taken to being S3 api compatible, we probably can't follow this 1:1 but I wonder if being nearly-compatible or at least inspired by would help folks migrate? in particular, stuff like legal holds probably encodes enough real-world experience to be worth looking at

Not opposed to this idea, I would still want to define this API as a way of clarifying the language around what pinning is and its different forms/options so that developers in the IPFS world have a common language to talk about it with.

[bonedaddy]

Having started to work with gRPC more, I think if the idea is to have a unified pinning service API that can be implemented across multiple providers, gRPC is a must have for a few reasons:

• The API can be defined in a single language, with reproducible ways of building clients+servers across a wide variety of language, and overall reduced time for developers involved whether they be those that are creating the API, or those that are using the API.
• Documentation can't get out of date due to the self-generating nature of protocol buffers + gRPC. Any update to the design of the API can be reflected in the API documentation automatically, or manually with very few commands. This i think is a major feature to have as across the IPFS ecosystem (official projects, non official projects, etc..) have issues with documentation being up-to-date. This is easily explained by the ecosystem moving so quickly that documentation is hard to keep updated, so anything that makes this easier is a must have imo
• Swagger integration
• Allows room for "implementation specific features" (ie, company A can implement all features of the API, while company B can choose to only implement 50% of the features). Protobuffs and gRPC handle this extremely easy
• I suppose GraphQL could be used, but it's slower than gRPC
• Protocol buffers are already being used in some libp2p/ipfs projects
• Enable service-interoperability between multiple service providers easily, since one's grpc server could also run a small client alongside it to forward pin requests to other providers and not have to worry about missing feature compatibility due to gRPC amazingness

These are my initial thoughts for now, however if I have more i'll update this post.

[lanzafame]

@postables I agree that gRPC would be awesome and it does seem to be coming along in this space, see Improbable's grpc-web blog post and gRPC's web protocol. My biggest issue with gRPC is that they don't allow for custom transports yet™.

Did you have any thoughts on the actual API defined here: https://app.swaggerhub.com/apis/lanzafame/ipfs-pinning-service/v0.0.0? In terms of usability? or how it integrates with your payment system? or the use of JWT to identify who pinned a cid?

[parkan]

my $0.02: broadly speaking, I'm hugely in favor of well-defined, testable, and documented APIs that 3rd parties can implement

however, gRPC/protobuf might be excessive for an API of limited complexity, small message size (aside from files themselves), and relatively low throughput requirements

you can still get the validation upside just by using swagger, and the logic duplication is pretty mild for an API of this cale

[bonedaddy]

@lanzafame Thanks for the links i'll check them out.

As for initial thoughts on the API that's define so far, it looks quite nice, and would be pretty easy to integrate with as is. The JWT in my system uses id instead of user_id so my personal preference would be to have id, however there isn't anything wrong with user_id and represents a fairly trivial switch for my backend to handle id.

Additionally I believe some other blockchain based authentication systems also use JWTs sousing JWT's in general is a good idea, as it helps encourage interoperability with different types of auth systems.

As for the payment system, my only suggestion would be to add 409 as a valid response status code that can be returned for all the POST /pins and POST /pins/{cid} calls, to indicate that a payment is required for this particular action. In the case of my API this is sent as a response code when a user doesn't have enough balance in the account to cover the storage costs. But generally speaking, this might be a good status code to have to allow various pinning service providers to handle payments in their normal ways.

The other suggestions I have are:

• Make 500 internal server error a response code that's part of the API
• POST /pins incorrectly lists 500 as an unauthorized status code.
• Make 401 unauthorized as response code that's part of the API

Those minor nits aside, this is off to a great start already 👌

@parkan That's a valid point and I suppose if the API never goes beyond it's current scope in functionality gRPC/protobuf would definitely be excessive

[lanzafame]

@postables I have updated the API with your corrections and suggestions, thanks.
And if anyone is wondering about the 409 error it is a throwback to RedAlert 2 😄.

    InsufficientFunds:
      description: Insufficient Funds (409)
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'

[bonedaddy]

Excellent 👌 thanks for implementing the suggestions, I think this gives a lot of room for overlaying this API ontop of existing implementations different service providers have.

[ghost]

Grpc using multiple transports is not ready and I suspect will come out around the same time a the final RFC for http/3 which is July. So one month.

I the meantime you can use grpc and convert it forward and back at design time using gnostic.
Google made this and I suspect for the same use cases / architectural reuse use cases.

So you can define thing in grpc and share the API to non golang projects over openapi. Those users can then gen a proxy client from the openapi spec into whatever language they are using.

The last bit is data validation. Open api is rich and uses JSON schema. So the client can do client side validation. I think gnostic supports translation of grpc data validation but not sure since I was last using it.

Grpc over quic ( uni- directional ) and grpc over webrtc ) bi directional ) will make things very easy. Pions have a great webrtc golang stack now with good momentum.

Ultimately grpc generators that support these new transports for other languages will be years away. Just how it is

Quic and Webrtc are both likely to get browser full support do that client will be in good shape.
However your always going to be stuck for other servers, desktops and mobiles that want client proxy code in their languages.
So gnostic and openapi will always be needed. OpenAPI suggests REST and HTTP and so all the push aspects are lost and so hooks need to be designed in to the base architecture ?

[bonedaddy]

@lanzafame As mentioned via email, I'm happy to get started on an open-source implementation of the Pinning Service API, and implement it within Temporal.

However before I get started it's probably best worth deciding on how it gets implemented. Since there isn't any consensus yet on whether or not to implement gRPC, I'm wondering if its best to start implementing the API using go-chi/chi. Although the nice thing with gRPC is we can actually tackle both a gRPC, and HTTP implementation of the Pinning Service API using https://github.com/grpc-ecosystem/grpc-gateway.

[lanzafame]

@postables that is awesome! thanks very much.

I think the interface should be defined in HTTP terms for now as not everyone supports communicating via gRPC on the server side, i.e. HTTP proxy<->gRPC service. So if Temporal supports gRPC, which I believe it does, you are free to use grpc-gateway, as long as the HTTP API that it produces is compatible with the swagger spec. This way we can move forward with this now and then once gRPC gets native support for other transports, we can reassess and go with gRPC in the future.

Also, I was just looking at the openid spec and it has a standard claim for the user id so I will be updating user_id to sub.

[yangwao]

I've just seen your lighting talk from IPFS camp 2019 and you have a perfect idea that could rise more pinning services around! Looking forward :)
🎥 https://www.youtube.com/watch?v=zdAvXTpBIfc&list=PLuhRWgmPaHtQVNQcBaCKg5kKhfOBv45Jb&index=12

[bonedaddy]

@lanzafame Awesome, I'm about 90% certain that it should be possible to craft gRPC code that will result in an identicile HTTP API spec. If not then I'll just stick to HTTP API with go-chi. I've been a little more busy then expected this last week, and will get started on this during the upcoming weekend.

[lanzafame]

@postables I have updated the API spec with the changes to JWT claims as mentioned above: https://app.swaggerhub.com/apis/lanzafame/ipfs-pinning-service/0.0.1.

[bonedaddy]

Perfect that is very helpful. I have one thing I wanted to mention, while exploring code generation capabilities using go-swagger, I noticed that all of the API routes, have the path of lanzafame/ipfs-pinning-service/versionNumber as seen here.

Now I can quite easily change this to a different API path by altering the url here. However since the idea is that this is an interoperable API, I'm not sure whether or not I should do that.

Should we keep with the lanzafame/ipfs-pinning-service routes, or should we use a more generic one?

[lanzafame]

I have added a second entry to the servers list so that the generator doesn't use the mock api in the routes. I haven't version bumped the spec, as it hasn't actually changed but it has been updated.

[bonedaddy]

I noticed another weird thing with how the Cid model is being created

package api

type Cid struct {

	_ string `json:"/,omitempty"`
}

Here is the openapi type definitions

    Cid:
      type: object
      properties:
        /:
          type: string

It appears that for some reason the properties of the Cid type are creating components of Golang structs with the name _. Is this something to keep or should this be changed? I'm not sure about other languages but I guess this could potentially be something that may cause an issue?

For the metadata type definitions, they get generated with names:

    Metadata:
      type: object
      additionalProperties:
        type: string
        `

type Pin struct {

	Cid Cid `json:"cid,omitempty"`

	Metadata map[string]string `json:"metadata,omitempty"`

	Replication int32 `json:"replication,omitempty"`
}

Update: Yup the name of the _ parameter in the Cid type is causing issues.

	var request models.Pin
	if err := c.BindJSON(&request); err != nil {
		// if this fails, it means the user sent an invalid request
		// and we were unable to bind to the Pin model correctly
		a.handleError(c, http.StatusBadRequest, err.Error())
		return
	}
	request.Cid. // we can't access the element, element, because it is named _ and therefore unexported (this is only a problem if the models are stored in another "package" which I'm currently doing)

[obo20]

Hey all! Apologies for being so late to hop into this thread.

Great idea @lanzafame. I like the idea of having some sort of standard API that all pinning services can adhere to. That being said it will probably take some time to get right as pinning services are still evolving to meet the changing needs of users.

A couple of initial thoughts:

• This appears to be purely centered around pinning by hash. I don't see endpoints for uploading content itself. While it is good to start small, many of our users use our API to directly upload their content and never upload anywhere else. Do you have plans to expand this to handling direct content uploads or is this purely aimed at being a "hash management" type API? Essentially, what's the "scope" of this proposed API?

• This could get fairly tricky when accounting for the difference in pinning service payment strategies. What are the thoughts on how to deal with payment strategies? For example, here's a few different strategies that I've seen already:

  • Subscription tier model (you pay $5 a month to get up to x number of pins / and y number of GB stored)
  • Credits system (you buy credits ahead of time and tell the service how long you want to pin each item)
  • Pay as you go (you pin your content and get charged on a monthly basis based on how much you're storing)

[lanzafame]

@obo20 Thanks for taking a look.

Do you have plans to expand this to handling direct content uploads or is this purely aimed at being a "hash management" type API? Essentially, what's the "scope" of this proposed API?

I am happy to include an /add endpoint, I have taken a quick crack at it, deriving most of it from the IPFS /add endpoint:

/add
    *     arg [file]: The path to a file to be added to ipfs. Required: yes.
    *     recursive [bool]: Add directory paths recursively. Required: no.
    *     hidden [bool]: Include files that are hidden. Only takes effect on recursive add. Required: no.
    *     trickle [bool]: Use trickle-dag format for dag generation. Required: no.
    *     wrap-with-directory [bool]: Wrap files with a directory object. Required: no.
    *     chunker [string]: Chunking algorithm, size-[bytes] or rabin-[min]-[avg]-[max]. Default: "size-262144". Required: no.
    *     raw-leaves [bool]: Use raw blocks for leaf nodes. (experimental). Required: no.
    *     cid-version [int]: CID version. Defaults to 0 unless an option that depends on CIDv1 is passed. (experimental). Required: no.
    *     hash [string]: Hash function to use. Implies CIDv1 if not sha2-256. (experimental). Default: "sha2-256". Required: no.

My main aim with the options I have left is to allow the user to decide exactly how the DAG is structured if they wish. Essentially, they should be able to know how to add the file/directory on their machine and get the same CID.

As for the payment side of things, the only strategy that you mentioned that has an impact is the Credits system as it requires a duration and I can't think of a nice way to get around that. Any thoughts?

[lanzafame]

@postables Sorry, I have been on holidays the last few weeks and am just catching up with everything. How do you think it would be best to resolve the Cid issue, I wrote the swagger spec from the JSON output perspective but if that isn't generating the correct code, then I am happy to correct it.

[lidel]

IPFS GUI Team is looking into adding native Pinning Services support to ipfs-webui / ipfs-desktop and ipfs-companion in Q3.

@lanzafame what is the latest status of the Pinning API spec? Is the swagger prototype prefered form?

Is there a repo where we can PR changes to the swagger prototype?
If not, is it ok to create ipfs-shipyard/pinning-services-api-spec and continue related spec and ecosystem work there?

The timeline is pretty aggressive and we'd like to finalize the spec in next 2-3 weeks (provide feedback and invite stakeholders

[lanzafame]

@lidel ipfs-shipyard would be an excellent place to continue this conversation and iteration. There isn't currently a repo so just copy across the spec from swagger hub and continue.

As for the latest status, I am not quite sure other than what is here, there has been a lot of discussion around it that I have not been able to keep up with due to life intervening.

[lidel]

Ok, let's continue in https://github.com/ipfs/pinning-services-api-spec
(already imported swagger spec there, and started internal meetings this week, will reach out to pinning services next week and ask for feedback there)

@lanzafame mind closing this issue?

[lidel]

👉 https://github.com/ipfs/pinning-services-api-spec ✨