feature: API pubkeys

[mikeal]

So, we have this very reasonable feature request (#560), people want to delete things.

The only problem is, a few people have been putting their API tokens in their web frontends. Not a lot, but enough of them that we don’t want to go and give those keys the ability to destroy user data and we doubt this pattern of behavior will go away no matter how many warnings we might try to put on it.

We also have another very reasonable feature request (#571), people want to do a one-time-use signature for an upload so they can guard their token in their backend without having to proxy the upload.

We’ve also wanted to get wallet auth for a long time (nftstorage/nft.storage#619) and we’ve made progress on a prototype for a MetaPlex integration (nftstorage/nft.storage#431) that is giving us a much better understanding of the developer ergonomics.

We also have a bunch of finer grained permissions we’d like to do that we’ve explored as token scoping (#314).

We’ve also had requests for a feature, similar to Pinata metadata, that would allow users to tag their uploads with metadata they can query against in the service.

Finally, we want to get an API going for mutable references and keep it open to multiple decentralized mutable naming/reference systems in the future (#511).

API PubKeys

• Users create a public/private keypair locally and add the public key to their account as an API PubKey.
• An API PubKey is tied to an account, same as a token is today.
• An API PubKey is given explicit permission grants for each service feature it can use:
  • PUT
  • DELETE
  • TAG:{NAMESPACE} (details below)
• The signer signs a message with even more fine grained permissions and tags (see below) that functions as a one-time-use token.
  • PUT:{CID}:{CARHASH}
  • DELETE:{CID}:{CARHASH}
  • TAG:{NAMESPACE}:{ USERNAME | USER WALLET ADDRESS | CHAIN | ETC }
  • TIMESTAMP

This resolves #571 (one-time-signature), #560 (deletes), and has pretty obvious future extensions to handle everything in #314 (token scoping).

Tagging

Since tags have to be in the signature they are secured by the developer’s private key and can’t be tampered with by the end-user.

The thing that sucks about tagging is that it’s in our database, which is not decentralized. But, we could just write this data to a chain. It’s quite small.

Since the TAG:{NAMESPACE} is an explicit permission grant the developer sets in their acocunt, we could allow developers to configure our service to write data into specific protocols set on each namespace.

I can easily imagine a feature in which we mint an NFT for each USER/TAG:{NAMESPACE} and use https://github.com/Gozala/ipnftx/blob/main/Readme.md to write this into the chain. Then developers can use the DID of the NFT as a reference rather than anything specific to our service, and if they ever leave the service we can just transfer the NFT to them. You could imagine doing the same thing if you wanted to attach an IPNS or ENS client to a tag namespace, but we’d need to do some key custody for that so we shouldn’t take it lightly (maybe next year).

I’ve been talking about these as “tags” but we could also have a feature called /last which would function as a mutable reference to the last tag.

In the integrations we do like Metaplex, we could embed the wallet address and signature and all the other data we want to use in order to control abuse in these endpoints that are open to any wallet signature.

---

This may sound like a bunch of features all jammed together, and therefor a lot of work, but it’s actually not. This is actually pretty simple and relatively boring security (we should consider just using hawk https://github.com/mozilla/hawk).

It also has some obvious incremental steps we can break it into and deliver user value along each step without getting stuck in a long timeline, and some good alignment with where we want to go long term (everything in the service is just a passthrough to a decentralized protocol or an index of it).

[hugomrdias]

looks really interesting, gonna look into it to be able to provide proper feedback.

[yusefnapora]

This does sound like it would solve a lot of problems...

It wouldn't be hard to re-work what I've got so far for the metaplex auth so that it fits this model, if I'm getting it correctly.

Basically I'm thinking something like this:

export type Permission = 'PUT' // in the future: 'DELETE' | 'TAG', etc
export type ResourceType = 'CID' // in the future: 'NAMESPACE', etc.
export type ResourceId = string

export interface RequestScope {
  permission: Permission
  resourceType: ResourceType
  resourceId: ResourceId
}

You'd include that in your request in the signed payload message, and the putCar handler would validate that permission == 'PUT' && resourceType == 'CID' and that resourceId equals the root CID of the car.

Then where I've currently got a ChainContext object that includes e.g. the name of the blockchain and the recent blockhash, those would just be tags, e.g.

const authPayload = {
  scope: {
    permission: 'PUT',
    resourceType: 'CID',
    resourceId: 'bafy123...'
  },

  tags: {
    blockchain: 'solana',
    network: 'mainnet',
    solanaRecentBlockhash: 'xyz...',
  },
  
  timestamp: '2021-10-29T18:26:34Z'
}

Does that sound right @mikeal?

[mikeal]

The issue description covers this feature from the point of view of features we want, but I’ll followup with a higher level view of the whole system and what it gives us.

The Problem

• We have a relationship with developers and they have a relationship with end users.
• We want to accept data from end users directly so that developers don’t have to proxy all data requests through a backend. They don’t want to do this so much that some of them put their API tokens in the web frontend.
• Almost all the features we want to add to the nft/web3.storage beyond accepting and making data available needs to be verified by the developer even though it likely needs to be sent along with the data from the end user.

The Solution

We bootstrap a simple protocol in a signed message. Developers add public keys to their account and set permission scopes. Developers put this private key on their private server and accept small requests from the web frontend (CID, CAR HASH, any other metadata). Since this is their private server they can validate this is a legitimate request from one of their users and that the user has permission to do certain tasks. Developers may also want to add additional metadata about that user to the upload for record keeping in the service (abuse detection, usage reports, etc). And we’ll integrate all of this into our libraries to make it very simple to set this up.

These signed messages are effectively a protocol for all future service features that we want secured by the developer rather than the end user. They are presented as a single one-time-use token, using off-the-shelf public key encryption. Not only does this avoid the work of integrating with more complex auth systems, it’s far more efficient than any system where we would need to validate these permissions or tokens against a central authority (it’ll actually be faster than our current magic.link auth).

We’ll need to spec out and design the message format. Because I’m me, I’m leaning towards a single IPLD Block, maybe just a CID with an identity multihash containing the full message, this would introduce maximal flexibility later, and wouldn’t necessarily require the client to include CBOR since we can encode with DAG-JSON (we aren’t storing these, no need to shave bytes off the block format).

I’m thinking we use a union where each key is a service API that is being called by the token.

{ put: [ [ cid, carMultihash ], [ “user:mikeal” /* tags */ ] ] }

{ list: [ cid, cursor, maxLength ] }

{ delete: [ cid, carMultihash ] }

Or, in IPLD Schema:

type Message union {
  | Put "put"
  | ListUploads "list"
  | CAR "delete"
} representation keyed

type Tags [ String ]
type CAR struct {
  root Link
  hash Bytes  // maybe we should add a multiformat for each CAR version so this can be a CID
} representation tuple

type Put struct {
  car CAR
  tags Tags  
} representation tuple

type ListUploads {
  cid Link
  cursor optional Integer
  cursor optional Integer
}

We’ll want to discuss where the best place for an API version would be. We have a few options:

• in the signed message as another top level union union { v1 Message “v1” } representation keyed
• in an extra field in the authorization header (https://github.com/mozilla/hawk/blob/main/API.md)
• in an extra header in the request

Putting it in the header means we can swap out the entire system, but then it’s not part of the signature.

[hhff]

This is rad @mikeal - but it’s not clear to me how the above would solve for mutable references. Is that something that tags would facilitate?

[gitaaron]

This feature sounds interesting. Would this let me as an app dev. interact with a wallet such as metamask and hand off the responsibility of verifying a user has permission to use the web3.storage service without getting in the middle?

It sounds like I would still have to verify the user is valid in my system which is something I am trying to avoid.

[mikeal]

@hhff exactly, we’d be able to expose an API for the “latest” item with a particular tag. best of all, if you configure those tags to be written to a blockchain we can make the read interface an index of the chain rather than relying on any internal (centralized) state in our database/API

@gitaaron there’s a bit of a chicken-and-egg problem we have to deal with when accepting data signed by wallets. users need a guarantee their data is persisted before they write references to that data to a chain, and we need some method of ensuring that this is really data being written to a chain by this user which won’t be verifiable until later on.

the approach we’ve been taking is to write integrations for different libraries and marketplaces that are minting NFT’s. these tools tend to have some data ready that they can share with us about the user and the upcoming transaction (beyond just a signature that could be any newly generated wallet address) and we’ve been accepting that data as valid auth mechanism on a one-off basis for each integration.

this new approach would standardize this quite a bit more. we’d still need to write integrations, or work with others to write integrations, that tag incoming data with the wallet address and any other information we can use for abuse control, but we’d use the same auth mechanism described here for each one and could standardize the tagging such that we make it easier for others.

we should get to the point where developers don’t need their own accounts or auth backend if they’re using known libraries and marketplaces because these will already be written. for instance, Holaplex already uses nft.storage, and we’re working on a Metaplex integration right now. so if you’re a Web3 developer working directly with wallet signatures you don’t have to run your own backend if you’re using these tools, and we’re working toward making this experience universal across every toolchain.

[mikeal]

After discussing with @Gozala, there’s a nice way we can accomplish the features laid out in this proposal with UCAN (https://fission.codes/blog/auth-without-backend/). It wouldn’t take any additional effort on our part and it would leave the door open to some future workflows that leverage permission delegation.

@Gozala is writing a demo and i’ll work on a product spec to break down exactly what we’d need to implement now vs later.

[gitaaron]

Thanks for the update, @mikeal !

Would this implementation require my users to have a fission account? If so, out of curiosity would you happen to know if their DID method is interoperable with the solid project DID? It would be nice if I am not locking my users in to authenticating with one specific identity provider.

[mikeal]

Would this implementation require my users to have a fission account?

Nope, UCAN is a standard that Fission wrote but is independent. It’s just a token signing scheme, using JWT, that allows for delegating permissions from one signer to another.

If so, out of curiosity would you happen to know if their DID method is interoperable with the solid project DID? It would be nice if I am not locking my users in to authenticating with one specific identity provider.

This is a question for @Gozala, but I’m pretty sure that any DID method we want to support we’ll have to add some code to our token validator.

My requirement for this initial version is that we can authenticate a token without doing more than a single database lookup for the root key. If we can validate delegates using other DID methods with nothing more than validating the signature then I have no objection to adding them (unless they blow up our bundle size with additional crypto lib requirements).

[Gozala]

Would this implementation require my users to have a fission account?

It would not, UCANs use did:key for identifying actors (users and services) in the system. And did:key is basically a public key prefixed with did:key

[Gozala]

If so, out of curiosity would you happen to know if their DID method is interoperable with the solid project DID? It would be nice if I am not locking my users in to authenticating with one specific identity provider.

I do not know what solid is using, however given that UCANs just use did:key it should be fairly simple to integrate with anything. Furthermore UCANs don’t require services to use did:key they could be arbitrary did’s, as long as verifier can somehow verify that issuer has signed the token. Users are did:key‘s to allow delegation chain verifications (signature verifications) without having to do side trips to figure out corresponding keys.

[dchoi27]

@mikeal does the current UCAN plan + w3name cover the use cases discussed in this issue (end user delegated uploading, tagging of files)?

[gobengo]

Related work from w3c-ccg that is similar to UCAN but with json-ld extensibility and mapping to other serializations, e.g. CBOR (but also VCs have a JWT serialization not unlike UCAN). https://w3c-ccg.github.io/zcap-ld/#delegation

[dchoi27]

Closing this in favor of #261, lmk if anyone thinks we should keep