feat: add nft.storage upload option

[yusefnapora]

Hey there, Yusef from Protocol Labs here 😄. I've been working on getting Metaplex and NFT.Storage to play nicely together for a bit, and it seems close enough to get some feedback on. Some things are not yet wired up in our backend, so I'm just opening as a draft for now.

I'm planning to also add this functionality to the new JS SDK, but it seemed easier to test things out here.

The main work of the integration is happening in the metaplex-dotstorage-auth package that I've been writing up. It's not yet published to NPM, so there's currently a github dependency in the package.json for the candy machine cli package. That will be removed before this is converted from a draft PR.

The auth works entirely locally - the user just creates a self-signed JWT using the EdDSA algorithm with their private Solana Ed25519 key as the signing key. There's no Solana transaction required, just the signature and a small payload embedded in the JWT that looks like this:

{
  "iss": "did:key:<users-public-key>",
  "req": {
    "put": {
      "rootCID": "bafy123...",
      "tags": {
        "chain": "solana",
        "solana-cluster": "devnet"
      }
    }
  }
}

The JWT payload includes the Metaplex user's public key (encoded as a DID) in the iss field. The req field contains a description of the request, including the root CID of the content being uploaded and some tags to help keep track of things on our end.

The CID is calculated on the client by packing files into CAR archives, which are sent to the nft.storage backend in chunks of ~10MB at a time.

Status

There are some remaining pieces needed before this will be functional:

• [-] nft.storage backend updated to allow uploads with X-Web3-Auth: Metaplex <token> header support
  • currently deployed to staging environment. See below for usage
• metaplex-dotstorage-auth library published to NPM

I'll convert the PR from draft once it's possible to actually test this end to end. In the meantime, if any maintainers here have input on the code or any concerns, let me know and I'll try to address them ASAP.

Update (2021-11-23):

We've now published @nftstorage/metaplex-auth to npm, so I've updated this PR to depend on it.

The API changes are currently deployed to our staging environment, but not yet in production. To test, you can set the NFT_STORAGE_ENDPOINT environment variable to https://api-staging.nft.storage before running the candy-machine upload command.

For example:

NFT_STORAGE_ENDPOINT=https://api-staging.nft.storage npx ts-node ./src/candy-machine-cli.ts upload --storage nft.storage -k ~/.config/solana/devnet.json path/to/nft

[vercel]

@yusefnapora is attempting to deploy a commit to the Metaplex Team on Vercel.

A member of the Team first needs to authorize it.

[vercel]

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployments, click below or on the icon next to each commit.

squidstuff – ./js

🔍 Inspect: https://vercel.com/metaplex/squidstuff/4eAb6oJZFFKtAsuX1PRRXMY2tscj
✅ Preview: Failed

ono-store – ./js

🔍 Inspect: https://vercel.com/metaplex/ono-store/4B47MLhqRw7DbLxU7wodRJ8vYU1L
✅ Preview: Failed

rnftz – ./js/packages/fair-launch

🔍 Inspect: https://vercel.com/metaplex/rnftz/32hnL7cKoi55MPQ8qGWG1pyjbagu
✅ Preview: https://rnftz-git-fork-yusefnapora-feat-nftstorage-metaplex.vercel.app

nephrops – ./js

🔍 Inspect: https://vercel.com/metaplex/nephrops/6uFyCRcSD45cDy5ssZDSCSPCTPyi
✅ Preview: Failed

pinkskull – ./js

🔍 Inspect: https://vercel.com/metaplex/pinkskull/C5npuDgegnQA6ZALBrUAB9onMDq9
✅ Preview: Failed

litjesus – ./js/packages/fair-launch

🔍 Inspect: https://vercel.com/metaplex/litjesus/DbTyKqQnkFtWXrTuvwCXo1sFB6ct
✅ Preview: https://litjesus-git-fork-yusefnapora-feat-nftstorage-metaplex.vercel.app

pplpleasr – ./js/packages/fair-launch

🔍 Inspect: https://vercel.com/metaplex/pplpleasr/FDzSmPY1Fg33pgKKEyw9c6p6hBPX
✅ Preview: https://pplpleasr-git-fork-yusefnapora-feat-nftstorage-metaplex.vercel.app

fair-launch – ./js/packages/fair-launch

🔍 Inspect: https://vercel.com/metaplex/fair-launch/Ce9mE89ub6FtGtqwcwLhUDfg4jb1
✅ Preview: https://fair-launch-git-fork-yusefnapora-feat-nftstorage-metaplex.vercel.app

helium – ./js

🔍 Inspect: https://vercel.com/metaplex/helium/FBACNaCp7wNm5opeuzSqPbhzZwCH
✅ Preview: Failed

candy-machine-ui – ./js/packages/candy-machine-ui

🔍 Inspect: https://vercel.com/metaplex/candy-machine-ui/9nH3AvwbgSnQuPqGAXgNhN7GR4oP
✅ Preview: https://candy-machine-ui-git-fork-yusefnapora-feat-nftstorage-metaplex.vercel.app

[domjag]

I would love it if I could merge with nft.storage! Will keep a eye on this, if I have time I will contribute.

[yusefnapora]

I'm going to mark this PR as ready for review, with the caveat that the backend is only working on staging at the moment. It should be merged to production within a day or two, but in the meantime, see the instructions in the PR description for running against staging

[yusefnapora]

To follow up on the "path to production" here, before we deploy for to prod we'll need someone from the metaplex team to make an nft.storage account and give us the API token they'd like to use for the integration.

That account will be associated with all the uploads made using the /metaplex/upload endpoint so we can keep track of things in our backend.

[Moikapy]

Since users will be creating their own storefronts would it be possible to allow us to add our own api keys instead of relying on metaplex.

[dchoi27]

@Moikapy think the latest plan is to have NFT.Storage have custody of the NFT.Storage account associated with this endpoint rather than Metaplex being able to see all uploads to this endpoint. Moving forward we'll also have solutions using things like delegated uploads via UCAN so that end users can upload data associated with a given marketplaces's account without that marketplace having to share their private keys in the browser. Let us know your thoughts for the short-term and long-term solutions!

[yusefnapora]

Just a note that I've updated this PR to use the new storeNFTFromFilesystem method in the metaplex-auth lib. All the metadata manipulation happens in the library now, so this PR is much simpler.

Here's some example metadata from my manual testing (gateway link):

{
  "name": "Test NFT.Storage / Metaplex NFT",
  "symbol": "PLTEST-1",
  "description": "nothing to see here.",
  "seller_fee_basis_points": 0,
  "image": "https://dweb.link/ipfs/bafybeihqxtn5otahwbxjvkbhloszxrbjmvhf663rtq7sp5i2xzyl35bwtu/0.png",
  "properties": {
    "files": [
      {
        "uri": "https://dweb.link/ipfs/bafybeihqxtn5otahwbxjvkbhloszxrbjmvhf663rtq7sp5i2xzyl35bwtu/0.png",
        "type": "image/png",
        "cdn": true
      },
      {
        "uri": "ipfs://bafybeihqxtn5otahwbxjvkbhloszxrbjmvhf663rtq7sp5i2xzyl35bwtu/0.png",
        "type": "image/png",
        "cdn": false
      }
    ],
    "category": "image",
    "creators": [
      {
        "address": "3eoKfwCQ4jMBD5CEpLM8aCTZVN7wGQ9KTUBHhksmaZNh",
        "share": 100
      }
    ]
  }
}

[github-actions]

This PR has received no activity for 30 days. We will close it in 2 days, please reopen if you would still like to get this change in.

[yusefnapora]

Hello to anyone watching this PR 😄

I'm back from holidays and so on and have updated this to work with Candy Machine V2, as well as V1. It ended up being much simpler to squash everything before rebasing, so I've force pushed over the original branch.

So far I've done a basic test with a CMv2 config that looks like this:

{
    "price": 1.0,
    "number": 1,
    "gatekeeper": null,
    "solTreasuryAccount": "3eoKfwCQ4jMBD5CEpLM8aCTZVN7wGQ9KTUBHhksmaZNh",
    "splTokenAccount": null,
    "splToken": null,
    "goLiveDate": "25 Dec 2022 00:00:00 GMT",
    "endSettings": null,
    "whitelistMintSettings": null,
    "hiddenSettings": null,
    "storage": "nft.storage",
    "ipfsInfuraProjectId": null,
    "ipfsInfuraSecret": null,
    "awsS3Bucket": null,
    "noRetainAuthority": false,
    "noMutable": false
}

And it uploaded successfully:

• metadata
• image

There seems to be a typescript error in one of the dependencies that CI isn't happy about. I'll chase that down and update here when it seems good to go.

[roederw]

@yusefnapora rebase please.

[yusefnapora]

Hey @roederw, thanks for looking at this - I've just rebased to the latest master.

[roederw]

Looks like your builds are failing.

[yusefnapora]

@roederw sorry bout that - I didn't notice locally before pushing it up. Seems like I must have transitively pulled in a version of the mime package with a breaking API change? I'll dig in and see what's causing it.

[yusefnapora]

@roederw I believe I fixed the issue with the mime package by adding @types/mime to the devDependencies. However, the build is failing with a different type error that's been described previously in #1560.

I don't get the build failure when building master, so I rebased again onto the latest master. I still get the error, but as part of the rebasing I deleted yarn.lock and regenerated it, so my guess is that there's a dependency pinned in the lockfile that's making master build successfully.

The FormListFieldData type from the error message is from Ant Design, so that seems like a place to start. I'll check it out and see if I can find out what's up :)

[yusefnapora]

@roederw I made a new PR real quick to fix the type errors, since they affect master: #1599

[roederw]

Fix errors.

[austbot]

There was acleaner pr and it was merged

[yusefnapora]

Hey @austbot, glad to see that nft.storage support is merged in 👍

I think that the approach taken in this PR may still be valuable, since it allows users to upload to NFT.Storage without needing an NFT.Storage API key first, and it also supports files larger than 100 MiB by chunking them on the client.

If those features sound useful, I'd be happy to rework this PR to make the NFT.Storage API key optional and use the signature based auth scheme if the user doesn't have an NFT.Storage key.

Either way, I'm also down to add docs for using nft.storage to https://github.com/metaplex-foundation/docs - I'll plan to make a PR there describing how to use the flag added in #1512.

Cheers :)