w32023-to-w3up

[gobengo]

Motivation:

• i can migrate CAR links from old.web3.storage to w3up via store/add invocations #2

Dependencies

• feat: add concurrency to migration #4
• add migrate-to-w3up bin script with spacey wizard
• add test to ensure car part bytes are copied when store/add says to
• document migrate-to-w3up cli usage in README
• test with known complex old.web3.storage accounts
  • I verified I was unable to find any status=upload when running with new --expect-store-add-status=done on big accounts. But I did run into other errors that I want to improve in followup PRs as described below.
• tolerate errors when migrating parts, and allow migrating the rest and logging failures #6

[gobengo]

Now this invokes store/add for each part CID and sends to w3up, logging receipts if successful.
Next I need to upload the CAR bytes when the receipt.out.ok.status === 'upload'

[gobengo]

⚡ npm -s start -w packages/w3 -- list --json | migrate-w32023-to-w3up --space "$space"

{
  type: 'Add',
  object: 'bagbaierakuersmo7wndedhwk43e5xwcpzwenuda3dhpcsvkfibewg5gxl7oa',
  upload: W32023Upload {
    _id: '315318962269342672',
    type: 'Car',
    name: 'Upload at 2024-01-22T19:10:44.048Z',
    created: '2024-01-22T19:10:44.048+00:00',
    updated: '2024-01-22T19:10:44.048+00:00',
    cid: 'bafybeieevwnu57cbcp5u6jsy6wxpj2waq5gfq5gc4spss4skpzk34vvxyy',
    dagSize: 8848144,
    pins: [
      {
        status: 'Pinned',
        updated: '2024-01-22T19:10:44.048+00:00',
        peerId: 'bafzbeibhqavlasjc7dvbiopygwncnrtvjd2xmryk5laib7zyjor6kf3avm',
        peerName: 'elastic-ipfs',
        region: null
      },
      [length]: 1
    ],
    parts: [
      'bagbaierakuersmo7wndedhwk43e5xwcpzwenuda3dhpcsvkfibewg5gxl7oa',
      [length]: 1
    ],
    deals: [ [length]: 0 ]
  },
  receipt: {
    cid: 'bafyreieu5or326j53kx6hj4b7auwkkjnsvnh65xmcggizntm3whldonf7u',
    out: {
      ok: {
        url: 'https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaierakuersmo7wndedhwk43e5xwcpzwenuda3dhpcsvkfibewg5gxl7oa/bagbaierakuersmo7wndedhwk43e5xwcpzwenuda3dhpcsvkfibewg5gxl7oa.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMMTKTJNYUK%2F20240124%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240124T010036Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjELH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJIMEYCIQDz9cNO3kFylWZFDizltYCY%2F5sls%2BDsjHSh9ALw7B9c%2FwIhAL4SfeRvlK1FskxwBNFqX3cGKiSI5qMgMuIXwJopKhTLKtMDCGoQABoMNTA1NTk1Mzc0MzYxIgxSuSj7Lgz1dmmkUsoqsAPQ3ZILHFB%2Br8CKkql%2FfD%2Fg6V7FTnKteZx8UvAMnzVWatV8t0Scurs0qhIj0RsDKuM52V4XoSDFFdM2c1EBZTmbqIpoEobvnGCMZa%2B5Lrv8a2n156Mt%2B8r4siA3zmwuDe1FSRTn0CSbnzlF8NIdDZctCpPXxdjBg7t0YauBU436KOV7M4Ol6rxSZ17qircN2RtLScquSvxVccjUQmKDbFwSkITOHry8e%2Bek8KFn6g30Z4cFdPEXO%2BHn1vqNQir3iTaFLqbIxBPKm2guvbtagLAs7jKfOVx8ZmGPmZjzb4swPRl1SfURhDM8RVLdxXIcofhh%2FTph0PVZFoXCNq3LxhYxmFWw1aGoCOjYXIhDxgscOKa1fr7yB8alPIdlVQS6XHlB3F6tPG8M8ED5sLFc2GSgJ2tDwkFdLBr3%2Fe1HQRK6%2BAXUivBmSoXQ7JVpeVoszu8iaDj3vtjgtrc2Fc7D1sYSTQIW4ALFpI4v37j3ZeYDDMEOlZQkEZeEYcgfJwuiSgs%2FGUksbxT%2BA8IzxoZFlsT3kL9nhLbmauR%2BfZ3mMzPvzHwlIoglssUVHioxaxPEEFowkMDBrQY6nQEDj1ypsOZSYuR9YyX342nxno0CCXsL3wakMwtql%2B3U7XDK2aPka%2FYoyB457vyvq7qAZEDweTKdhH8j5warLfbcpM7G2bNm9oFrazlL2%2FesBkUe5HxJUDAiJ8PcZ6AinHxYfohoH5f3lEbK6Z0%2B3l1wed%2FKLVG7hWHjmoSdJ6sXR39ti7x%2FsNUQhQQZXqVhQE5%2FvflB0LDTEOSPBT9P&X-Amz-Signature=50ac93317f142d0e55d2acf5aff09725d2a17a9a262bb7d4f7bd24ab12170706&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject',
        link: CID(bagbaierakuersmo7wndedhwk43e5xwcpzwenuda3dhpcsvkfibewg5gxl7oa),
        with: 'did:key:z6MkmvzdDBYoNcCg9NKkUEq6w1mDP4JnciG28p9EUhxyf8A5',
        status: 'upload',
        headers: {
          'content-length': '8848144',
          'x-amz-checksum-sha256': 'VQkZMd+zRkGeyubJ29hPzYjaDBsZ3ilVRUBJY3TXX9w='
        },
        allocated: 0
      }
    }
  }
}
{
  type: 'Add',
  object: 'bagbaierayx2sfg6oq5pgazwlzxvi6pcexp7eq5hleut5khpwr4nl4y3rfuhq',
  upload: W32023Upload {
    _id: '315318962269342671',
    type: 'Car',
    name: 'Upload at 2024-01-22T19:10:23.808Z',
    created: '2024-01-22T19:10:23.808+00:00',
    updated: '2024-01-22T19:10:23.808+00:00',
    cid: 'bafybeicmidbuoilkli5w3bttmcndhpikk7zt6ydv3m6rihxl266c546qpi',
    dagSize: 8848085,
    pins: [
      {
        status: 'Pinned',
        updated: '2024-01-22T19:10:23.808+00:00',
        peerId: 'bafzbeibhqavlasjc7dvbiopygwncnrtvjd2xmryk5laib7zyjor6kf3avm',
        peerName: 'elastic-ipfs',
        region: null
      },
      [length]: 1
    ],
    parts: [
      'bagbaierayx2sfg6oq5pgazwlzxvi6pcexp7eq5hleut5khpwr4nl4y3rfuhq',
      [length]: 1
    ],
    deals: [ [length]: 0 ]
  },
  receipt: {
    cid: 'bafyreichyyergghpyfzarxufnpyp43tsyoy2zj25sxpnhej4lc3apywl2m',
    out: {
      ok: {
        url: 'https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaierayx2sfg6oq5pgazwlzxvi6pcexp7eq5hleut5khpwr4nl4y3rfuhq/bagbaierayx2sfg6oq5pgazwlzxvi6pcexp7eq5hleut5khpwr4nl4y3rfuhq.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMMQSPAZJ2R%2F20240124%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240124T010036Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjELH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQD6t8F2JBaJdm9NFiKEvrwnql%2BdndIz58A9rEC3ayYA3wIgbLmpJR8nQO8n0634bu%2FxfOGpuXEC19hJrct7gZtUYtAq0wMIahAAGgw1MDU1OTUzNzQzNjEiDD1bqTo9VZeYaBG0wSqwA3tAx0oT56bNGge0m%2BF5JyRAH8JVuOcL3NlRuAf1UvHeAdBDb7aTrrYUmkNUHtjRXoYuw%2FmJZh2JxLyg9CtGDyKuSxXL5bVKVHstOBm4zjP7ZkMXLsR7E%2BotOUucpG5zufxZKhOk9woH6Ammn8W1f7HzfFtMcrUcSKa83HpmguBzCO1qomD1qkWr3kZ%2FH%2BAUOMsatrAjjXnSI3qMtxSATg53KWDuSmRoN8GUmlWUFcPpWjvtXO8UgVOc1qeNjx5TTUuw%2FqXc%2F%2B%2FNJ1GmsemRlkGZU1O1%2Bk8Da%2FXDuKMkt49DYQerJ7c%2B48SltpP7DG%2B9DsqoWQ%2BYt%2Fzq3W2UhrxfNqUUvHf71i8IDpCQEUOuY4ve%2F5Xp0prErtO48EneAP%2BOv6Cj71Ib67nqZFIttsqYXHC%2FdSpB9JTlAvJfCGDsTYkBEJv3fBu9ZjTxjaiHVNgggcNC1%2BolTGvKMvs86R6rtHoxPJxMWJUY2Df%2FOZ33%2FPGzfOlTHtH5MGAPdssZDbDY2o1Y0QyFbUnR9D%2BSa6CFePgNy1g59yavNpzQdv8PA%2FuX60DGd4ReMZgCo2tNgxPL8zDEwcGtBjqeAVAqlu0VFIP%2Fl8Ve29VxqdIZCyVuds1bcM9dbqcBFkvtG%2BFHTOwjeXJO4ccqz8veGU04zsBsryHKAuEKBuoSBzbMu74H43Z1PskC0Zd5l8l7RKydQXEtVHpVssNoXPb6O5%2BQmRmmusdI1vfbJ0z6mThoh4alfyi464HYR9qBHgPlaE2znOVbLdIK9ZOMfmU7jgpLVCZ%2BpcjsC%2Bd1awfi&X-Amz-Signature=8746a24686431c32ec9c91f015298cfeb1e3a139b4394437925e3e8139df67f3&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject',
        link: CID(bagbaierayx2sfg6oq5pgazwlzxvi6pcexp7eq5hleut5khpwr4nl4y3rfuhq),
        with: 'did:key:z6MkmvzdDBYoNcCg9NKkUEq6w1mDP4JnciG28p9EUhxyf8A5',
        status: 'upload',
        headers: {
          'content-length': '8848085',
          'x-amz-checksum-sha256': 'xfUim86HXmBmy83qjzxEu/5IdOslJ9Ud9o8avmNxLQ8='
        },
        allocated: 0
      }
    }
  }
}
```

[gobengo]

I merged #6 into here.
Now you can migrate with a log file, tolerating errors, and derive a second migration run to fix the errors from the log file. https://github.com/web3-storage/migrate-to-w3up/tree/w32023-to-w3up?tab=readme-ov-file#migrate-w-log

[alanshaw]

This looks great! ❤️

[alanshaw]

This is a little buried - I'd perhaps have a separate heading to highlight this.

[gobengo]

ok I'll add a separate heading for the 'retry' stuff, which is I think what you mean. lmk if you meant some other way of grouping it.

[alanshaw]

What is errParseCid and errParseBase32 doing? They are unused?

[gobengo]

I changed this to include them in the error.cause that gets thrown if a CID cannot be created from the input

[gobengo]

0a3418a