refactor: move s3 url signing to car-store

[olizilla]

• Add createUploadUrl to car-store.js as creating presigned s3 urls is a job for the bucket abstraction and having a thing called signer that wasn't a keypair for a DID for ucanto was confusing.
• Switch to the @aws-sdk/s3-request-presigner for creation of signed s3 urls to simplify credential management. The existing s3 client is reused.
• Adds content-length to the list of headers that form part of the url signature.

fixes #38

Notes

When presigning a URL for s3, some headers are "hoisted" to the URLSearchParams; they appear in the query string so you can hand the url off as a string, and the params needed for auth are baked right in there. S3 support unpacking those params from either the query or from http request headers. The query params are part of the signature, so they can't be tampered with.

You can also "hoist" and sign any other params you want... it is tempting to move x-amz-checksum-sha256 to the query so a user wouldn't need to provide it separately. However it appears that aws does not support pulling that value out of the query, only from a request header.

Meanwhile, it also appears that minio does not validate the PUT payload matches the x-amz-checksum-sha256 header at all. I include a test here that fails to show minio accepting a presigned PUT, but sending data that doesn't match the checksum.

It is unknown how s3 reacts to sending a content-length that doesn't match the payload size, but there are tests for it in storacha/sigv4#7

TODO:

• add in content-length assertion for presigned url
• check if we can remove the amz header from store/add response. The checksum must be enforced via the URL, not from an additional header that the client may(not) send. I think the signing of headers bakes in an assertion that those headers will be provided when using the presigned url.

License: MIT
Signed-off-by: Oli Evans oli@protocol.ai

[seed-deploy]

View stack outputs

• pr66-upload-api-ApiStack

  Name	Value
  ApiEndpoint	https://4wqg7s7lo8.execute-api.us-east-2.amazonaws.com
  CustomDomain	https://pr66.up.web3.storage

• pr66-upload-api-BusStack

• pr66-upload-api-CarparkStack

• pr66-upload-api-SatnavStack

[olizilla]

these are the cool headers i get on the signed url

URLSearchParams {
  'X-Amz-Algorithm' => 'AWS4-HMAC-SHA256',
  'X-Amz-Content-Sha256' => 'UNSIGNED-PAYLOAD',
  'X-Amz-Credential' => 'minioadmin/20221205/us-west-2/s3/aws4_request',
  'X-Amz-Date' => '20221205T231949Z',
  'X-Amz-Expires' => '86400',
  'X-Amz-Signature' => '4bf836f67afea8349a5441c5d798c9b1fb07e233e5da24c309ac6e2df2a1d5d8',
  'X-Amz-SignedHeaders' => 'content-length;host',
  'x-amz-checksum-sha256' => 'go8RUmHWc07TANIf//M+HV8oQaKr93qElP0pc4qkHTA=',
  'x-id' => 'PutObject' }

[olizilla]

docs

• https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
• https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html

TODO: figure out what s3 does with content length. It appears to be using it as one of the signed headers, so an upload should fail if the client sends a different content length header than the one used to create the url, as the signature wouldn't match.

[vasco-santos]

@olizilla tsc is complaining 😅

[vasco-santos]

This looks good to me! Ci is failing because of lint, we need to confirm all the tests are running.

Using @aws-sdk/s3-request-presigner seems quite straightforward. Would love to know the reasons that we wrote sigv4 though. Is there something that we are missing?

[olizilla]

@hugomrdias wrote it for using in things like cloudflare workers where we don't want to pull in a bunch of aws SDK and don't already have an S3 client set up.

[olizilla]

@vasco-santos fixed merge conflicts, appeased tsc, and made tests actually work given they depend on global fetch... now, as I am about to drop a PR that upgrades us to use node 18 here, I've chosen to just get it working by enabling the node >=16.15 support for global fetch with the `--experimental-fetch' flag, just for the tests. I can remove that when we are all node 18

[olizilla]

Holy cow. Consider #72 closed! I just got that test passing in CI using the latest minio from docker hub. When i removed my local 4 month old image and let it pull the latest the test started passing locally too!

[vasco-santos]

Test description does not match this flow. Can we change it or decouple in 2 tests?

[olizilla]

strictly this is a "signature not match" failure as the content-length header that was used to sign the url has the value 5, and when using it we send 6, so it fails when validating the url sig.

[olizilla]

i've split the tests anyway