feat: put write to cluster behind a flag (#1785)

- Set `ENABLE_ADD_TO_CLUSTER` to send upload to ipfs-cluster after
sending response to user. The current behavior is to always try to write
user uploads to cluster. With this PR we disable that but leave a
feature flag to restore it if we need to.
- Adds LinkIndexer to decode and record links from each block as we
iterate through them. It allows us to set the DAG structure as
'Complete' where we know the upload contains all the blocks for a
complete DAG with no dangling links.
- Adds task to call to `LINKDEX_API` after the response is sent where
the DAG is `Partial`, to check if it is complete across other CARs under
the same upload.
- Where we have a complete CAR we set the status as "Pinned" early as
it's already in elastic-ipfs. This is debatable, as there is a lag until
it's discoverable, but I think it's reasonable as I want to push us
towards telling users when we safely have their stuff early.
- We write uploads to more than one place, by adding an R2Bucket binding
to our worker env called `carpark` where we write cars keyed by their
car CID.
- R2 only supports an [md5 integrity check on
put](https://developers.cloudflare.com/r2/runtime-apis/#r2putoptions),
so this PR re-uses the md5 lib that the aws-s3-sdk uses to generate that
and also stores it in the r2 metadata so we see what hash was used to
verify it later.
- CAR CIDs are generated as [w3up-cli does
it](https://github.com/web3-storage/w3up-cli/blob/b313c1221a0fb93a7d7cc3ced5fd1d4bce537a7a/src/utils.js#L46-L50)
with the [`0x202`
code](https://github.com/multiformats/multicodec/blob/955e61fe6221454c682ca9ba4901558bfc1bac96/table.csv#L130),
and added to the /car response body.
- miniflare is now available as a global in tests so we can make
assertions about [r2 bucket contents in
tests](https://miniflare.dev/storage/r2#manipulating-outside-workers),
which is nice.
- we make use of [miniflare's new fetch mocking
support](https://miniflare.dev/core/standards#mocking-outbound-fetch-requests)
to test calls to the linkdex-api happen when we expect them to.

_NOTE: We've been using ipfs-cluster as a DAG completeness check-ing
service, and we're using DAG completeness as a proxy for "we got all the
CARs and the DAG is stored". We need to do some reframing as it's a
feature of DAGs and CARs and IPFS that you could upload some `dab-json`
that links to the complete wikipedia archive. We want to inform you when
your blocks are safely stored, but we don't want to necessarily also
pull in all of wikipedia, nor do we want to avoid telling you your
blocks are pinned where they link to some other dataset._

Fixes: #1784

License: MIT
Signed-off-by: Oli Evans <oli@protocol.ai>

.env.tpl

@@ -3,7 +3,7 @@
 # Copy me to `.env` and fill out the missing values.
 # This file is only for DEV. In prod, set the env vars a process needs as secrets.
 ENV=dev
-
+# DEBUG=true
 
 ## ---- common ---------------------------------------------------------------
 
@@ -33,6 +33,7 @@ SALT="Q6d8sTZa+wpIPrppPq6VdIKEbknjrsSCQklh/hVU4U0="
 # base64 test:test - the creds for the local cluster test container
 CLUSTER_BASIC_AUTH_TOKEN="dGVzdDp0ZXN0"
 CLUSTER_API_URL=http://127.0.0.1:9094
+# ENABLE_ADD_TO_CLUSTER=true
 
 # IPFS Gateway URL pointing to dockerised ipfs instance
 GATEWAY_URL=http://localhost:8080
@@ -47,6 +48,12 @@ S3_SECRET_ACCESS_KEY_ID = 'minioadmin'
 # stripe.com secret key - check 'stripe.com' in your team secret manager
 # STRIPE_SECRET_KEY='sk_'
 
+# URL prefix for CARs stored in R2
+# CARPARK_URL = "https://carpark.web3.storage"
+
+# URL prefix for the linkdex-api
+# LINKDEX_URL = "https://linkdex.web3.storage"
+
 ## ---- website ---------------------------------------------------------------
 
 # vars to expose to the public website build have to be prefixed with NEXT_PUBLIC_