Write verified cars to carpark bucket #136
Comments
olizilla
added a commit
that referenced
this issue
Aug 15, 2023
wip on #136 License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>
1 task
olizilla
added a commit
that referenced
this issue
Aug 16, 2023
Update car key to `${carCid}/${carCid}.car`, so we can start writing
them to carpark.
Calculate the sha256 of the car bytes from the stream as we produce the
linkdex report to determine if the car is complete / valid
Log total bytes per CAR as `carSize`. Use this to calculate the
throughput in loki. Needed as we wont be able to use the size of the
bucket once we start writing to the shared carpark.
Fixes #136
TODO
- [ ] update the production and staging env to write to carpark bucket.
License: MIT
---------
Signed-off-by: Oli Evans <oli@protocol.ai>
Merged
olizilla
added a commit
that referenced
this issue
Aug 17, 2023
Write to the existing carpark bucket rather than creating a dedicated one for pickup. Updater the pin status on successful write to s3 from the pickup worker. Previously we'd listen for object_created events and trigger a lambda to update the pin status, but we're moving to a shared bucket, so handle it in the worker. Key changes - Update sst stack to adopt bucket by name rather than create it's own - Delete the pinUpdate queue, as we now update the pin directly from the pickup worker - Improve the error handling so we fail pins that are too large rather than retying them... as we are updating the pin status from the worker now this is feasible. Previously we'd only mark a pin as failed using the dead-letter queue handler. - Fix pickup tests to verify pin status update in the dynamo table. - Pass the abort signal for the car fetch to the s3 upload so we abort the upload correctly. this fixes timeout and error handling issues we were seeing in the tests where the fetch timeout and chunk timeout would trigger the abort signal, but an error was not thrown immediately, which was making tests hang. Part of #136 --------- Signed-off-by: Oli Evans <oli@protocol.ai> Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We want all cars to end up in the carpark bucket.
The key structure must be
${carCid}/${carCid}.carWe need a way to track how much data has been uploaded via pickup. Currently this is handled by checking the size of the s3 bucket.
The text was updated successfully, but these errors were encountered: