Timeout 4h not working #101
Comments
olizilla
added a commit
that referenced
this issue
Mar 13, 2023
fixes #101 License: MIT Signed-off-by: Oli Evans <oli@protocol.ai>
olizilla
added a commit
that referenced
this issue
Mar 14, 2023
- switch to an sqs lib that polls for new messages concurrently rather
than in batches. **This is rad** as now we'll make better use of each
container!
- treat timeouts as a regular failure. Let the message go back on the
queue for another node to try. After 3 goes it'll go to the dead letter
queue and be marked as failed. This is fine, and simplifies the pickup
worker a lot, as it doesn't need to talk to dynamo or determine the
cause of an error.
- rewrite pickup worker so we can compose it out of
single-responsibility pieces instead of having to pass through the giant
config ball. _It's so much simpler now!_ You can figure our what it does
from it's parts! `sqsPoller` + `carFetcher` + `s3Uploader`
```js
const pickup = createPickup({
sqsPoller: createSqsPoller({
queueUrl: SQS_QUEUE_URL,
maxInFlight: BATCH_SIZE
}),
carFetcher: new CarFetcher({
ipfsApiUrl: IPFS_API_URL,
fetchTimeoutMs: TIMEOUT_FETCH
}),
s3Uploader: new S3Uploader({
bucket: VALIDATION_BUCKET
})
})
```
see: https://github.com/PruvoNet/squiss-ts/
fixes #13
fixes #116
fixes #101
License: MIT
---------
Signed-off-by: Oli Evans <oli@protocol.ai>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The Consumer manager in pickup has a timeout set by the variable
handleMessageTimeoutto 4 hours.This timeout should stop the download of the
carbut instead is not working at all.Investigate if this timeout is required and define a policy for the timeout
The text was updated successfully, but these errors were encountered: