Worker node fails to pull new message when processing 1 large item batch #116
Comments
olizilla
added a commit
that referenced
this issue
Mar 14, 2023
- switch to an sqs lib that polls for new messages concurrently rather
than in batches. **This is rad** as now we'll make better use of each
container!
- treat timeouts as a regular failure. Let the message go back on the
queue for another node to try. After 3 goes it'll go to the dead letter
queue and be marked as failed. This is fine, and simplifies the pickup
worker a lot, as it doesn't need to talk to dynamo or determine the
cause of an error.
- rewrite pickup worker so we can compose it out of
single-responsibility pieces instead of having to pass through the giant
config ball. _It's so much simpler now!_ You can figure our what it does
from it's parts! `sqsPoller` + `carFetcher` + `s3Uploader`
```js
const pickup = createPickup({
sqsPoller: createSqsPoller({
queueUrl: SQS_QUEUE_URL,
maxInFlight: BATCH_SIZE
}),
carFetcher: new CarFetcher({
ipfsApiUrl: IPFS_API_URL,
fetchTimeoutMs: TIMEOUT_FETCH
}),
s3Uploader: new S3Uploader({
bucket: VALIDATION_BUCKET
})
})
```
see: https://github.com/PruvoNet/squiss-ts/
fixes #13
fixes #116
fixes #101
License: MIT
---------
Signed-off-by: Oli Evans <oli@protocol.ai>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm seeing 3 worker nodes each slowly downloading 1 large dag each, while messages backup in the queue. I'd expect each node to be pulling additional messages from the queue up to
BATCH_SIZErather than waiting for the single item to complete.The text was updated successfully, but these errors were encountered: