You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There is a bug in the io.kestra.plugin.aws.sqs.Consume task. In SQS, the consumed message must be deleted from the queue after being consumed. Atm, this is not happening, leading to a potential of a never-ending loop, or issues that sometimes a new message doesn't trigger an execution when used with SQS trigger task.
When two messages are sent, i.e., 2 tasks - each sending an SQS message, I would expect that each of these messages would trigger an execution of the flow that has an SQS trigger:
id: sqsPublishMessagenamespace: prodtasks:
- id: generateMessagetype: io.kestra.core.tasks.debugs.Returnformat: "Hi from a Kestra task {{task.id}}"
- id: publishMessagetype: io.kestra.plugin.aws.sqs.PublishaccessKeyId: "{{envs.aws_access_key_id}}"secretKeyId: "{{envs.aws_secret_access_key}}"region: eu-central-1queueUrl: https://sqs.eu-central-1.amazonaws.com/338306982838/kestrafrom:
data: | {{outputs.generateMessage.value}}
- id: publishMessage2type: io.kestra.plugin.aws.sqs.PublishaccessKeyId: "{{envs.aws_access_key_id}}"secretKeyId: "{{envs.aws_secret_access_key}}"region: eu-central-1queueUrl: https://sqs.eu-central-1.amazonaws.com/338306982838/kestrafrom:
data: This is a new message to check again
and reacting to those messages:
id: sqsReactToMessagenamespace: prodtasks:
- id: consumeMessagetype: io.kestra.plugin.aws.sqs.ConsumeaccessKeyId: "{{envs.aws_access_key_id}}"secretKeyId: "{{envs.aws_secret_access_key}}"region: eu-central-1queueUrl: https://sqs.eu-central-1.amazonaws.com/338306982838/kestramaxRecords: 1
- id: printtype: io.kestra.core.tasks.scripts.Bashcommands:
- "echo received message stored in {{outputs.consumeMessage.uri}}"triggers:
- id: sqstype: io.kestra.plugin.aws.sqs.Triggerdescription: "to activate it, set disabled to false"disabled: falsequeueUrl: https://sqs.eu-central-1.amazonaws.com/338306982838/kestraaccessKeyId: "{{envs.aws_access_key_id}}"secretKeyId: "{{envs.aws_secret_access_key}}"region: eu-central-1maxRecords: 1
all messages remain in the Queue - they should be deleted as soon as they have been consumed
Actual Behaviour
Apart from that, it would be great if the output would be stored as a variable rather than as a file in internal storage. The message is a small piece of text/metadata so it makes sense to make it available as a variable without the ion file's overhead in internal storage. The UX would benefit from that by making it possible to view the message output and pass it to downstream tasks.
So instead of: {{outputs.consumeMessage.uri}}
Having: {{outputs.consumeMessage.value}} would be beneficial
Steps To Reproduce
No response
Environment Information
Kestra Version: develop-full docker image
Example flow
No response
The text was updated successfully, but these errors were encountered:
Apart from that, it would be great if the output would be stored as a variable rather than as a file in internal storage. The message is a small piece of text/metadata so it makes sense to make it available as a variable without the ion file's overhead in internal storage. The UX would benefit from that by making it possible to view the message output and pass it to downstream tasks.
We usually not provide this functionality for message broker, consume and trigger task will consume a list of messages and messages in a broker can be of any size so we strongly encourage user to store them inside our internal storage.
You can open an enhancement request for that, we may revisit this but it would need to add task properties to choose between STORE or FETCH (the same concept we have in a lot of database query-based tasks).
Expected Behavior
There is a bug in the
io.kestra.plugin.aws.sqs.Consume
task. In SQS, the consumed message must be deleted from the queue after being consumed. Atm, this is not happening, leading to a potential of a never-ending loop, or issues that sometimes a new message doesn't trigger an execution when used with SQS trigger task.you can reproduce it using:
https://github.com/anna-geller/kestra-terraform-examples/blob/main/aws_sqs_tf/sqs.tf
When two messages are sent, i.e., 2 tasks - each sending an SQS message, I would expect that each of these messages would trigger an execution of the flow that has an SQS trigger:
and reacting to those messages:
all messages remain in the Queue - they should be deleted as soon as they have been consumed
Actual Behaviour
Apart from that, it would be great if the output would be stored as a variable rather than as a file in internal storage. The message is a small piece of text/metadata so it makes sense to make it available as a variable without the ion file's overhead in internal storage. The UX would benefit from that by making it possible to view the message output and pass it to downstream tasks.
So instead of:
{{outputs.consumeMessage.uri}}
Having:
{{outputs.consumeMessage.value}}
would be beneficialSteps To Reproduce
No response
Environment Information
Example flow
No response
The text was updated successfully, but these errors were encountered: