A bug with SQS FIFO and DLQ

[moltar]

The dlq check is naive, and appends -dlq to the end of the queue name.

However, fifo queues require to have the .fifo suffix, and thereby the final queue name becomes foo-bar.fifo-dlq which is an invalid queue name.

outpost/internal/mqinfra/awssqs.go

Line 87 in 732545e

dlqName := infra.cfg.AWSSQS.Topic + "-dlq"

I think the correct solution is to allow the end user to configure the name of all queues via config and/or env, and thereby don't need to have this check at all.

[alexbouchardd]

@moltar, just to clarify, what are you hoping to accomplish with FIFO? Given that the delivery can't be guaranteed to be in order, I'm not sure what the point of FIFO is here other than limiting throughput.

[alexbouchardd]

I think the correct solution is to allow the end user to configure the name of all queues via config and/or env, and thereby don't need to have this check at all.

That's reasonable, we could also account for the fact that it's a FIFO queue when appending the -dlq

[obazoud]

I had the same problem with this suffix even for a normal SQS. It is incompatible with IaC tools.

[alexluong]

I think the fundamental issue is Outpost's philosophy was that the mqs should be provisioned and managed by Outpost, therefore issues with FIFO or IaC shouldn't be an issue.

The thought process was that it's simpler for users (operators) to provide credentials and let Outpost does its thing. The alternative is we have to give instruction or specify some required configuration (visibility timeout, max retry, dlq, etc.).

Seems like there are enough interest where operators want to provision their own infra and Outpost would only interact with the mq themselves. In that case, we won't have to think about DLQ or naming convention in the first place, which should work as well.

For this, I'm thinking an environment variable of some sort such as MQS_PROVISIONED, MQS_MANAGED or something like that to disable this behavior. Would that solve this issue for yall? @moltar, @obazoud

[obazoud]

Yes, that could help, ideally configurable per queue.
Sharing the configuration that Outpost used could be help too.

Also, does Outpost actually use the DLQ — for example, to redrive messages — or is it just a configuration check?
I see that it creates the redrive policy, but I can’t find where it’s used to move messages from the DLQ back to the main queue.

And it would be naive to assume Outpost can handle every possible SQS configuration case by itself. For example:

• How would Outpost manage SQS tags? Currently, I don’t think it does, which means losing the ability to track costs.
• How would Outpost create an SQS queue using my own KMS key ?
  ...

FYI on my Outpost instance, I actually removed those checks from the code and provisioned my own SQS queues.

[alexluong]

Hi @obazoud,

does Outpost actually use the DLQ

no, we do not use the DLQ at this moment

it would be naive to assume Outpost can handle every possible SQS configuration case by itself

I do agree with this. I shared our original philosophy when it comes to mqs infra, but it's still early and we definitely hope to work with users to make sure it works for most use cases. And likewise, happy to provide configuration option to allow power users to make Outpost work for them.

ideally configurable per queue

Can you elaborate more on your use case here? And if you have a suggestion on what the configuration would look like from your end, happy to take that into consideration.

[moltar]

@moltar, just to clarify, what are you hoping to accomplish with FIFO? Given that the delivery can't be guaranteed to be in order, I'm not sure what the point of FIFO is here other than limiting throughput.

The docs state that FIFO queues are supported:

See instructions.md

But once I ran into the issue and researched guarantees I agree FIFO queues don't make sense much.

Though one feature of FIFO SQS in particular I miss is message duplication.

[alexbouchardd]

Good feedback, seems like we'll want to allow for self-managed mq configuration

Though one feature of FIFO SQS in particular I miss is message duplication.

Outpost does handle deduplication itself 🫡