Consumer Exits Permanently on Errors without Auto-Recovery

[miekassu]

When the SQS consumer encounters an error, the consumer loop exits permanently and never restarts. The service appears healthy (container running, HTTP responding) but stops processing all messages from the queue. This causes silent failures where new messages accumulate but are never processed until someone manually restarts the service.

We experienced this when a message with malformed data caused an unmarshal error (json: cannot unmarshal string into Go struct field PublishedEvent.data of type map[string]interface{}). The handler attempted to Nack the message, which failed due to missing sqs:ChangeMessageVisibility IAM permission. This caused the consumer to exit silently and the service appeared healthy, but stopped processing messages until manual restart.

To Reproduce

1. Deploy Outpost with AWS SQS but missing sqs:ChangeMessageVisibility permission
2. Message arrives that causes any handler error (malformed data, processing failure, etc.)
3. Handler attempts to Nack the failed message
4. GoCloud driver tries to call ChangeMessageVisibilityBatch
5. AWS returns 403 permission denied
6. Error bubbles up from consumer.Run() to startPublishMQConsumer (api/api.go:253)
7. Consumer goroutine logs error and exits permanently
8. Service continues running, appears healthy
9. All subsequent messages accumulate unprocessed with no alerts or visible indication

[alexluong]

Hi there, thanks for sharing the issue. We'll prioritize this accordingly and hope to include this in an upcoming release.

[moltar]

@alexluong Do you have an approximate ETA for the next release?

Are Docker images published on every merge to main?

Could we contribute a fix and get a fast release to test it?

[alexbouchardd]

@moltar No, we cut the new release from main depending on the changes. We'll cut a new minor release as soon as the fix is in main.

Could we contribute a fix and get a fast release to test it?

Absolutely 🙏

[miekassu]

Issues we are encountering likely relates to this too #306

[alexluong]

@moltar, @miekassu we're going to aim for a release this week the latest. Please feel free to open a PR and we'd be happy to support!

[alexluong]

@miekassu I want to confirm our approach with you, what we would do is to make sure the /healthz endpoint is up-to-date with the status of each workload within the service. So in your example, if the SQS consumer crashes, /healthz will return 503 which will signal the orchestrator to provision a new node.

Would this approach be sufficient for your use case? On that note, it seems yall are running Outpost on an AWS stack. I'd love to learn more about your use case and your stack if you're open to sharing more.

[moltar]

That or follow the k8s approach to have two endpoints:

• /livez
• /readyz

Where /readyz specifically checks for readiness (connectivity to resources).

[alexluong]

I thought about that but it's not super clear to me what the differentiation for us would be. Mainly because we only start the HTTP server after we're ready, so in a sense readyz is almost always going to be 200.

Do you make use of having 2 separate endpoint for readyness and liveness check? Would love to understand this better to account for it and potentially support the readyness endpoint if that makes sense.

[moltar]

• /livez answers "Is the process alive?" (server up)

• /readyz answers "Is the process ready to receive traffic?"

• /livez should return success as long as the application should not be restarted. If /livez fails, Kubernetes/ECS will restart the container.

• /readyz should return success only when the application can serve requests. If /readyz fails, Kubernetes/ECS will stop routing traffic to the pod, but it will not restart it.

We currently do not differentiate, but we could, if the option were available.

What we do mostly care truly is if the process is all healthy or otherwise restart it (/livez)

[alexluong]

Hi @moltar, sorry for a small delay, I opened a PR here #558, please feel free to check it out if this works for you.

The change to /healthz endpoint

Healthy response (200):

{
  "status": "healthy",
  "timestamp": "2025-01-15T10:30:00Z",
  "workers": {
    "http-server": {"status": "healthy"},
    "retrymq-consumer": {"status": "healthy"},
    "deliverymq-consumer": {"status": "healthy"}
  }
}

Unhealthy response (503):

{
  "status": "failed",
  "timestamp": "2025-01-15T10:30:00Z",
  "workers": {
    "http-server": {"status": "healthy"},
    "retrymq-consumer": {"status": "healthy"},
    "deliverymq-consumer": {"status": "failed"}
  }
}

This works for delivery & log service as well.