bug: step functions catch logic broken in 0.14.3 [pro]

[smbkr]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I have a service I am running via Step Functions which I test with Localstack. The state machine catches certain errors that may be thrown, retries up to n times, and if the service is still failing after retries, publishes a message to EventBridge.

In 0.14.3 and latest, the service will fail and retry up to the specified number of times, however no event is published to the event bus once all the retries are exhausted.

I do not see anything related to this in the logs, only the lambda execution logs.

Expected Behavior

In 0.14.2, it works as expected. After n retries are exhausted, a message is published to the event bus specified in my state machine definition.

How are you starting LocalStack?

version: '3.8'

networks:
    default:
        external:
            name: abc
services:
    localstack:
        container_name: localstack_main
        image: localstack/localstack:0.14.3
        ports:
            - '127.0.0.1:53:53'
            - '127.0.0.1:443:443'
            - '127.0.0.1:4510-4530:4510-4530'
            - '127.0.0.1:4566-4620:4566-4620'
        environment:
            - COMPOSE_HTTP_TIMEOUT=120
            - LOCALSTACK_API_KEY=${LOCALSTACK_API_KEY}
            - SERVICES=acm,cloudformation,cloudfront,cloudwatch,ec2,events,kms,route53,serverless,sqs,ssm,sts,states,stepfunctions
            - DEBUG=1
            - DATA_DIR=/tmp/localstack/data
            - DOCKER_HOST=unix:///var/run/docker.sock
            - HOST_TMP_FOLDER=${TMPDIR:-/tmp/}localstack
            - LAMBDA_REMOTE_DOCKER=false
            - LAMBDA_DOCKER_NETWORK=abc
            - LOCALSTACK_HOSTNAME=127.0.0.1
        volumes:
            - '${TMPDIR:-/tmp/}localstack:/tmp/localstack'
            - '/var/run/docker.sock:/var/run/docker.sock'

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

docker-compose up

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

• deploy a state machine which:
  • invokes a lambda function (for a repro it can just throw immediately on execution)
  • catches errors thrown by the lambda execution
  • publishes a message to an event bus with the caught error

Here's a modified snippet of my state machine definition. The whole thing is more complex and I've had to remove parts:

foo:
  id: foo
  name: foo
  role:
    Fn::GetAtt: [foobar, Arn]
  definition:
    Comment: State machine for foo bar
    StartAt: InvokeFunction
    States:
      InvokeFunction:
        Type: Task
        Resource: arn:aws:states:::lambda:invoke
        ResultPath: null
        Parameters:
          Payload.$: $
          FunctionName: my function arn
        Retry:
          - ErrorEquals:
              - FOO.BarError
              - Error
              - Lambda.Unknown
              - ThrottlingException
            IntervalSeconds: 2
            MaxAttempts: 3
            BackoffRate: 2
            Comment: Retry failed execution
        Catch:
          - ErrorEquals:
              - States.ALL
            Comment: Catch when max retries exceeded or other error occurs
            Next: SendToDLQ
            ResultPath: $.error # Append 'error' to detail
        Next: PublishSentEvent
      SendToDLQ:
        Type: Task
        Resource: arn:aws:states:::sqs:sendMessage
        Parameters:
          QueueUrl: my queue url
          MessageBody.$: $.detail
        ResultPath: null
        Next: PublishFailedEvent
      PublishFailedEvent:
        Type: Task
        Resource: arn:aws:states:::events:putEvents
        Parameters:
          Entries:
            - Detail.$: $.detail
              DetailType: State Machine Failure
              EventBusName: my-event-bus
              Source: 'foo.state-machine'
        ResultPath: null
        Next: Fail

Environment

- OS: Mac OS 11.6.1
- LocalStack: latest, 0.14.3

Anything else?

No response

[smbkr]

I have tried 0.14.3.1 and behaviour is the same as 0.14.3

[thrau]

Hey Stuart @smbkr, based on your sample and @dominikschubert's test case from #6144, we were able to pinpoint the issue in the new SQS Query API implementation introduced recently. The fix should be out in about an hour.