SQS created queues do not persist

[lonewarrior556]

I'm trying to initialize my env once for further use.
Heres my docker-compose.yml

  localstack-aws:
    image: localstack/localstack
    container_name: localstack-aws
    ports:
      - $S3_PORT:4572
      - $SQS_PORT:4576
    volumes:
      - localstackdata:/tmp/localstack
    environment:
      DATA_DIR: /tmp/localstack/data
      HOSTNAME_EXTERNAL: localstack-aws
      SERVICES: s3,sqs

volumes:
  localstackdata:

I have an init script that runs the following 3 commands

pip install awscli-local
awslocal s3api create-bucket --bucket bucket-name
awslocal sqs create-queue --queue-name localSQS.fifo

I restart the container; docker exec -it localstack-aws bash...

> bash-4.3# awslocal s3 ls
2006-02-03 16:45:09 bucket-name
> bash-4.3# awslocal sqs list-queues
> bash-4.3#

Is there anyway to get created sqs queues to persist?

[lonewarrior556]

I saw persistent data (currently only supported for these services: Kinesis, DynamoDB, Elasticsearch, S3) as a temporary work around create a init.sh that creates the resources you need and modify supervisord

[supervisord]
nodaemon=true
logfile=/tmp/supervisord.log


[program:infra]
command=make infra
autostart=true
autorestart=true
stdout_logfile=/dev/fd/1
stdout_logfile_maxbytes=0
stderr_logfile=/dev/fd/2
stderr_logfile_maxbytes=0

[program:dashboard]
command=make web
autostart=true
autorestart=true
stdout_logfile=/dev/fd/1
stdout_logfile_maxbytes=0
stderr_logfile=/dev/fd/2
stderr_logfile_maxbytes=0

[program:init]
command=bash -c 'sleep 5 && bash /scripts/init.sh'

then build from Dockerfile

from localstack/localstack

RUN  rm /etc/supervisord.conf
COPY supervisord.conf /etc/supervisord.conf

# expose default environment (required for aws-cli to work)
ENV MAVEN_CONFIG=/opt/code/localstack \
    USER=localstack

# expose service & web dashboard ports
EXPOSE 4567-4583 8080

# define command at startup
ENTRYPOINT ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf"]

docker-compose.yml

  localstack-aws:
    build:
      context: ./localstack
    container_name: localstack-aws
    ports:
      - $S3_PORT:4572
      - $SQS_PORT:4576
    volumes:
      - ./services/localstack:/scripts
      - localstackdata:/tmp/localstack
    environment:
      DATA_DIR: /tmp/localstack/data
      HOSTNAME_EXTERNAL: localstack-aws
      SERVICES: s3,sqs 

[mberlanda]

@lonewarrior556 would you like to have a solution like #819 ? Currently elasticmq supports the possibility to bootstrap some default resources when launching the SQS mock. This may be backported to localstack as well

[lonewarrior556]

To be honest, I modified supervisord.conf and have been launching everything with this startup script

#! /bin/bash

until cat /tmp/infra.log  | grep Ready

do
  sleep .1
done


sleep 1
awslocal sqs create-queue --queue-name localSQS --attributes DelaySeconds=1,VisibilityTimeout=600
awslocal sns create-topic  --name localSNS
awslocal s3api create-bucket --bucket local-bucket

# clear log file
cat /dev/null > /tmp/infra.log

this solution has worked very well for me since I can bootstrap every service at launch with cli.
I wouldn't want to use something else that just launches sqs and not the other services.

If something, I would love for a way to include a bash script somehow that gets executed automatically once the service is up.

[mberlanda]

hey @lonewarrior556, your approach is definitely very interesting. The SQS implementation is based on elasticmq and this project allows to bootstrap some resources on startup.
I was actually just wondering if it may be enough for you to add a config file describing the queues to create or you were seeking for another kind of persistency (e.g. I stop my locastack container with some unconsumed messages, we restarting the container they will still be there)

[shadowspawn]

I was just looking for solutions to this too. I am looking for bootstrap like a config file to create expected queues, and not actual message or queue persistence.

[whummer]

Thanks for reporting @shadowspawn @mberlanda @lonewarrior556 @mberlanda

I am looking for bootstrap like a config file to create expected queues

Please refer to the README on how to specify a bootstrap script that creates resources in a fresh instance: https://github.com/localstack/localstack#initializing-a-fresh-instance Thanks