Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: [PRO] Creating Aurora RDS cluster using Pulumi doesn't use mysql:latest image #10928

Closed
1 task done
hugo-abhishek opened this issue May 30, 2024 · 11 comments
Closed
1 task done
Assignees
Labels
area: integration/pulumi Issues related to Pulumi aws:rds Amazon Relational Database Service status: backlog Triaged but not yet being worked on type: bug Bug report

Comments

@hugo-abhishek
Copy link

hugo-abhishek commented May 30, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I am using Pulumi script to create aurora mysql AWS::RDS::Cluster using localstack-pro:latest image. With the older images (localstack-pro:3.4.0) of localstack we noticed mysql:latest images were being used, however, with the latest one's it is using mysql:8.0.28-oracle image.

I am attaching a minimum reproducible code below using Pulumi:
`import pulumi_aws as aws

aws.rds.Cluster("default",
cluster_identifier="aurora-cluster-demo",
engine=aws.rds.EngineType.AURORA_MYSQL,
engine_version="8.0.mysql_aurora.3.04.1",
availability_zones=[
"us-west-2a",
"us-west-2b",
],
database_name="mydb",
master_username="foo",
master_password="bar",
backup_retention_period=5,
preferred_backup_window="07:00-09:00"
)`

Expected Behavior

The later versions of localstack should be using mysql:latest image.

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

docker run localstack/localstack

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

awslocal s3 mb s3://mybucket

Environment

- OS:
- LocalStack:
  LocalStack version:
  LocalStack Docker image sha: 
  LocalStack build date:
  LocalStack build git hash:

Anything else?

Screenshot 2024-05-30 at 8 32 06 AM This version of oracle has some issues during restarts of mysql as mentioned in this [issue](https://github.com/docker-library/mysql/issues/852) and hence we are unable to use it.
@hugo-abhishek hugo-abhishek added status: triage needed Requires evaluation by maintainers type: bug Bug report labels May 30, 2024
@localstack-bot
Copy link
Collaborator

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

@Anze1508 Anze1508 added aws:rds Amazon Relational Database Service area: integration/pulumi Issues related to Pulumi status: backlog Triaged but not yet being worked on and removed status: triage needed Requires evaluation by maintainers labels May 30, 2024
@cloutierMat
Copy link
Contributor

Hi @hugo-abhishek,

For parity with Aurora, we are mapping the version of the mysql engine based on the aurora version configured. As you can see on RDS documentation, 8.0.mysql_aurora.3.04.1 is compatible with 8.0.28.

Furthermore, since the returned engine is mysql:8.0.28-oracle, I assume you are using an ARM CPU. I remember an issue with running non-oracle images on ARM, so all ARM aurora-mysql are currently locked to 8.0.28 with a warning in your LocalStack logs.

I am curious as if there is a reason for wanting to map 8.0.mysql_aurora.3.04.1 to latest instead of the AWS compatible version?

@cloutierMat cloutierMat self-assigned this Jun 11, 2024
@hugo-abhishek
Copy link
Author

hugo-abhishek commented Jun 12, 2024

Hi @cloutierMat, as mentioned above this version of oracle has issue for chown on broken symlink, this causes subsequent restarts to fail.

This version of oracle has some issues during restarts of mysql as mentioned in this issue and hence we are unable to use it.

The issue was fixed in this PR and it doesn't seem like the image localstack is using has a fix for this. This is the sha for 8.0.28-oracle eb3f4a2767a37c46e2f5424f1f0e8e79ace6fad52400c3220a6b7f7092b52aff.

Upon inspecting the above image, I can confirm above version doesn't contain the fix, and hence the ask for using the latest image (I believe we are fine with any version having this fix)

@cloutierMat
Copy link
Contributor

Thank you @hugo-abhishek for the added context. We will be looking into a fix for this and hopefully be able to pin a version with the appropriate fix while keeping mysql version on par with RDS version compatibility.

In the meantime, I can offer this workaround. You can manually pull the desired image and tag it 8.0.28-oracle on your machine. That way when LocalStack attempts to pull it, Docker will use the one you tagged.

@hugo-abhishek
Copy link
Author

Thanks @cloutierMat for the alternate solution.

@cloutierMat
Copy link
Contributor

Hi @hugo-abhishek, This issue has now been fixed and will be available in the next latest image.

We are no longer using the -oracle images for arm, and the minimum version of aurora MySQL we support will now be 8.0.mysql_aurora.3.05.0 mapping to mysql:8.0.32. Any lower version can still be created but will default to 8.0.32.

I will mark this issue as closed but do not hesitate to contact us again if there are any further issues.

@hugo-abhishek
Copy link
Author

Thanks @cloutierMat

@hrishi-hugo
Copy link

@cloutierMat There is still an issue we are facing. Now it is using the correct image but we are still not able to create rds clusters using aurora mysql.
The Mysql containers in docker are in exited state with this error logs:

2024-07-31 11:21:40 2024-07-31 05:51:40+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.30-1.el8 started.
2024-07-31 11:21:40 2024-07-31 05:51:40+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-07-31 11:21:40 2024-07-31 05:51:40+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.30-1.el8 started.
2024-07-31 11:21:41 '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
2024-07-31 11:21:42 2024-07-31T05:51:41.996434Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-07-31 11:21:42 2024-07-31T05:51:42.002042Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 1
2024-07-31 11:21:42 2024-07-31T05:51:42.003971Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
2024-07-31 11:21:42 2024-07-31T05:51:42.015651Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-07-31 11:21:42 2024-07-31T05:51:42.728776Z 1 [ERROR] [MY-013171] [InnoDB] Cannot boot server version 80030 on data directory built by version 80200. Downgrade is not supported
2024-07-31 11:21:42 mysqld: Can't open file: 'mysql.ibd' (errno: 0 - )
2024-07-31 11:21:43 2024-07-31T05:51:43.160403Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2024-07-31 11:21:43 2024-07-31T05:51:43.160599Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2024-07-31 11:21:43 2024-07-31T05:51:43.160608Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-07-31 11:21:43 2024-07-31T05:51:43.161199Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30)  MySQL Community Server - GPL.

And Localstack Error stack trace:

2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 275, in _raise_for_status
2024-07-31 11:22:56     response.raise_for_status()
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
2024-07-31 11:22:56     raise HTTPError(http_error_msg, response=self)
2024-07-31 11:22:56 requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.43/containers/43973e2ebf6c9b5012d2655a8bc62ad2c6db0f6f86bd3780337e640c4e30c4c7/exec
2024-07-31 11:22:56 
2024-07-31 11:22:56 The above exception was the direct cause of the following exception:
2024-07-31 11:22:56 
2024-07-31 11:22:56 Traceback (most recent call last):
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/container_utils/docker_sdk_client.py", line 818, in exec_in_container
2024-07-31 11:22:56     result = container.exec_run(
2024-07-31 11:22:56              ^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/models/containers.py", line 208, in exec_run
2024-07-31 11:22:56     resp = self.client.api.exec_create(
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
2024-07-31 11:22:56     return f(self, resource_id, *args, **kwargs)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/exec_api.py", line 78, in exec_create
2024-07-31 11:22:56     return self._result(res, True)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 281, in _result
2024-07-31 11:22:56     self._raise_for_status(response)
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 277, in _raise_for_status
2024-07-31 11:22:56     raise create_api_error_from_http_exception(e) from e
2024-07-31 11:22:56           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
2024-07-31 11:22:56     raise cls(e, response=response, explanation=explanation) from e
2024-07-31 11:22:56 docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.43/containers/43973e2ebf6c9b5012d2655a8bc62ad2c6db0f6f86bd3780337e640c4e30c4c7/exec: Conflict ("Container 43973e2ebf6c9b5012d2655a8bc62ad2c6db0f6f86bd3780337e640c4e30c4c7 is not running")
2024-07-31 11:22:56 
2024-07-31 11:22:56 The above exception was the direct cause of the following exception:
2024-07-31 11:22:56 
2024-07-31 11:22:56 Traceback (most recent call last):
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/db_utils.py.enc", line 63, in A
2024-07-31 11:22:56     try:B and B(status='creating');G=F.start_db_instance(*E,**D);B and B(status=_D);LOG.debug('DB started successfully.');return G
2024-07-31 11:22:56                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 61, in start_db_instance
2024-07-31 11:22:56     retry(T,sleep=3,retries=30,sleep_before=2)
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/sync.py", line 63, in retry
2024-07-31 11:22:56     raise raise_error
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/sync.py", line 59, in retry
2024-07-31 11:22:56     return function(**kwargs)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 60, in T
2024-07-31 11:22:56     def T():A='mysqladmin ping -p$MYSQL_ROOT_PASSWORD --protocol tcp';A=[P,'-c',A];assert D.container.exec(command=A)
2024-07-31 11:22:56                                                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/container/container.py.enc", line 21, in B
2024-07-31 11:22:56     return A(self,*B,**C)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/container/docker_container.py.enc", line 19, in exec
2024-07-31 11:22:56     def exec(self,command):return DOCKER_CLIENT.exec_in_container(container_name_or_id=self.id,command=command)
2024-07-31 11:22:56                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/container_utils/docker_sdk_client.py", line 859, in exec_in_container
2024-07-31 11:22:56     raise ContainerException() from e
2024-07-31 11:22:56 localstack.utils.container_utils.container_client.ContainerException
2024-07-31 11:22:56  stdout: 'None' stderr: 'None'
2024-07-31 11:22:56 2024-07-31T05:52:56.705  WARN --- [functhread16] l.p.c.s.rds.db_utils       : Unable to startup DB instance:  Traceback (most recent call last):
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 275, in _raise_for_status
2024-07-31 11:22:56     response.raise_for_status()
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
2024-07-31 11:22:56     raise HTTPError(http_error_msg, response=self)
2024-07-31 11:22:56 requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.43/containers/df626d62cb79cffdc7c50ec819bf465ae98bd0fe9c99b1a3931d62a199f9eff0/exec
2024-07-31 11:22:56 
2024-07-31 11:22:56 The above exception was the direct cause of the following exception:
2024-07-31 11:22:56 
2024-07-31 11:22:56 Traceback (most recent call last):
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/container_utils/docker_sdk_client.py", line 818, in exec_in_container
2024-07-31 11:22:56     result = container.exec_run(
2024-07-31 11:22:56              ^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/models/containers.py", line 208, in exec_run
2024-07-31 11:22:56     resp = self.client.api.exec_create(
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
2024-07-31 11:22:56     return f(self, resource_id, *args, **kwargs)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/exec_api.py", line 78, in exec_create
2024-07-31 11:22:56     return self._result(res, True)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 281, in _result
2024-07-31 11:22:56     self._raise_for_status(response)
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 277, in _raise_for_status
2024-07-31 11:22:56     raise create_api_error_from_http_exception(e) from e
2024-07-31 11:22:56           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
2024-07-31 11:22:56     raise cls(e, response=response, explanation=explanation) from e
2024-07-31 11:22:56 docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.43/containers/df626d62cb79cffdc7c50ec819bf465ae98bd0fe9c99b1a3931d62a199f9eff0/exec: Conflict ("Container df626d62cb79cffdc7c50ec819bf465ae98bd0fe9c99b1a3931d62a199f9eff0 is not running")
2024-07-31 11:22:56 
2024-07-31 11:22:56 The above exception was the direct cause of the following exception:
2024-07-31 11:22:56 
2024-07-31 11:22:56 Traceback (most recent call last):
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/db_utils.py.enc", line 63, in A
2024-07-31 11:22:56     try:B and B(status='creating');G=F.start_db_instance(*E,**D);B and B(status=_D);LOG.debug('DB started successfully.');return G
2024-07-31 11:22:56                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 61, in start_db_instance
2024-07-31 11:22:56     retry(T,sleep=3,retries=30,sleep_before=2)
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/sync.py", line 63, in retry
2024-07-31 11:22:56     raise raise_error
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/sync.py", line 59, in retry
2024-07-31 11:22:56     return function(**kwargs)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 60, in T
2024-07-31 11:22:56     def T():A='mysqladmin ping -p$MYSQL_ROOT_PASSWORD --protocol tcp';A=[P,'-c',A];assert D.container.exec(command=A)
2024-07-31 11:22:56                                                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/container/container.py.enc", line 21, in B
2024-07-31 11:22:56     return A(self,*B,**C)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/container/docker_container.py.enc", line 19, in exec
2024-07-31 11:22:56     def exec(self,command):return DOCKER_CLIENT.exec_in_container(container_name_or_id=self.id,command=command)
2024-07-31 11:22:56                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/container_utils/docker_sdk_client.py", line 859, in exec_in_container
2024-07-31 11:22:56     raise ContainerException() from e
2024-07-31 11:22:56 localstack.utils.container_utils.container_client.ContainerException
2024-07-31 11:22:56  stdout: 'None' stderr: 'None'
2024-07-31 11:22:56 2024-07-31T05:52:56.705 DEBUG --- [functhread12] l.p.c.s.rds.db_utils       : Container Logs:
2024-07-31 11:22:56 2024-07-31T05:52:56.706 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : Container Logs:
2024-07-31 11:22:56 2024-07-31T05:52:56.707  WARN --- [functhread15] l.p.c.s.rds.db_utils       : Unable to startup DB instance:  Traceback (most recent call last):
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 275, in _raise_for_status
2024-07-31 11:22:56     response.raise_for_status()
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
2024-07-31 11:22:56     raise HTTPError(http_error_msg, response=self)
2024-07-31 11:22:56 requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.43/containers/f7c27ba3b10c93061bb2169c9db9cf811b9c4adc8b18be81ff663cff32e46a7f/exec
2024-07-31 11:22:56 
2024-07-31 11:22:56 The above exception was the direct cause of the following exception:
2024-07-31 11:22:56 
2024-07-31 11:22:56 Traceback (most recent call last):
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/container_utils/docker_sdk_client.py", line 818, in exec_in_container
2024-07-31 11:22:56     result = container.exec_run(
2024-07-31 11:22:56              ^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/models/containers.py", line 208, in exec_run
2024-07-31 11:22:56     resp = self.client.api.exec_create(
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
2024-07-31 11:22:56     return f(self, resource_id, *args, **kwargs)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/exec_api.py", line 78, in exec_create
2024-07-31 11:22:56     return self._result(res, True)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 281, in _result
2024-07-31 11:22:56     self._raise_for_status(response)
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 277, in _raise_for_status
2024-07-31 11:22:56     raise create_api_error_from_http_exception(e) from e
2024-07-31 11:22:56           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
2024-07-31 11:22:56     raise cls(e, response=response, explanation=explanation) from e
2024-07-31 11:22:56 docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.43/containers/f7c27ba3b10c93061bb2169c9db9cf811b9c4adc8b18be81ff663cff32e46a7f/exec: Conflict ("Container f7c27ba3b10c93061bb2169c9db9cf811b9c4adc8b18be81ff663cff32e46a7f is not running")
2024-07-31 11:22:56 
2024-07-31 11:22:56 The above exception was the direct cause of the following exception:
2024-07-31 11:22:56 
2024-07-31 11:22:56 Traceback (most recent call last):
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/db_utils.py.enc", line 63, in A
2024-07-31 11:22:56     try:B and B(status='creating');G=F.start_db_instance(*E,**D);B and B(status=_D);LOG.debug('DB started successfully.');return G
2024-07-31 11:22:56                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 61, in start_db_instance
2024-07-31 11:22:56     retry(T,sleep=3,retries=30,sleep_before=2)
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/sync.py", line 63, in retry
2024-07-31 11:22:56     raise raise_error
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/sync.py", line 59, in retry
2024-07-31 11:22:56     return function(**kwargs)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 60, in T
2024-07-31 11:22:56     def T():A='mysqladmin ping -p$MYSQL_ROOT_PASSWORD --protocol tcp';A=[P,'-c',A];assert D.container.exec(command=A)
2024-07-31 11:22:56                                                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/container/container.py.enc", line 21, in B
2024-07-31 11:22:56     return A(self,*B,**C)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/container/docker_container.py.enc", line 19, in exec
2024-07-31 11:22:56     def exec(self,command):return DOCKER_CLIENT.exec_in_container(container_name_or_id=self.id,command=command)
2024-07-31 11:22:56                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/container_utils/docker_sdk_client.py", line 859, in exec_in_container
2024-07-31 11:22:56     raise ContainerException() from e
2024-07-31 11:22:56 localstack.utils.container_utils.container_client.ContainerException
2024-07-31 11:22:56  stdout: 'None' stderr: 'None'
2024-07-31 11:22:56 2024-07-31T05:52:56.710 DEBUG --- [functhread15] l.p.c.s.rds.db_utils       : Container Logs:
2024-07-31 11:22:56 2024-07-31T05:52:56.708  WARN --- [functhread13] l.p.c.s.rds.db_utils       : Unable to startup DB instance:  Traceback (most recent call last):
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 275, in _raise_for_status
2024-07-31 11:22:56     response.raise_for_status()
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
2024-07-31 11:22:56     raise HTTPError(http_error_msg, response=self)
2024-07-31 11:22:56 requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.43/containers/a947ff4096c47ddb968d1cca1651b5474003fca39d8ee5bc8a82dff39d6732a0/exec
2024-07-31 11:22:56 
2024-07-31 11:22:56 The above exception was the direct cause of the following exception:
2024-07-31 11:22:56 
2024-07-31 11:22:56 Traceback (most recent call last):
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/container_utils/docker_sdk_client.py", line 818, in exec_in_container
2024-07-31 11:22:56     result = container.exec_run(
2024-07-31 11:22:56              ^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/models/containers.py", line 208, in exec_run
2024-07-31 11:22:56     resp = self.client.api.exec_create(
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
2024-07-31 11:22:56     return f(self, resource_id, *args, **kwargs)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/exec_api.py", line 78, in exec_create
2024-07-31 11:22:56     return self._result(res, True)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 281, in _result
2024-07-31 11:22:56     self._raise_for_status(response)
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 277, in _raise_for_status
2024-07-31 11:22:56     raise create_api_error_from_http_exception(e) from e
2024-07-31 11:22:56           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
2024-07-31 11:22:56     raise cls(e, response=response, explanation=explanation) from e
2024-07-31 11:22:56 docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.43/containers/a947ff4096c47ddb968d1cca1651b5474003fca39d8ee5bc8a82dff39d6732a0/exec: Conflict ("Container a947ff4096c47ddb968d1cca1651b5474003fca39d8ee5bc8a82dff39d6732a0 is not running")
2024-07-31 11:22:56 
2024-07-31 11:22:56 The above exception was the direct cause of the following exception:
2024-07-31 11:22:56 
2024-07-31 11:22:56 Traceback (most recent call last):
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/db_utils.py.enc", line 63, in A
2024-07-31 11:22:56     try:B and B(status='creating');G=F.start_db_instance(*E,**D);B and B(status=_D);LOG.debug('DB started successfully.');return G
2024-07-31 11:22:56                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 61, in start_db_instance
2024-07-31 11:22:56     retry(T,sleep=3,retries=30,sleep_before=2)
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/sync.py", line 63, in retry
2024-07-31 11:22:56     raise raise_error
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/sync.py", line 59, in retry
2024-07-31 11:22:56     return function(**kwargs)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 60, in T
2024-07-31 11:22:56     def T():A='mysqladmin ping -p$MYSQL_ROOT_PASSWORD --protocol tcp';A=[P,'-c',A];assert D.container.exec(command=A)
2024-07-31 11:22:56                                                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/container/container.py.enc", line 21, in B
2024-07-31 11:22:56     return A(self,*B,**C)
2024-07-31 11:22:56            ^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/container/docker_container.py.enc", line 19, in exec
2024-07-31 11:22:56     def exec(self,command):return DOCKER_CLIENT.exec_in_container(container_name_or_id=self.id,command=command)
2024-07-31 11:22:56                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-31 11:22:56   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/container_utils/docker_sdk_client.py", line 859, in exec_in_container
2024-07-31 11:22:56     raise ContainerException() from e
2024-07-31 11:22:56 localstack.utils.container_utils.container_client.ContainerException
2024-07-31 11:22:56  stdout: 'None' stderr: 'None'
2024-07-31 11:22:56 2024-07-31T05:52:56.711 DEBUG --- [functhread13] l.p.c.s.rds.db_utils       : Container Logs:
2024-07-31 11:22:56 2024-07-31T05:52:56.722 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31 05:51:40+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.30-1.el8 started.
2024-07-31 11:22:56 2024-07-31T05:52:56.722 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31 05:51:40+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31 05:51:40+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.30-1.el8 started.
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31T05:51:41.996434Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31T05:51:42.002042Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 1
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31T05:51:42.003971Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31T05:51:42.015651Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31T05:51:42.728776Z 1 [ERROR] [MY-013171] [InnoDB] Cannot boot server version 80030 on data directory built by version 80200. Downgrade is not supported
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] mysqld: Can't open file: 'mysql.ibd' (errno: 0 - )
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31T05:51:43.160403Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31T05:51:43.160599Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31T05:51:43.160608Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-07-31 11:22:56 2024-07-31T05:52:56.723 DEBUG --- [functhread16] l.p.c.s.rds.db_utils       : [ls-mysql-7fd6a10f] 2024-07-31T05:51:43.161199Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30)  MySQL Community Server - GPL.
2024-07-31 11:22:56 2024-07-31T05:52:56.725 DEBUG --- [functhread12] l.p.c.s.rds.db_utils       : [ls-mysql-a7346c06] 2024-07-31 05:51:40+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.30-1.el8 started.
2024-07-31 11:22:56 2024-07-31T05:52:56.725 DEBUG --- [functhread12] l.p.c.s.rds.db_utils       : [ls-mysql-a7346c06] 2024-07-31 05:51:40+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-07-31 11:22:56 2024-07-31T05:52:56.725 DEBUG --- [functhread12] l.p.c.s.rds.db_utils       : [ls-mysql-a7346c06] 2024-07-31 05:51:41+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.30-1.el8 started.

@cloutierMat
Copy link
Contributor

This log seems relevant Cannot boot server version 80030 on data directory built by version 80200. Downgrade is not supported. It seems to be trying to to start with an older version of mysql data directory?

Can you please try and clear the volume folder?
https://docs.localstack.cloud/references/filesystem/#localstack-volume

@hrishi-hugo
Copy link

Thanks @cloutierMat. Working after deleting the old data folder.

@cloutierMat
Copy link
Contributor

Awesome! I love to hear that!

Thank you for your feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: integration/pulumi Issues related to Pulumi aws:rds Amazon Relational Database Service status: backlog Triaged but not yet being worked on type: bug Bug report
Projects
None yet
Development

No branches or pull requests

5 participants