Docker Compose Ingestion Container Bootstrap Error

[leobiscassi]

Affected module
Does it impact the UI, backend or Ingestion Framework?

It affects the backend / ingestion framework.

Describe the bug

I was trying to deploy OM using the docker-compose-postgres.yml file available in the releases page (tried with versions 1.0.3, 1.0.4 and 1.0.5) and I got the following error in the openmetadata_ingestion container logs:

sed: -e expression #1, char 90: unknown option to `s'
Traceback (most recent call last):
 File "/home/airflow/.local/bin/airflow", line 5, in <module>
  from airflow.__main__ import main
 File "/home/airflow/.local/lib/python3.9/site-packages/airflow/__init__.py", line 35, in <module>
  from airflow import settings
 File "/home/airflow/.local/lib/python3.9/site-packages/airflow/settings.py", line 35, in <module>
  from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf # NOQA F401
 File "/home/airflow/.local/lib/python3.9/site-packages/airflow/configuration.py", line 1604, in <module>
  conf.validate()
 File "/home/airflow/.local/lib/python3.9/site-packages/airflow/configuration.py", line 314, in validate
  self._validate_config_dependencies()
 File "/home/airflow/.local/lib/python3.9/site-packages/airflow/configuration.py", line 411, in _validate_config_dependencies
  raise AirflowConfigException(f"error: cannot use sqlite with the {self.get('core', 'executor')}")
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor

My .env file looks like this:

# POSTGRES Environment Variables
DB_DRIVER_CLASS='org.postgresql.Driver'
DB_SCHEME='postgresql'
DB_USE_SSL='true'
DB_USER_PASSWORD='my_password'
DB_HOST='my-rds-endpoint.us-west-2.rds.amazonaws.com'
DB_USER='postgres'
OM_DATABASE='my_openmetadata_db'
DB_PORT='5432'
# ElasticSearch Environment Variables
ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60'
ELASTICSEARCH_USER='my_openmetadata_opensearch'
ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5'
ELASTICSEARCH_PORT='443'
ELASTICSEARCH_SCHEME='https'
ELASTICSEARCH_BATCH_SIZE='10'
ELASTICSEARCH_HOST='my-opensearch-endpoint.us-west-2.es.amazonaws.com'
ELASTICSEARCH_PASSWORD='my_password'
# Airflow
AIRFLOW_DB_HOST='my-rds-endpoint.us-west-2.rds.amazonaws.com'
AIRFLOW_DB_PORT='5432'
AIRFLOW_DB='my_airflow_db'
AIRFLOW_DB_USER='postgres'
AIRFLOW_DB_SCHEME='postgresql+psycopg2'
AIRFLOW_DB_PASSWORD='my_password'

After getting a call with @akash-jain-10 he had pointed out that probably is related to the password with special characters automatically created by AWS Secrets Manager. After rotating the password we were able to deploy properly.

To Reproduce

This is the password (rs&FJy}((i:Skl1u3X:qEm.3$l6#) that I was getting the error in the bootstrap of openmetadata_ingestion container.

Screenshots or steps to reproduce

Expected behavior
A clear and concise description of what you expected to happen.

Version:

• OS: [e.g. iOS] Amazon Linux 2
• Python version: shipped with collate docker images
• OpenMetadata version: [e.g. 0.8] 1.0.3, 1.0.4 and 1.0.5
• OpenMetadata Ingestion package version: [e.g. openmetadata-ingestion[docker]==XYZ] shipped with collate docker images

Additional context
Add any other context about the problem here.

[dhruvinmaniar123]

Hello @leobiscassi ,

We have tested the above scenario on our end and followed below steps to resolved it.

Steps:

1.) When utilizing a custom password containing special characters such as "%#@!" in your environment file, it is crucial to convert the password string to URL encoding. For instance, if the provided password is encoded, it will resemble the following: rs%26FJy%7D%28%28i%3ASkl1u3X%3AqEm.3%24l6%23. However, it is important to note that a single % must be replaced with %% to ensure proper value parsing. Consequently, the encoded password should appear as follows: rs%%26FJy%%7D%%28%%28i%%3ASkl1u3X%%3AqEm.3%%24l6%%23.

2.) Once you have completed the step-1, execute the compose file once again and verify whether you can establish a successful connection. We have successfully achieved this outcome, and we would like you to try the same and let us know how it goes.

Please share your results with us, and feel free to reach out if you require any further assistance.

CC: @akash-jain-10

Thanks,
Dhruvin.

[leobiscassi]

Hey @dhruvinmaniar123,
I'm not having this issue, just placed this ticket for us to find the issue and add a note in the docs, since this password was automatically created by secrets manager. I saw that you already opened a doc update for this, so I'm closing it. Thanks a lot!