The repository for docker-compose files.
The files reference private Docker Hub repository and cannot be used without proper login.
If you feel authorized, please request the access from the authors.
Please make sure you have an acces to private docker repo for download TDspora.
Clean up previous installation
If you have previous installation, please remove old volume manually:
docker volume ls --filter name=.jars -q | xargs docker volume rm -fDownload start.sh file and execute it
curl -LO https://raw.githubusercontent.com/tdspora/deploy/main/start.sh && chmod +x start.sh && ./start.shNote: Add -d parameter to start the application as a backgroud process (daemon).
curl -LO https://raw.githubusercontent.com/tdspora/deploy/main/start.sh && chmod +x start.sh && ./start.sh -dThe start.sh script will pull new images every time you run it. To avoid excessive downloads, use below command to start the application normally
docker-compose up -dIf you want to change some of the default values
- Create file
.envin the same folder asdocker-compose.yml - Add values from the list below you would like to override
TDM_FERNET_KEY=jNICCd9MFHW++hGZ9cUAGmsdOCUCtcaI1ZM+2F+pXGs=Encryption key for the Spark job descriptors that can contain secrets such as database user and password.
The simpliest way to generate new key is to run dd if=/dev/urandom bs=32 count=1 2>/dev/null | openssl base64 in bash.
COMPOSE_PROJECT_NAME=localAll the containers will have this prefix added to the container name.
TDM_ADMIN_EMAIL=admin@companyname.comNot used. Will be implemented later.
TDM_ADMIN_USERNAME=adminAdministrator user name. This is theonly user name supported.
TDM_ADMIN_PASSWORD=<password>Administrator password. There is no rules or limitation for the password complexity.
TDM_APPLICATION_KEY=594df9ea-3565-4145-b9c4-9bbc47ff6200UUID for the application. Static value used in combination with API key to access REST API.
TDM_RABBIT_MQ_USER=tdmUser name for the RabbitMQ container.
TDM_RABBIT_MQ_PASSWORD=<rabbit_mq_password>Password for the default RabbitMQ user.
TDM_SPARK_SSH_USER=tdmName of the user the application use to access internal Spark container and submitting Spark jobs.
TDM_SPARK_SSH_PASSWORD=<spark_ssh_password>Password for the SSH user.
TDM_DB_USER=tdmUser name the application will use to access internal PostgreSQL instance.
TDM_DB_PASSWORD=<tdm_db_password>The PostgreSQL user password.
TDM_POSTGRES_PORT=5432The internal PostgreSQL instance will expose this port for external connections.
TDM_PORT=80The user interface for the application will be available on this port. Default URL: http://localhost:80/
TDM_UI_REFINITIV_PORT=32082The application user interface with altered theme will be available on this port. Default URL: http://localhost:32082/
TDM_SPARK_UI_PORT=8080The port exposed for the Apache Spark master node UI.
TDM_SPARK_HISTORY_PORT=18080The port exposed for the Apache Spark History Server UI.
TDM_LOG_LEVEL=DEBUGGranularity level of logs collected during execution of pipelines. Valid values are ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and OFF.
TDM_HOSTNAME=localhostThe FQDN of the host serving the application REST API endpoints. Along with TDM_API_PORT defines full URL (TDM_URL) that the backend includes into job descriptors. The pipelines execution engine reports status of the job to the ${TDM_URL}/v1/executions/<executionId>/progress endpoint. If this endpoint is not accessible from the Apache Spark cluster, the status will not be updated dynamically, but only at the end of the job execution.
If you are performing an upgrade you have to backup all your configuration data persisted in a PostgreSQL database.
Check the variable COMPOSE_PROJECT_NAME in your .env. file
cat .env | grep COMPOSE_PROJECT_NAME- At first, you have to save data from the Postgres container.
You can run a simple command which will save Postgres dump on your
/tmpfolder (not in the container!)
docker exec -it -u postgres <project_name>_postgres_1 pg_dumpall > /tmp/tdm.dumpBefore you will start the restore process, please make sure you have the dump file in your folder!
- Go to the folder with the docker-compose file and stop all the containers
docker-compose down- Let's take a look at the volumes we have
docker volume ls- Postgres volume must be something like
<compose_project_name>_postgres_1. Delete it with
docker volume rm <compose_project_name>_postgres- Launch TDspora with docker-compose
docker-compose pull && docker-compose up -d- Do not login into TDspora at that moment*. Copy dump to the container:
docker cp /tmp/tdm.dump <container_name>:/tmp*If you performed login into TDspora you have to restart the PostgreSQL container because login operation creating session within Postgres DB and you will not be able to drop database
- Login into PostgreSQL container (or use Portainer) with PostgreSQL user and bash shell:
docker exec -it -u postgres <container_name> bash- Run the following commands within the Postgres container
psql -U postgres -c "DROP DATABASE tdm;"
psql -U postgres -c "CREATE DATABASE tdm;"
psql -U postgres -d tdm -f /tmp/tdm.dump &> /tmp/restore_tdm.logYou can see some errors in log file such as
psql:/tmp/tdm.dump:14: ERROR: role "postgres" already exists
psql:/tmp/tdm.dump:16: ERROR: role "tdm" already exists
psql:/tmp/tdm.dump:28: ERROR: database "tdm" already exists
psql:/tmp/tdm.dump:30: ERROR: database "tdm_test" already existsIt's ok and should not affect on any data.
If you want ot rollback to the specific version copy rollback.sh file near to docker-compose.yml file and launch it with
bash rollback.shAfter that you have to provide a version which you want to rollback to. The script will make a separate folder (named by current date) in your home folder and copy an old docker-compose.yml file and postgres database dump into it.