-
Notifications
You must be signed in to change notification settings - Fork 369
feat: add helm chart and docker compose for hybrid executors #4084
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Binary file added
BIN
+626 KB
...ures/scheduler/hybrid_executors/hybrid-executors_standard-hybrid-deployment.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
119 changes: 119 additions & 0 deletions
119
docs/cloud/features/scheduler/hybrid_executors_docker_compose.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,119 @@ | ||||||||
| # Tobiko Cloud Hybrid Executors - Docker Compose Setup | ||||||||
|
|
||||||||
| This Docker Compose configuration allows you to run Tobiko Cloud hybrid executors locally or on any server that supports Docker Compose. | ||||||||
|
|
||||||||
| Hybrid executors allow you to run operations on your own infrastructure while leveraging Tobiko Cloud for orchestration. | ||||||||
|
|
||||||||
| ## What this setup provides | ||||||||
|
|
||||||||
| This setup deploys two hybrid executors that pass work tasks from Tobiko Cloud to your data warehouse in a secure way: | ||||||||
|
|
||||||||
| - **Apply Executor**: Handles applying changes to the data warehouse | ||||||||
| - **Run Executor**: Handles scheduled model execution | ||||||||
|
|
||||||||
| Both executors must be properly configured with environment variables to connect to Tobiko Cloud and your data warehouse. | ||||||||
|
|
||||||||
| ## Prerequisites | ||||||||
|
|
||||||||
| - Access to a [data warehouse supported by Tobiko Cloud](../../../integrations/overview.md#execution-engines) (e.g., Postgres, Snowflake, BigQuery) | ||||||||
| - Docker and Docker Compose | ||||||||
| - A Tobiko Cloud account with [client ID and client secret](../single_sign_on.md#provisioning-client-credentials) | ||||||||
|
|
||||||||
| ## Quick start guide | ||||||||
|
|
||||||||
| 1. **Get docker-compose file**: | ||||||||
|
|
||||||||
| Download the [docker-compose.yml](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/docker-compose.yml) and [.env.example](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/.env.example) files to a local directory. | ||||||||
|
|
||||||||
| 2. **Create your environment file**: | ||||||||
|
|
||||||||
| Copy the downloaded example environment file into a new `.env` file: | ||||||||
|
|
||||||||
| ```bash | ||||||||
| cp .env.example .env | ||||||||
| ``` | ||||||||
|
|
||||||||
| 3. **Edit the .env file** with your project's configuration: | ||||||||
|
|
||||||||
| - Set your Tobiko Cloud organization, project, client ID, and client secret | ||||||||
| - Configure your gateway connection details | ||||||||
| - Adjust resource limits if needed | ||||||||
|
|
||||||||
| 4. **Start the executors**: | ||||||||
|
|
||||||||
| ```bash | ||||||||
| docker compose up -d | ||||||||
| ``` | ||||||||
|
|
||||||||
| 5. **Check the logs**: | ||||||||
|
|
||||||||
| ```bash | ||||||||
| docker compose logs -f | ||||||||
| ``` | ||||||||
|
|
||||||||
| ## Configuration options | ||||||||
|
|
||||||||
| ### Gateway configuration | ||||||||
|
|
||||||||
| The default configuration in the `docker-compose.yml` file uses Postgres, but you can use [any supported SQL engine](../../../integrations/overview.md#execution-engines) by adjusting the connection parameters in your `.env` file. | ||||||||
|
|
||||||||
| #### Multiple gateways | ||||||||
|
|
||||||||
| To configure multiple gateways, add additional environment variables for each gateway the `docker-compose.yml` file: | ||||||||
|
|
||||||||
| ```yaml | ||||||||
| environment: | ||||||||
| # First gateway | ||||||||
| SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__TYPE: ${DB_TYPE:-postgres} | ||||||||
| # ... other GATEWAY_A configuration ... | ||||||||
|
|
||||||||
| # Second gateway | ||||||||
| SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__TYPE: snowflake | ||||||||
| SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__ACCOUNT: ${SNOWFLAKE_ACCOUNT} | ||||||||
| # ... other GATEWAY_B configuration ... | ||||||||
| ``` | ||||||||
|
|
||||||||
| ## Health checking | ||||||||
|
|
||||||||
| Verify the health of your executors by running these commands: | ||||||||
|
|
||||||||
| ```bash | ||||||||
| docker compose exec apply-executor /app/pex executor apply --check | ||||||||
| docker compose exec run-executor /app/pex executor run --check | ||||||||
| ``` | ||||||||
|
|
||||||||
| Example successful output: | ||||||||
|
|
||||||||
| ```bash | ||||||||
| > docker compose exec apply-executor /app/pex executor apply --check | ||||||||
| 2025-04-09 21:24:49,873 - MainThread - httpx - INFO - HTTP Request: GET https://cloud.tobikodata.com/sqlmesh/<YOUR ORG>/<YOUR PROJECT>/api/state-sync/enterprise-version/upgrade "HTTP/1.1 200 OK" (_client.py:1025) | ||||||||
| 2025-04-09 21:24:49,889 - MainThread - tobikodata.tcloud.installer - INFO - Executor is installed (installer.py:180) | ||||||||
| ``` | ||||||||
|
|
||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||
| In addition, ensure the executors are healthy by running `echo $?` to confirm the check command returned exit code 0. | ||||||||
|
|
||||||||
| ## Stopping the executors | ||||||||
|
|
||||||||
| To stop the executors: | ||||||||
|
|
||||||||
| ```bash | ||||||||
| docker compose down | ||||||||
| ``` | ||||||||
|
|
||||||||
| ## Troubleshooting | ||||||||
|
|
||||||||
| If you encounter issues: | ||||||||
|
|
||||||||
| 1. Check the logs: `docker compose logs -f` | ||||||||
| 2. Verify your connection settings in the `.env` file | ||||||||
| 3. Ensure your client ID and client secret are correct | ||||||||
| 4. Check that your SQL engine is accessible from the Docker containers | ||||||||
|
|
||||||||
| ## Security considerations | ||||||||
|
|
||||||||
| !!! warning "Never commit .env to version control" | ||||||||
|
|
||||||||
| The `.env` file contains sensitive information. Never commit it to version control. | ||||||||
|
|
||||||||
| - Consider using Docker secrets or a secrets management solution in production environments. | ||||||||
| - For production deployments, consider using the Kubernetes Helm chart instead, which offers more robust reliability and secret management options. | ||||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I run the check command and see output like below, do I still need to run
echo $?to check the exit code?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the log looks like what is provided then it is unlikely that the exit code isn't 0, but there is still a chance. Therefore it would probably be best to make sure.