Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
119 changes: 119 additions & 0 deletions docs/cloud/features/scheduler/hybrid_executors_docker_compose.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# Tobiko Cloud Hybrid Executors - Docker Compose Setup

This Docker Compose configuration allows you to run Tobiko Cloud hybrid executors locally or on any server that supports Docker Compose.

Hybrid executors allow you to run operations on your own infrastructure while leveraging Tobiko Cloud for orchestration.

## What this setup provides

This setup deploys two hybrid executors that pass work tasks from Tobiko Cloud to your data warehouse in a secure way:

- **Apply Executor**: Handles applying changes to the data warehouse
- **Run Executor**: Handles scheduled model execution

Both executors must be properly configured with environment variables to connect to Tobiko Cloud and your data warehouse.

## Prerequisites

- Access to a [data warehouse supported by Tobiko Cloud](../../../integrations/overview.md#execution-engines) (e.g., Postgres, Snowflake, BigQuery)
- Docker and Docker Compose
- A Tobiko Cloud account with [client ID and client secret](../single_sign_on.md#provisioning-client-credentials)

## Quick start guide

1. **Get docker-compose file**:

Download the [docker-compose.yml](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/docker-compose.yml) and [.env.example](https://raw.githubusercontent.com/TobikoData/sqlmesh/refs/heads/main/docs/cloud/features/scheduler/scheduler/.env.example) files to a local directory.

2. **Create your environment file**:

Copy the downloaded example environment file into a new `.env` file:

```bash
cp .env.example .env
```

3. **Edit the .env file** with your project's configuration:

- Set your Tobiko Cloud organization, project, client ID, and client secret
- Configure your gateway connection details
- Adjust resource limits if needed

4. **Start the executors**:

```bash
docker compose up -d
```

5. **Check the logs**:

```bash
docker compose logs -f
```

## Configuration options

### Gateway configuration

The default configuration in the `docker-compose.yml` file uses Postgres, but you can use [any supported SQL engine](../../../integrations/overview.md#execution-engines) by adjusting the connection parameters in your `.env` file.

#### Multiple gateways

To configure multiple gateways, add additional environment variables for each gateway the `docker-compose.yml` file:

```yaml
environment:
# First gateway
SQLMESH__GATEWAYS__GATEWAY_A__CONNECTION__TYPE: ${DB_TYPE:-postgres}
# ... other GATEWAY_A configuration ...

# Second gateway
SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__TYPE: snowflake
SQLMESH__GATEWAYS__GATEWAY_B__CONNECTION__ACCOUNT: ${SNOWFLAKE_ACCOUNT}
# ... other GATEWAY_B configuration ...
```

## Health checking

Verify the health of your executors by running these commands:

```bash
docker compose exec apply-executor /app/pex executor apply --check
docker compose exec run-executor /app/pex executor run --check
```

Example successful output:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I run the check command and see output like below, do I still need to run echo $? to check the exit code?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the log looks like what is provided then it is unlikely that the exit code isn't 0, but there is still a chance. Therefore it would probably be best to make sure.


```bash
> docker compose exec apply-executor /app/pex executor apply --check
2025-04-09 21:24:49,873 - MainThread - httpx - INFO - HTTP Request: GET https://cloud.tobikodata.com/sqlmesh/<YOUR ORG>/<YOUR PROJECT>/api/state-sync/enterprise-version/upgrade "HTTP/1.1 200 OK" (_client.py:1025)
2025-04-09 21:24:49,889 - MainThread - tobikodata.tcloud.installer - INFO - Executor is installed (installer.py:180)
```

Copy link
Copy Markdown
Contributor

@treysp treysp Apr 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In addition, ensure the executors are healthy by running `echo $?` to confirm the check command returned exit code 0.

In addition, ensure the executors are healthy by running `echo $?` to confirm the check command returned exit code 0.

## Stopping the executors

To stop the executors:

```bash
docker compose down
```

## Troubleshooting

If you encounter issues:

1. Check the logs: `docker compose logs -f`
2. Verify your connection settings in the `.env` file
3. Ensure your client ID and client secret are correct
4. Check that your SQL engine is accessible from the Docker containers

## Security considerations

!!! warning "Never commit .env to version control"

The `.env` file contains sensitive information. Never commit it to version control.

- Consider using Docker secrets or a secrets management solution in production environments.
- For production deployments, consider using the Kubernetes Helm chart instead, which offers more robust reliability and secret management options.
Loading