diff --git a/.github/actions/spelling/allow.txt b/.github/actions/spelling/allow.txt index 50ba613..c4a31b6 100644 --- a/.github/actions/spelling/allow.txt +++ b/.github/actions/spelling/allow.txt @@ -786,4 +786,11 @@ WSS executability NTZ snowflakecomputing +aat +cicd +lpt +unconfigured +unpublishes + + diff --git a/docs/images/cicd_config_deployment_flow.png b/docs/images/cicd_config_deployment_flow.png new file mode 100644 index 0000000..ee9ac99 Binary files /dev/null and b/docs/images/cicd_config_deployment_flow.png differ diff --git a/docs/images/create_project.png b/docs/images/create_project.png new file mode 100644 index 0000000..7b26ecf Binary files /dev/null and b/docs/images/create_project.png differ diff --git a/docs/images/export_project.png b/docs/images/export_project.png new file mode 100644 index 0000000..eb005aa Binary files /dev/null and b/docs/images/export_project.png differ diff --git a/docs/images/git_config.png b/docs/images/git_config.png new file mode 100644 index 0000000..6344580 Binary files /dev/null and b/docs/images/git_config.png differ diff --git a/docs/images/github_actions_secrets_variables.png b/docs/images/github_actions_secrets_variables.png new file mode 100644 index 0000000..bda3ffe Binary files /dev/null and b/docs/images/github_actions_secrets_variables.png differ diff --git a/docs/images/github_environments_required_reviewers.png b/docs/images/github_environments_required_reviewers.png new file mode 100644 index 0000000..117bfcf Binary files /dev/null and b/docs/images/github_environments_required_reviewers.png differ diff --git a/docs/images/tapdata_import_project_preview.png b/docs/images/tapdata_import_project_preview.png new file mode 100644 index 0000000..e703109 Binary files /dev/null and b/docs/images/tapdata_import_project_preview.png differ diff --git a/docs/platform-ops/manage-project/README.md b/docs/platform-ops/manage-project/README.md new file mode 100644 index 0000000..ad0994e --- /dev/null +++ b/docs/platform-ops/manage-project/README.md @@ -0,0 +1,9 @@ +# Manage projects + +Use TapData projects to package tasks, APIs, and their dependent connections into versioned deployment units. With GitHub and automated pipelines, teams can promote the same configuration across development, test, and production environments with traceable changes and fewer manual steps. + +This section explains how project management works, how to set up the deployment pipeline, and how to create and deploy a project. If you only need to export an archive and import it manually in another environment, see [Create and deploy a project](deploy-project.md). + +import DocCardList from '@theme/DocCardList'; + + diff --git a/docs/platform-ops/manage-project/deploy-project.md b/docs/platform-ops/manage-project/deploy-project.md new file mode 100644 index 0000000..1711fd6 --- /dev/null +++ b/docs/platform-ops/manage-project/deploy-project.md @@ -0,0 +1,186 @@ +# Create and deploy a project + +After engineers configure TapData connections, tasks, and APIs, they can package those resources as a project, export the configuration, and deploy it to another environment. Deployment can run automatically through GitHub Actions or manually through file export and import. + +:::tip +This guide covers both automated deployment and manual import/export. If you plan to use GitHub and GitHub Actions, first complete [Set up an automated deployment pipeline](setup-pipeline.md). If you only need manual import/export, follow this guide and see [Appendix: Manually import configuration](#appendix-manually-import-configuration). +::: + +## Example scenario + +This guide uses a common real-time data warehouse scenario. A team synchronizes data from an Oracle source database to a Doris warehouse. The team has already verified wide-table synchronization tasks and an external API in the development environment. The next step is to promote the same configuration to system integration testing (SIT), and then to production. + +The following workflow shows how to create a project, export configuration, deploy automatically, and publish manually when needed. + +:::tip +Use the same connection names across environments. During automated deployment, TapData matches connection names to GitHub Secrets and Variables, then injects the real address, username, and password for the target environment. Use connection names that contain only letters, numbers, and underscores, such as `oracle_source`. +::: + +## Step 1: Create a project and select resources + +Package the team's tasks and API as one project. The project becomes the unit that you export, review, deploy, and roll back. + +1. Log in to the TapData console. In the left navigation pane, choose **Advanced Settings > Project Management**. +2. At the top of the left panel, click **+** to create a project. Enter a project name, such as `dw-pipeline`. Use the same name as the GitHub tenant repository when possible. +3. In the middle panel, switch between **Migrate Task**, **Transformation Task**, and **API**. Select `CRM_TO_DW`, `ORDER_TO_DW`, and `customer-api`, then click **Add selected >** to move them to the selected list. + + ![Create a project and select resources](../../images/create_project.png) + + :::tip + When you select tasks or APIs, TapData automatically includes dependent connections. In this example, `oracle_source` and `doris_target` are included automatically. + ::: + +4. Click **Save**. + +## Step 2: Connect the Git repository + +Connect the TapData project to the GitHub tenant repository. After the connection is configured, exports can be pushed to the repository and opened as Pull Requests without downloading and uploading files manually. + +1. In the upper-right corner, click **Git Configuration**. +2. In the dialog box, enter the GitHub tenant repository URL and personal access token. + + ![Configure the Git repository](../../images/git_config.png) + +3. Click **Save**. + +:::tip +If you do not want to integrate with GitHub, skip this step. In [Step 3](#step-3-export-the-configuration), choose file export and import the archive manually. +::: + +## Step 3: Export the configuration + +Export the project configuration from the development environment and submit it to GitHub for later deployment. + +1. On the **Project Management** page, click **Export** in the upper-right corner. In the export dialog box, select the project to export. +2. Select an **Export type**. + + ![Export a project](../../images/export_project.png) + + - **Git Export**: Available after a Git repository is connected. TapData pushes the configuration files to GitHub and creates a Pull Request. Enter the following information: + + | Field | Description | + | --- | --- | + | Branch name | The system generates a branch name that starts with `feat_` and includes a timestamp. You can edit the branch name. | + | PR title | A short summary of the change for review. | + | PR description | Optional. Describe why the change is needed and what it affects. | + + - **File Export**: Downloads the configuration as a compressed archive. Use this option when Git integration is not configured. For the import steps, see [Appendix: Manually import configuration](#appendix-manually-import-configuration). + +3. In the resource list, review the tasks and APIs to be exported. If the list is correct, click **Confirm Export**. + + :::tip + Connection credentials are masked during export. Database passwords and other sensitive fields are not written to the configuration files. Enable **Rerun** only when the target environment needs the task to run a full synchronization again, for example after adding source tables or changing primary keys. For routine changes, keep the default setting so tasks continue from the last checkpoint. + ::: + +
+Exported file structure + +Exported configuration is organized as a directory. Git export commits this directory to the repository. File export packages it as an archive. + +```text +{project-name}_tapdata_export/ +├── GroupInfo.json # Project metadata: project name, Git repository, and resource list +├── Connection/ # Connection configuration, including dependencies of tasks and APIs +│ ├── {id}_Connection_Config.json # Connection parameters with sensitive fields masked +│ └── {id}_Connection_Metadata.json # Table metadata for the connection +├── Task/ # Task configuration +│ ├── {id}_MigrateTask.json # Data replication task +│ └── {id}_SyncTask.json # Data transformation task +├── API/ # API configuration +│ ├── {id}_Module.json # API definition, including path, fields, and query logic +│ └── MetadataDefinition.json +└── User/ # User and role information for restoring operator context + ├── Users.json + ├── Roles.json + ├── RoleMappings.json + └── UserIdEmailMap.json +``` + +Notes: + +- **Connections**: TapData detects and exports connections based on task and API dependencies. You do not need to select them manually. +- **Sensitive information**: Usernames, passwords, and other credential fields are cleared during export. In automated deployment, TapData injects real values from GitHub Secrets and Variables. In manual import, update the connection values after import. +- **User data**: The `User` directory contains operator account and role information so the target environment can restore the user context. Passwords are stored as hashes and do not include plaintext values. + +
+ +## Step 4: Optional: Merge the Pull Request to deploy to development validation + +If the `dev` environment is configured, merge the Pull Request in GitHub to deploy the exported configuration to the development validation environment. This step verifies that the configuration files can be imported before you promote them further. If your process only uses SIT and production, skip this stage and adjust the tenant deployment workflow accordingly. + +1. In the GitHub tenant repository, open the Pull Request created by TapData. Review the exported configuration, then click **Merge**. +2. The merge triggers the GitHub Actions `TapData Deploy` workflow and deploys the configuration to the development validation environment. +3. If the preview shows changes to connections, tasks, or APIs, approve the `deploy` gate on the **Actions** page. +4. After deployment finishes, sign in to the development validation TapData environment. Verify that `CRM_TO_DW`, `ORDER_TO_DW`, and `customer-api` were imported correctly and that the connections pass the connection test. + +## Step 5: Create a tag to deploy to SIT + +After the configuration is ready for testing, create and push a Git tag to trigger deployment to the SIT environment. + +```bash +git tag v1.0.0 +git push origin v1.0.0 +``` + +After the tag is pushed, GitHub Actions starts the SIT deployment. If the preview shows changes to connections, tasks, or APIs, approve the `deploy` gate before the import continues. + +When deployment finishes, complete business validation in the SIT environment. Check functional correctness, data volume, synchronization latency, and other acceptance criteria. If validation passes, continue to production deployment. + +## Step 6: Trigger production deployment manually + +After SIT validation passes, deploy the same tag to production. Using the same tag ensures that production receives the exact configuration that passed testing. + +The official tenant template includes `dev`, `sit`, and `lpt` as manual deployment options by default. To deploy to `prod`, add `prod` to the tenant workflow options first. If your process includes performance testing or user acceptance testing, deploy to `lpt` or `aat` before production in the same way. + +1. In the GitHub tenant repository, go to **Actions**, and select `TapData Deploy`. +2. Click **Run workflow**. For **Branch**, select the tag, such as `v1.0.0`. For **Target environment**, select `prod`. If you need to deploy to performance testing or user acceptance testing first, select the configured `lpt` or `aat` environment. +3. Click **Run workflow**. If the preview shows changes to connections, tasks, or APIs, approve the `deploy` gate on the **Actions** page. +4. After deployment finishes, sign in to the target TapData environment. Verify task status and API availability, then complete the production release. + +## Roll back a deployment + +If a deployment does not behave as expected, such as when a task enters an abnormal state, roll the environment back to a previous stable tag. + +1. In the GitHub tenant repository, go to **Actions**, and select `TapData Rollback`. +2. Click **Run workflow**. Enter the target environment, such as `prod`, and the tag to roll back to, such as `v0.9.0`. +3. The rollback workflow stops the current tasks, unpublishes active APIs, clears the existing configuration, imports the configuration from the selected tag, and restores the runtime state for tasks and APIs that were active before rollback. +4. After rollback finishes, sign in to the target environment and verify the task status and API availability. + +Rollback affects only the selected target environment. Other environments are not changed. + +## FAQ + +**Q: How does project import handle existing resources?** + +Both automated deployment and manual import update existing connections, tasks, and APIs in the target environment based on the imported configuration. Unchanged resources remain unchanged. Resources that do not exist in the target environment are created. + +With GitHub-based deployment, real connection addresses, usernames, and passwords for new connections are injected from the target Environment Secrets and Variables based on connection names. With manual import, credentials are not injected automatically. Update or complete the connection values in the target environment after import. + +**Q: Why does GitHub Actions report `Could not find reusable workflow`?** + +- Check whether the Worker repository visibility is **Internal**. +- Check whether the Worker repository path in the tenant workflow has been replaced with the real value. + +**Q: The deployment succeeded, but the database password was not injected. What should I check?** + +- Check whether connection credentials are configured in the target Environment Secrets or Variables, not repository-level Secrets. +- Check whether the variable names exactly match the connection names in TapData after applying the naming rules. + +**Q: What should I do if the import script fails?** + +- Check whether `{ENV}_TAPDATA_ACCESS_CODE` is configured correctly and is still valid. +- Open the GitHub Actions logs, find the error returned by the TapData API, and troubleshoot based on that response. + +## Appendix: Manually import configuration + +Use manual import when GitHub integration is not configured or when you need to import a specific exported version directly into a target environment. + +1. On the **Advanced Settings > Export/Import** page, click **Import**. +2. Upload the archive exported from the development environment. +3. Select a conflict handling policy, such as **Skip** or **Update existing configuration**. +4. Click **OK**. TapData validates the file format and shows an import preview, including the affected connections, tasks, and APIs. + The following image shows the import preview page, which is used to confirm the resources to be added or updated. + + ![View project import preview](../../images/tapdata_import_project_preview.png) +5. Review the preview and run the import. +6. After import finishes, sign in to the target TapData environment. Update the real connection addresses, usernames, and passwords, test the database connections, verify task status and API availability, and then start the tasks. diff --git a/docs/platform-ops/manage-project/introduction.md b/docs/platform-ops/manage-project/introduction.md new file mode 100644 index 0000000..649675e --- /dev/null +++ b/docs/platform-ops/manage-project/introduction.md @@ -0,0 +1,49 @@ +# Overview + +Project management packages TapData connections, tasks, APIs, and related configuration resources into versioned deployment units. With GitHub and GitHub Actions, teams can promote tested configurations across development, test, and production environments while keeping deployments auditable and repeatable. + +## Why use projects + +Enterprise TapData deployments usually include multiple environments, such as development, system integration testing, performance testing, and production. Each environment has its own source databases, target databases, service endpoints, accounts, and network settings. + +Without a release process, engineers must recreate or update connections, tasks, and APIs in each environment by hand. As the number of resources grows, teams are more likely to miss dependencies, use the wrong parameters, or discover configuration drift only after a production incident. Manual promotion also makes it difficult to answer who changed what and when. + +TapData project management standardizes this process. It treats configuration as code, so teams can version, review, deploy, audit, and roll back data integration configurations through a controlled workflow. + +
+ +What is a project? + +In TapData, a **project** is a group of tasks, APIs, and dependent connections that share the same business goal. A project is the basic unit for unified management, export, deployment, and version tracking. + +Projects let you move related resources across environments as one package. This reduces the risk of missing dependent connections and keeps exported configuration and change history aligned with a business scope. + +
+ +## Deployment flow + +![Project deployment flow](../../images/cicd_config_deployment_flow.png) + +When a data integration requirement is ready, developers configure and verify tasks, APIs, and connections in a source environment. They then export the configuration as a TapData project and commit it to a GitHub repository for version control. + +When the configuration needs to move to the next environment, a GitHub Actions workflow starts the deployment. The workflow selects the target environment, such as system integration testing, performance testing, or production, and loads the corresponding connection details from GitHub Environment Secrets and Variables. + +Before importing resources, the pipeline previews the differences. After approval, it imports the configuration into the target TapData environment. Engineers no longer need to sign in to each environment and recreate resources one by one. Configuration files stay versioned, credentials stay isolated, and deployment history stays traceable. + +## Key capabilities + +- **Project-based packaging**: Group tasks, APIs, and connections into one managed project. TapData includes dependencies automatically so the exported package stays complete. +- **Configuration as code**: Store exported configuration in GitHub. Each change has a Git history, supports review, and can be compared or rolled back. +- **Automated releases**: Use GitHub Actions for cross-environment deployment, with conditional triggers, difference previews, and manual approval gates. +- **Environment-specific credentials**: Reuse the same project configuration across environments while injecting the real connection values during deployment. +- **Credential isolation**: Exported configuration has sensitive fields masked. Passwords and other secrets are not committed to the repository and are managed through GitHub Secrets and Variables. +- **Version rollback**: Roll back an environment to a known stable Git tag when a release does not behave as expected. + +## Use cases + +Project management and automated deployment are useful in the following scenarios: + +- **Multi-environment promotion**: Promote data integration configuration from development to test and production without recreating resources manually. +- **Team-based delivery**: Let multiple engineers work on data integration changes through Git branches, Pull Requests, review, and workflow checks. +- **Security and audit requirements**: Keep configuration changes traceable in Git while separating credentials from exported project files. +- **Recoverable production changes**: Validate configuration in lower environments and roll back to a previous tagged version if a production release has issues. diff --git a/docs/platform-ops/manage-project/setup-pipeline.md b/docs/platform-ops/manage-project/setup-pipeline.md new file mode 100644 index 0000000..dd7ef80 --- /dev/null +++ b/docs/platform-ops/manage-project/setup-pipeline.md @@ -0,0 +1,182 @@ +# Set up an automated deployment pipeline + +Before you deploy TapData projects across environments with GitHub and GitHub Actions, prepare the repositories, environments, credentials, and self-hosted runner that the pipeline requires. This guide is intended for operations and implementation teams. + +## Prerequisites + +Prepare the following resources and information before you configure the pipeline: + +| Resource | Requirement | +| --- | --- | +| **GitHub organization** | You have administrator access to at least one GitHub organization. The Worker repository and tenant repositories can be in the same organization. | +| **TapData environments** | You have at least one system integration testing (SIT) environment and one production environment. Add development, performance testing, or user acceptance testing environments as needed. | +| **Internal runner host** | You have at least one Linux server, Ubuntu 20.04 or later recommended, that can access GitHub and the TapData service ports for all target environments. Install `git`, `bash`, `jq`, and `curl`, and register the runner with the `tapdata` label. For more information, see [Adding self-hosted runners](https://docs.github.com/en/actions/how-tos/manage-runners/self-hosted-runners/add-runners). | +| **Database connection details** | You have obtained the connection addresses, usernames, and passwords for each environment from the database administrator. | +| **Deployment approver** | You have designated at least one GitHub account to approve resource imports. | + +## Plan the pipeline + +### Repository model + +Automated deployment uses two types of GitHub repositories: + +| Repository type | Purpose | Visibility | +| --- | --- | --- | +| **Worker repository** | Stores the shared deployment scripts and reusable workflows. It is maintained by the operations team and called by tenant repositories. | Internal | +| **Tenant repository** | Stores TapData project configuration exported by a team or business domain. Use one tenant repository per team or business domain. | Internal or Private | + +### Environment model + +Start with SIT and production, then add intermediate environments if your release process requires them. The following lowercase values are fixed names used by GitHub Environments and workflows: + +- `sit`: System integration testing environment. A pushed Git tag, such as `v1.2.3`, triggers deployment to this environment. +- `prod`: Production environment. Deployments are triggered manually from GitHub Actions. To use it, add this option to `workflow_dispatch.target_env.options` in the tenant repository workflow. +- `deploy`: Resource import approval gate. This environment does not store credentials. It is used to pause the workflow for manual approval before connections, tasks, or APIs are imported. + +:::tip + +If you use additional stages, create environments such as `dev` for development validation, `lpt` for performance testing, or `aat` for user acceptance testing. The official tenant template deploys to `dev` after changes are merged into `main`. If you do not use a `dev` environment, update the tenant deployment workflow so merged Pull Requests do not trigger an unconfigured environment. + +::: + +### Permissions and security + +TapData masks sensitive fields in exported project configuration. Repositories store only business configuration. Real connection values, such as database URLs, usernames, and passwords, are stored in the corresponding GitHub Environment Secrets and Variables and are injected during deployment. + +- Protect the `main` branch in each tenant repository. Disallow direct pushes, require Pull Requests, and require code review and workflow checks before merge. +- Use an independent operations approver for the `deploy` approval gate. Developers should not approve their own deployment changes. +- Store shared values, such as `GH_DEPLOY_TOKEN`, TapData URLs, and TapData access codes, as organization-level Secrets or Variables. +- Store environment-specific connection values as Environment Secrets or Variables. + +### Personal access token and naming rules + +The `GH_DEPLOY_TOKEN` is used by the runner to pull deployment scripts from the Worker repository and by TapData to push configuration and create Pull Requests in tenant repositories. + +Use the following permission model: + +- If the Worker repository and tenant repositories are in the same organization, use a fine-grained personal access token (PAT). Grant `Contents` read access to the Worker repository, and grant `Contents` and `Pull requests` read and write access to tenant repositories. +- If the repositories are in different organizations, you can use a classic PAT with the `repo` and `workflow` scopes. + +Secrets and Variables for TapData connections use the exported TapData connection name as the lookup key, with lowercase letters converted to uppercase. The worker does not replace spaces or hyphens with underscores. For compatibility with GitHub Secrets and Variables, name TapData connections with letters, numbers, and underscores before export. For example, the connection name `oracle_source` uses the prefix `ORACLE_SOURCE`. + +## Initialize the pipeline + +Follow these steps to turn the plan into an executable GitHub deployment pipeline. + +### Step 1: Create the GitHub repositories + +Use a two-repository model to separate deployment logic from business configuration. The **Worker repository** stores the shared deployment logic. Each **tenant repository** stores project configuration for one team or business domain and calls the Worker repository workflows. + +1. Create a copy of the official Worker repository, [tapdata/tapdata-cicd-worker](https://github.com/tapdata/tapdata-cicd-worker/tree/main), in your GitHub organization. You can use **Use this template** or clone the repository and push it to a new repository. + + Name the repository `tapdata-cicd-worker` and set its visibility to **Internal**. + + The Worker repository contains deployment orchestration, rollback orchestration, and scripts that call TapData APIs: + + ```text + tapdata-cicd-worker/ + ├── .github/workflows/ + │ ├── tapdata-deploy.yml # Core deployment workflow + │ └── tapdata-rollback.yml # Core rollback workflow + ├── conf/ + │ └── Task_Run_Order.json # Task startup order configuration + ├── scripts/ # TapData API scripts + └── tenant-template/.github/workflows/ + ├── tapdata-deploy.yml # Tenant deployment workflow template + └── tapdata-rollback.yml # Tenant rollback workflow template + ``` + +2. Create a tenant repository for the business team. Use a name that matches the TapData project name when possible, for example `user-center-sync`. + +3. In the tenant repository, add two lightweight workflow route files copied from `tenant-template/.github/workflows/` in the Worker repository: + + - **`tapdata-deploy.yml`**: Listens for exported configuration changes, such as changes under `*_tapdata_export/**` on the `main` branch, pushed tags, and manual dispatch events. By default, it uses the tenant repository name as the TapData project name. + - **`tapdata-rollback.yml`**: Accepts manual rollback requests by target environment and rollback tag. + + :::tip + In both copied workflow files, replace the `{WORKER_REPO}` placeholder with the Worker repository path you created, for example `your-org/tapdata-cicd-worker`. If the TapData project name differs from the tenant repository name, update the `project` input. + ::: + +4. Commit the workflow files and push them to the `main` branch of the tenant repository. + +### Step 2: Configure organization-level Secrets and Variables + +Configure shared credentials and access endpoints at the organization level so GitHub Actions can connect to TapData environments. + +1. Sign in to GitHub with an account that has repository permissions, and go to **Settings > Developer settings > Personal access tokens**. + +2. Generate a token named `tapdata-deploy`. Use an expiration of 90 days or less, and copy the token immediately after it is created. + + :::tip + If the Worker repository and tenant repositories are in the same GitHub organization, prefer a fine-grained PAT. Grant `Contents` read access to the Worker repository, and grant `Contents` and `Pull requests` read and write access to tenant repositories. + ::: + +3. Go to **Organization settings > Secrets and variables > Actions**. + +4. On the **Secrets** tab, add the following encrypted values: + + ![Set GitHub Actions Secrets 和 Variables](../../images/github_actions_secrets_variables.png) + + | Secret name | Description | + | --- | --- | + | `GH_DEPLOY_TOKEN` | The PAT created in the previous step. | + | `SIT_TAPDATA_ACCESS_CODE` | Access code for the SIT TapData instance. | + | `PROD_TAPDATA_ACCESS_CODE` | Access code for the production TapData instance. | + | `{ENV}_TAPDATA_ACCESS_CODE` | Optional. Add an access code for each intermediate environment you enable, such as `DEV_TAPDATA_ACCESS_CODE`. | + | `VAULT_ENCRYPTION_KEY` | Optional. Encrypts the generated `vault.json` credential file. | + +5. On the **Variables** tab, add the following plain-text values: + + | Variable name | Example value | + | --- | --- | + | `SIT_TAPDATA_URL` | SIT environment URL, such as `http://10.0.0.2:3030`. | + | `PROD_TAPDATA_URL` | Production environment URL. | + | `{ENV}_TAPDATA_URL` | Optional. Add a URL for each intermediate environment you enable, such as `DEV_TAPDATA_URL`. | + + :::tip + To get a TapData access code, sign in to the corresponding TapData environment as an administrator and go to **System Settings > User Management** to view the user information. In some environments, the user can also copy the access code from **Profile** in the upper-right corner. + ::: + +### Step 3: Create Environments and configure connection values + +1. In the tenant repository, go to **Settings > Environments**. +2. Create at least the `sit`, `prod`, and `deploy` Environments. The `sit` and `prod` Environments represent the SIT and production TapData environments. The `deploy` Environment is the resource import approval gate. +3. For `deploy`, configure **Required reviewers**. + + The following image shows where to configure required reviewers for the `deploy` Environment. + + ![Configure GitHub Environment required reviewers](../../images/github_environments_required_reviewers.png) + +4. If user acceptance testing or production release approval also requires environment-level review, configure **Required reviewers** on the corresponding `aat` or `prod` Environment. +5. If your release process includes development validation, performance testing, or user acceptance testing, create the corresponding Environments. +6. Configure real connection values for the SIT, production, and any other enabled environments. Use one of the following formats: + + - **URI format**: Use this format for databases such as MongoDB where the connection string includes the username and password. Store the value as a Secret named `{PREFIX}_URI`, for example `FDM_URI`. + - **Host and port format**: Use this format for databases such as PostgreSQL, Oracle, and MySQL. Store the address and username as Variables, and store the password as a Secret. Use names such as `{PREFIX}_URL`, `{PREFIX}_USER`, and `{PREFIX}_PASSWORD`. For a TapData connection named `oracle_source`, configure `ORACLE_SOURCE_URL`, `ORACLE_SOURCE_USER`, and `ORACLE_SOURCE_PASSWORD`. + + :::caution + For automated deployment with the default worker, avoid connection names that contain spaces or hyphens. For example, rename `oracle-source` to `oracle_source` in TapData and export the project again before configuring `ORACLE_SOURCE_URL`, `ORACLE_SOURCE_USER`, and `ORACLE_SOURCE_PASSWORD`. + ::: + + If multiple connections can share the same fallback values, configure `DEFAULT_URL`, `DEFAULT_USER`, and `DEFAULT_PASSWORD`. + +### Step 4: Install a self-hosted runner + +GitHub-hosted runners usually cannot access TapData services and databases in an internal network. Deploy at least one self-hosted runner on an internal Linux server. Register it at the organization level if you want multiple repositories to share it. + +1. In the GitHub organization, go to **Settings > Actions > Runners**, and click **New self-hosted runner**. +2. On the prepared Linux server, follow the GitHub setup commands to download, register, and start the runner. Add the `tapdata` label during registration. +3. Return to the **Runners** page and confirm that the runner status is `Idle`. Verify that it has the `tapdata` label and can access the TapData service ports for all target environments. + +## Validate the setup + +Before the first automated deployment, check the following items: + +- [ ] The Worker repository is **Internal** and contains the deployment workflow, rollback workflow, and core scripts. +- [ ] The `{WORKER_REPO}` placeholder in tenant repository workflows has been replaced with the real Worker repository path. +- [ ] Organization-level Secrets and Variables include `GH_DEPLOY_TOKEN`, TapData access codes, and TapData URLs for SIT and production. +- [ ] The tenant repository contains the `sit`, `prod`, and `deploy` Environments, plus any intermediate Environments that are enabled. +- [ ] Connection values are configured for SIT, production, and other enabled environments according to the naming rules. +- [ ] At least one self-hosted runner is `Idle`, has the `tapdata` label, and can access all target TapData environments. + +After the checklist is complete, continue with [Create and deploy a project](deploy-project.md) to package TapData configuration and release it to a target environment. diff --git a/sidebars.js b/sidebars.js index d5d7c58..31a24c6 100644 --- a/sidebars.js +++ b/sidebars.js @@ -404,6 +404,16 @@ const sidebars = { label: 'Platform Operations', link: {type: 'doc', id: 'platform-ops/README'}, items: [ + { + type: 'category', + label: 'Project Management', + link: {type: 'doc', id: 'platform-ops/manage-project/README'}, + items:[ + 'platform-ops/manage-project/introduction', + 'platform-ops/manage-project/setup-pipeline', + 'platform-ops/manage-project/deploy-project', + ] + }, { type: 'category', label: 'Production Deployment',