From f8845e9d31f35d5dbd9f65968dfc42bb840fe0b2 Mon Sep 17 00:00:00 2001 From: zefengzhi Date: Thu, 29 May 2025 14:06:56 +0800 Subject: [PATCH 1/3] add transporter-agent docs --- .../getting_started/config_agent.mdx | 202 ++++++++++++++++++ .../getting_started/index.mdx | 14 +- .../installing_agent/index.mdx | 38 ++++ .../linux_arm64/edb-dms-agent_debian_12.mdx | 36 ++++ .../installing_agent/linux_arm64/index.mdx | 14 ++ .../linux_x86_64/edb-dms-agent_debian_11.mdx | 36 ++++ .../linux_x86_64/edb-dms-agent_debian_12.mdx | 36 ++++ .../edb-dms-agent_other_linux_8.mdx | 37 ++++ .../edb-dms-agent_other_linux_9.mdx | 37 ++++ .../linux_x86_64/edb-dms-agent_rhel_8.mdx | 36 ++++ .../linux_x86_64/edb-dms-agent_rhel_9.mdx | 36 ++++ .../linux_x86_64/edb-dms-agent_sles_15.mdx | 46 ++++ .../linux_x86_64/edb-dms-agent_ubuntu_22.mdx | 36 ++++ .../linux_x86_64/edb-dms-agent_ubuntu_24.mdx | 36 ++++ .../installing_agent/linux_x86_64/index.mdx | 50 +++++ 15 files changed, 686 insertions(+), 4 deletions(-) create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/config_agent.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/index.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_arm64/edb-dms-agent_debian_12.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_arm64/index.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_debian_11.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_debian_12.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_other_linux_8.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_other_linux_9.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_rhel_8.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_rhel_9.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_sles_15.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_ubuntu_22.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_ubuntu_24.mdx create mode 100644 advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/index.mdx diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/config_agent.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/config_agent.mdx new file mode 100644 index 00000000000..631ae57b18d --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/config_agent.mdx @@ -0,0 +1,202 @@ +--- +title: "Configuring and running the EDB DMS Agent" +deepToC: true +--- + +## Getting credentials + +1. On the HCP Portal, select the project you are using for your migration activities. + +1. Within your project, select **Migrate** > **Credentials**. + +1. Select **Create Migration Credential**, enter a name for your credentials, and select **Create Migration Credential**. + +1. After the credentials are generated, select **Download Credential**. + !!!note + This is the only opportunity you have to download the credentials. If you don't download them at this time, you will have to revoke them and create new credentials. + +1. Unzip the downloaded zip file and copy the generated folder to the host where the agent is installed. + +## Configuring the agent + +1. Ensure your machine has Java/OpenJDK 17 or later installed. + +1. Open the EDB DMS Agent located in `/opt/cdcagent/run-cdcagent.sh` and ensure you have write permissions. + +1. Set the variables according to your environment. Uncomment the lines as you edit or review them, the DMS can only perform migrations when all necessary values are provided. See [parameters](#parameters) for further guidance. The script is reproduced below. + +```shell +#!/bin/bash -e +# run_cdcagent.sh +# +# This script provides a convenient place to specify +# environment variables used to configure the +# EDB Data Migration Service Agent. +# + +########################################## +# DMS Agent General Configuration # +########################################## + +# This ID is used to identify DMS Agent +# and is specified by the user. +#export DBCONFIG_ID= + +# Determine the run mode of the DMS Agent +# The run mode can be set to either 'reader' or 'writer' +# 'reader' mode is used to read changes from a source database +# 'writer' mode is used to write changes to a target database +#export RUN_MODE=reader + +# This is the DMS backend service used by the Agent +# Consult your system administrators +#export RW_SERVICE_HOST=https://${RW_SERVICE_HOST}/transporter + +########################################## +# DMS Agent TLS Configuration # +########################################## + +# You need to create migration credentials in EDB Postgres AI platform +# and set set the path of the credentials directory +#export CREDENTIAL_DIRECTORY_PATH=$HOME/credentials + +########################################## +# DMS Agent DB Configuration # +########################################## + +# A sample configuration to create a single postgres database connection: +#export DBCONFIG_DATABASES_0__TYPE=POSTGRES +#export DBCONFIG_DATABASES_0__HOSTNAME=localhost +#export DBCONFIG_DATABASES_0__PORT=5432 +# The CATALOG is the database name +#export DBCONFIG_DATABASES_0__CATALOG=source +#export DBCONFIG_DATABASES_0__USERNAME=postgres +# The password env can be set without specifying it here +# but the env structure looks like this +#export DBCONFIG_DATABASES_0__PASSWORD=password + +# You can increase the index to configure more than +# one database for the DMS Agent +#export DBCONFIG_DATABASES_1__TYPE=ORACLE +#export DBCONFIG_DATABASES_1__HOSTNAME=localhost +#export DBCONFIG_DATABASES_1__PORT=1521 +# The CATALOG is the database name +#export DBCONFIG_DATABASES_1__CATALOG=ORCLCDB/ORCLPDB1 +#export DBCONFIG_DATABASES_1__USERNAME=oracle +# The password env can be set without specifying it here +# but the env structure looks like this +#export DBCONFIG_DATABASES_1__PASSWORD=password + +########################################## +# Optional Parameters Below # +########################################## + +# FIPS mode can be enabled by setting the FIPS_MODE env variable to true +#export FIPS_MODE=true + +# Configure logging +# Generic loglevel +#export QUARKUS_LOG_LEVEL=DEBUG +# Loglevel for a single package +#export QUARKUS_LOG_CATEGORY__COM_ENTERPRISEDB__LEVEL=DEBUG + +cd "$(dirname "$0")" + +echo "Starting in mode: $RUN_MODE" + +case "$RUN_MODE" in + reader) + java -jar ./reader/quarkus-run.jar + ;; + writer) + java -jar ./writer/quarkus-run.jar + ;; + *) + echo "Unknown RUN_MODE: $RUN_MODE" + exit 1 + ;; +esac + +``` + +## Parameters + +### DMS Agent general configuration + +These parameters define the information the EDB DMS Agent requires to access the HCP instance and project that you are using for the migration. + +#### `DBCONFIG_ID` + +This is the name you assign to identify a source/destination. This name will later appear as a _destination_ in the **Migrate** > **Sources** or **Migrate** > **Destinations** section of the HCP Portal. + +Consider the following ID guidelines: + +- The maximum character length for the ID is 255 characters. +- You can use lowercase and uppercase characters, numbers, underscores(_) and hyphens(-) for the ID. Other special characters are not supported. +- The ID must be unique within a project. Do not use the same ID for two or more source/destinations in the same project. + +#### `RUN_MODE` +This is the run mode of the EDB DMS Agent. Set it to `reader` if you are configuring the EDB DMS Reader, or to `writer` if you are configuring the EDB DMS Writer. +*Configure and run the EDB DMS Writer only if the destination for your migration is a self-managed Postgres database. + +#### `RW_SERVICE_HOST` + +Specifies the URL of the service that will host the migration. Set `RW_SERVICE_HOST` to the domain name or host associated with the HCP ingress. + +Derive `RW_SERVICE_HOST` from the `TRANSPORTER_RW_SERVICE_DOMAIN_NAME` that was assigned by the administrators or installers of your HCP instance via the `values.yaml` file. Alternatively, derive `RW_SERVICE_HOST` from the URL you use to access the HCP portal. + +For example, if the URL for an HCP Portal is: + +`https://portal.foo-bar.enterprise.network` + +The `RW_SERVICE_HOST` will be: + +`https://transporter-rw-service.foo-bar.enterprise.network/transporter` + +where you replace `portal` with `transporter-rw-service` and add `/transporter` at the end of the endpoint. + +#### `CREDENTIAL_DIRECTORY_PATH` + +Directory path to the credential you downloaded from the HCP Portal. + +### DMS Agent database configuration + +These parameters (`DBCONFIG_DATABASES` section) define a list of source/destination database information you require for the EDB DMS Agent to be able to connect to the correct database for the migration. + +You can configure the EDB DMS Agent to migrate multiple databases. The `DBCONFIG_DATABASES_0__TYPE` section delimits the information for the first database. You can use `DBCONFIG_DATABASES_1__TYPE` to provide data for a second database. Add more sections to the EDB DMS Agent (`DBCONFIG_DATABASES_2__TYPE`, `DBCONFIG_DATABASES_3__TYPE`) by increasing the index manually. + +#### `DBCONFIG_DATABASES_0__TYPE` + +This is the source/target database type. EDB DMS reader supports `ORACLE` and `POSTGRES`. EDB DMS Writer supports `POSTGRES`. + +#### `DBCONFIG_DATABASES_0__HOSTNAME` + +The hostname of the source/target database. + +#### `DBCONFIG_DATABASES_0__PORT` + +The port of the source/target database. + +#### `DBCONFIG_DATABASES_0__CATALOG` + +The database name in the source/target database server. + +#### `DBCONFIG_DATABASES_0__USERNAME` + +The database username of the source/target database. + +#### `DBCONFIG_DATABASES_0__PASSWORD` + +The password for the database username of the source/target database. + +## Running the EDB DMS Agent + +1. Start the agent: + +```shell +cdcagent +``` + +1. Go to the HCP Portal, and verify that a source/destination with the `DBCONFIG_ID` name is displayed in **Migrate** > **Sources** or **Migrate** > **Destinations**. + +You can select this source/destination for your [migration](create_migration). diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/index.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/index.mdx index dad42f7e7b4..377432ce4bc 100644 --- a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/index.mdx +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/index.mdx @@ -7,9 +7,11 @@ navigation: - prepare_schema - installing - installing2 + - installing_agent - preparing_db - config_reader - config_writer + - config_agent - create_migration - mark_completed - apply_constraints @@ -24,15 +26,19 @@ Creating a migration from an Oracle or Postgres database to EDB Postgres AI invo 1. **[Prepare the source database schema](prepare_schema)**: In your source machine, prepare the source database by exporting it and excluding unsupported constraints. Then, import the adapted schema to the target database. -1. **[Install the EDB DMS Reader](installing)**: In your source machine, install the EDB DMS Reader from the EDB repository. +1. **[Install the EDB DMS Reader(Only applicable to HCP with versions <= 1.2.0)](installing)**: In your source machine, install the EDB DMS Reader from the EDB repository. -1. ** Optional - [Install the EDB DMS Writer](installing2)**: Perform this step only if you want to migrate data to a destination database outside of the Hybrid Control Plane. In your destination machine, install the EDB DMS Writer from the EDB repository. +1. ** Optional - [Install the EDB DMS Writer(Only applicable to HCP with versions <= 1.2.0)](installing2)**: Perform this step only if you want to migrate data to a destination database outside of the Hybrid Control Plane. In your destination machine, install the EDB DMS Writer from the EDB repository. + +1. **[Install the EDB DMS Agent(For HCP with versions > 1.2.0)](installing_agent)**: In your source or destination machine, install the EDB DMS Agent from the EDB repository. DMS Agent is the package that will install the DMS Reader and DMS Writer together on your machine. 1. **[Prepare your Oracle or Postgres database](preparing_db)**: In your source/destination machine, prepare the database by altering settings and creating users that are required for the migration. Ensure your database can accept SSL connections. -1. **[Configure the EDB DMS Reader](config_reader)**: In the Hybrid Control Plane portal, download dedicated migration credentials. In your source machine, configure the EDB DMS Reader by exporting environment variables that allow the Reader to connect to the source. Execute the Reader. +1. **[Configure the EDB DMS Reader(Only applicable to HCP with versions <= 1.2.0)](config_reader)**: In the Hybrid Control Plane portal, download dedicated migration credentials. In your source machine, configure the EDB DMS Reader by exporting environment variables that allow the Reader to connect to the source. Execute the Reader. + +1. **Optional - [Configure the EDB DMS Writer(Only applicable to HCP with versions <= 1.2.0)](config_writer)**: Perform this step only if you want to migrate data to a destination database outside of the Hybrid Control Plane. In your destination machine, configure the EDB DMS Writer by exporting environment variables that allow the Writer to connect to the destination. Execute the Writer. -1. **Optional - [Configure the EDB DMS Writer](config_writer)**: Perform this step only if you want to migrate data to a destination database outside of the Hybrid Control Plane. In your destination machine, configure the EDB DMS Writer by exporting environment variables that allow the Writer to connect to the destination. Execute the Writer. +1. **[Configure the EDB DMS Agent(For HCP with versions > 1.2.0)](config_agent)**: In the Hybrid Control Plane portal, download dedicated migration credentials. In your source or destination machine, configure the EDB DMS Agent by exporting environment variables that allow the Agent to connect to the source and destination databases. Execute the Agent. 1. **[Create a new migration](create_migration)**: In the EDB Postgres AI Console, create a new migration by selecting the source generated by the Reader in the Console, and selecting the target database you created for this purpose. diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/index.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/index.mdx new file mode 100644 index 00000000000..1e2b0c61956 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/index.mdx @@ -0,0 +1,38 @@ +--- +navTitle: Installing Agent +title: Installing EDB Data Migration Service Agent on Linux +indexCards: none + +navigation: + - linux_x86_64 +--- + +Select a link to access the applicable installation instructions: + +## Linux [x86-64 (amd64)](linux_x86_64) + +### Red Hat Enterprise Linux (RHEL) and derivatives + +- [RHEL 9](linux_x86_64/edb-dms-agent_rhel_9), [RHEL 8](linux_x86_64/edb-dms-agent_rhel_8) + +- [Oracle Linux (OL) 9](linux_x86_64/edb-dms-agent_rhel_9), [Oracle Linux (OL) 8](linux_x86_64/edb-dms-agent_rhel_8) + +- [Rocky Linux 9](linux_x86_64/edb-dms-agent_other_linux_9), [Rocky Linux 8](linux_x86_64/edb-dms-agent_other_linux_8) + +- [AlmaLinux 9](linux_x86_64/edb-dms-agent_other_linux_9), [AlmaLinux 8](linux_x86_64/edb-dms-agent_other_linux_8) + +### SUSE Linux Enterprise (SLES) + +- [SLES 15](linux_x86_64/edb-dms-agent_sles_15) + +### Debian and derivatives + +- [Ubuntu 24.04](linux_x86_64/edb-dms-agent_ubuntu_24), [Ubuntu 22.04](linux_x86_64/edb-dms-agent_ubuntu_22) + +- [Debian 12](linux_x86_64/edb-dms-agent_debian_12), [Debian 11](linux_x86_64/edb-dms-agent_debian_11) + +## Linux [AArch64 (ARM64)](linux_arm64) + +### Debian and derivatives + +- [Debian 12](linux_arm64/edb-dms-agent_debian_12) diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_arm64/edb-dms-agent_debian_12.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_arm64/edb-dms-agent_debian_12.mdx new file mode 100644 index 00000000000..95c1e8afd55 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_arm64/edb-dms-agent_debian_12.mdx @@ -0,0 +1,36 @@ +--- +navTitle: Debian 12 +title: Installing EDB Data Migration Service Agent on Debian 12 arm64 +--- + +## Prerequisites + +Before you begin the installation process: + +- Set up the EDB repository. + + Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step. + + To determine if your repository exists, enter this command: + + `apt-cache search enterprisedb` + + If no output is generated, the repository isn't installed. + + To set up the EDB repository: + + 1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads). + + 1. Select the button that provides access to the EDB repository. + + 1. Select the platform and software that you want to download. + + 1. Follow the instructions for setting up the EDB repository. + +## Install the package + +Install the EDB DMS Agent (packaged as `cdcagent`): + +```shell +sudo apt-get install cdcagent +``` diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_arm64/index.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_arm64/index.mdx new file mode 100644 index 00000000000..cc67b1dc645 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_arm64/index.mdx @@ -0,0 +1,14 @@ +--- +title: "Installing EDB Data Migration Service Agent on Linux AArch64 (ARM64)" +navTitle: "On Linux ARM64" +indexCards: none + +navigation: + - edb-dms-agent_debian_12 +--- + +Operating system-specific install instructions are described in the corresponding documentation: + +### Debian and derivatives + +- [Debian 12](edb-dms-agent_debian_12) diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_debian_11.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_debian_11.mdx new file mode 100644 index 00000000000..f15a0719d53 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_debian_11.mdx @@ -0,0 +1,36 @@ +--- +navTitle: Debian 11 +title: Installing EDB Data Migration Service Agent on Debian 11 x86_64 +--- + +## Prerequisites + +Before you begin the installation process: + +- Set up the EDB repository. + + Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step. + + To determine if your repository exists, enter this command: + + `apt-cache search enterprisedb` + + If no output is generated, the repository isn't installed. + + To set up the EDB repository: + + 1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads). + + 1. Select the button that provides access to the EDB repository. + + 1. Select the platform and software that you want to download. + + 1. Follow the instructions for setting up the EDB repository. + +## Install the package + +Install the EDB DMS Agent (packaged as `cdcagent`): + +```shell +sudo apt-get install cdcagent +``` diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_debian_12.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_debian_12.mdx new file mode 100644 index 00000000000..66725abbadb --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_debian_12.mdx @@ -0,0 +1,36 @@ +--- +navTitle: Debian 12 +title: Installing EDB Data Migration Service Agent on Debian 12 x86_64 +--- + +## Prerequisites + +Before you begin the installation process: + +- Set up the EDB repository. + + Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step. + + To determine if your repository exists, enter this command: + + `apt-cache search enterprisedb` + + If no output is generated, the repository isn't installed. + + To set up the EDB repository: + + 1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads). + + 1. Select the button that provides access to the EDB repository. + + 1. Select the platform and software that you want to download. + + 1. Follow the instructions for setting up the EDB repository. + +## Install the package + +Install the EDB DMS Agent (packaged as `cdcagent`): + +```shell +sudo apt-get install cdcagent +``` diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_other_linux_8.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_other_linux_8.mdx new file mode 100644 index 00000000000..77185e97f74 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_other_linux_8.mdx @@ -0,0 +1,37 @@ +--- +navTitle: AlmaLinux 8 or Rocky Linux 8 +title: Installing EDB Data Migration Service Agent on AlmaLinux 8 or Rocky + Linux 8 x86_64 +--- + +## Prerequisites + +Before you begin the installation process: + +- Set up the EDB repository. + + Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step. + + To determine if your repository exists, enter this command: + + `dnf repolist | grep enterprisedb` + + If no output is generated, the repository isn't installed. + + To set up the EDB repository: + + 1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads). + + 1. Select the button that provides access to the EDB repository. + + 1. Select the platform and software that you want to download. + + 1. Follow the instructions for setting up the EDB repository. + +## Install the package + +Install the EDB DMS Agent (packaged as `cdcagent`): + +```shell +sudo dnf install cdcagent +``` diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_other_linux_9.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_other_linux_9.mdx new file mode 100644 index 00000000000..4e5036d6cb3 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_other_linux_9.mdx @@ -0,0 +1,37 @@ +--- +navTitle: AlmaLinux 9 or Rocky Linux 9 +title: Installing EDB Data Migration Service Agent on AlmaLinux 9 or Rocky + Linux 9 x86_64 +--- + +## Prerequisites + +Before you begin the installation process: + +- Set up the EDB repository. + + Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step. + + To determine if your repository exists, enter this command: + + `dnf repolist | grep enterprisedb` + + If no output is generated, the repository isn't installed. + + To set up the EDB repository: + + 1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads). + + 1. Select the button that provides access to the EDB repository. + + 1. Select the platform and software that you want to download. + + 1. Follow the instructions for setting up the EDB repository. + +## Install the package + +Install the EDB DMS Agent (packaged as `cdcagent`): + +```shell +sudo dnf install cdcagent +``` diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_rhel_8.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_rhel_8.mdx new file mode 100644 index 00000000000..d47e6df6bab --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_rhel_8.mdx @@ -0,0 +1,36 @@ +--- +navTitle: RHEL 8 or OL 8 +title: Installing EDB Data Migration Service Agent on RHEL 8 or OL 8 x86_64 +--- + +## Prerequisites + +Before you begin the installation process: + +- Set up the EDB repository. + + Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step. + + To determine if your repository exists, enter this command: + + `dnf repolist | grep enterprisedb` + + If no output is generated, the repository isn't installed. + + To set up the EDB repository: + + 1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads). + + 1. Select the button that provides access to the EDB repository. + + 1. Select the platform and software that you want to download. + + 1. Follow the instructions for setting up the EDB repository. + +## Install the package + +Install the EDB DMS Agent (packaged as `cdcagent`): + +```shell +sudo dnf install cdcagent +``` diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_rhel_9.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_rhel_9.mdx new file mode 100644 index 00000000000..91efa8ea3c6 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_rhel_9.mdx @@ -0,0 +1,36 @@ +--- +navTitle: RHEL 9 or OL 9 +title: Installing EDB Data Migration Service Agent on RHEL 9 or OL 9 x86_64 +--- + +## Prerequisites + +Before you begin the installation process: + +- Set up the EDB repository. + + Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step. + + To determine if your repository exists, enter this command: + + `dnf repolist | grep enterprisedb` + + If no output is generated, the repository isn't installed. + + To set up the EDB repository: + + 1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads). + + 1. Select the button that provides access to the EDB repository. + + 1. Select the platform and software that you want to download. + + 1. Follow the instructions for setting up the EDB repository. + +## Install the package + +Install the EDB DMS Agent (packaged as `cdcagent`): + +```shell +sudo dnf install cdcagent +``` diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_sles_15.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_sles_15.mdx new file mode 100644 index 00000000000..1f7548bd0a3 --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_sles_15.mdx @@ -0,0 +1,46 @@ +--- +navTitle: SLES 15 +title: Installing EDB Data Migration Service Agent on SLES 15 x86_64 +--- + +## Prerequisites + +Before you begin the installation process: + +- Set up the EDB repository. + + Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step. + + To determine if your repository exists, enter this command: + + `zypper lr -E | grep enterprisedb` + + If no output is generated, the repository isn't installed. + + To set up the EDB repository: + + 1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads). + + 1. Select the button that provides access to the EDB repository. + + 1. Select the platform and software that you want to download. + + 1. Follow the instructions for setting up the EDB repository. + +- Activate the required SUSE module: + ```shell + sudo SUSEConnect -p PackageHub/15.4/x86_64 + + ``` +- Refresh the metadata: + ```shell + sudo zypper refresh + ``` + +## Install the package + +Install the EDB DMS Agent (packaged as `cdcagent`): + +```shell +sudo zypper install cdcagent +``` diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_ubuntu_22.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_ubuntu_22.mdx new file mode 100644 index 00000000000..fd2247af4fb --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_ubuntu_22.mdx @@ -0,0 +1,36 @@ +--- +navTitle: Ubuntu 22.04 +title: Installing EDB Data Migration Service Agent on Ubuntu 22.04 x86_64 +--- + +## Prerequisites + +Before you begin the installation process: + +- Set up the EDB repository. + + Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step. + + To determine if your repository exists, enter this command: + + `apt-cache search enterprisedb` + + If no output is generated, the repository isn't installed. + + To set up the EDB repository: + + 1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads). + + 1. Select the button that provides access to the EDB repository. + + 1. Select the platform and software that you want to download. + + 1. Follow the instructions for setting up the EDB repository. + +## Install the package + +Install the EDB DMS Agent (packaged as `cdcagent`): + +```shell +sudo apt-get install cdcagent +``` diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_ubuntu_24.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_ubuntu_24.mdx new file mode 100644 index 00000000000..3a72d61c2ed --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/edb-dms-agent_ubuntu_24.mdx @@ -0,0 +1,36 @@ +--- +navTitle: Ubuntu 24.04 +title: Installing EDB Data Migration Service Agent on Ubuntu 24.04 x86_64 +--- + +## Prerequisites + +Before you begin the installation process: + +- Set up the EDB repository. + + Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step. + + To determine if your repository exists, enter this command: + + `apt-cache search enterprisedb` + + If no output is generated, the repository isn't installed. + + To set up the EDB repository: + + 1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads). + + 1. Select the button that provides access to the EDB repository. + + 1. Select the platform and software that you want to download. + + 1. Follow the instructions for setting up the EDB repository. + +## Install the package + +Install the EDB DMS Agent (packaged as `cdcagent`): + +```shell +sudo apt-get install cdcagent +``` diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/index.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/index.mdx new file mode 100644 index 00000000000..fda3f9c237c --- /dev/null +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/linux_x86_64/index.mdx @@ -0,0 +1,50 @@ +--- +title: "Installing EDB Data Migration Service Agent on Linux x86 (amd64)" +navTitle: "On Linux x86" +indexCards: none + +navigation: + - edb-dms-agent_rhel_9 + - edb-dms-agent_rhel_8 + - edb-dms-agent_other_linux_9 + - edb-dms-agent_other_linux_8 + - edb-dms-agent_sles_15 + - edb-dms-agent_ubuntu_24 + - edb-dms-agent_ubuntu_22 + - edb-dms-agent_debian_12 + - edb-dms-agent_debian_11 +--- + +Operating system-specific install instructions are described in the corresponding documentation: + +### Red Hat Enterprise Linux (RHEL) and derivatives + +- [RHEL 9](edb-dms-agent_rhel_9) + +- [RHEL 8](edb-dms-agent_rhel_8) + +- [Oracle Linux (OL) 9](edb-dms-agent_rhel_9) + +- [Oracle Linux (OL) 8](edb-dms-agent_rhel_8) + +- [Rocky Linux 9](edb-dms-agent_other_linux_9) + +- [Rocky Linux 8](edb-dms-agent_other_linux_8) + +- [AlmaLinux 9](edb-dms-agent_other_linux_9) + +- [AlmaLinux 8](edb-dms-agent_other_linux_8) + +### SUSE Linux Enterprise (SLES) + +- [SLES 15](edb-dms-agent_sles_15) + +### Debian and derivatives + +- [Ubuntu 24.04](edb-dms-agent_ubuntu_24) + +- [Ubuntu 22.04](edb-dms-agent_ubuntu_22) + +- [Debian 12](edb-dms-agent_debian_12) + +- [Debian 11](edb-dms-agent_debian_11) From d4878f5a1ac42a9f7511d9be8233aa2c084afcab Mon Sep 17 00:00:00 2001 From: Yidian Sun Date: Mon, 25 Aug 2025 17:33:50 +0800 Subject: [PATCH 2/3] docs(ET-2898): add null character limitation for oracle and tsvector workaround for pg --- .../data-migration-service/limitations.mdx | 34 +++++++++++++++++-- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/limitations.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/limitations.mdx index 98d0393d861..e624d433861 100644 --- a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/limitations.mdx +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/limitations.mdx @@ -25,7 +25,7 @@ These are the known issues, limitations, and notes for: A limited number of Oracle data types and features aren't supported by EDB Data Migration Service (EDB DMS). -See the [Debezium documentation](https://debezium.io/documentation/reference/2.2/connectors/oracle.html#oracle-data-type-mappings) for detailed comments on supported data types. +See the [Debezium documentation](https://debezium.io/documentation/reference/2.2/connectors/oracle.html#oracle-data-type-mappings) for detailed comments on supported data types. Unsupported Oracle data types include: @@ -43,6 +43,8 @@ EDB DMS supports replicating Oracle tables that contain BLOB, CLOB, or NCLOB col `BINARY_FLOAT` and `BINARY_DOUBLE` types in Oracle that might contain `Nan`, `+INF`, and `-INF` values are not supported by EDB DMS. +For text field types like `VARCHAR` and `CLOB` in Oracle that might contain null character(0x00) values are not supported by EDB DMS. + ### Oracle 21c databases The Data Migration Service Reader and Writer support migrating Oracle 21c databases. However, Migration Portal doesn't. This means that to migrate 21c databases, you must manually perform the schema migration. @@ -63,11 +65,37 @@ The EDB DMS doesn't support migrating tables with columns that have user-defined - XML - POINT - LTREE +- TSVECTOR + +### TSVECTOR type migration workaround + +This workaround based on the following example `messages` table: +```sql +CREATE TABLE messages ( + title text, + body text, + tsv tsvector +); +``` +The first option is creating a trigger to update the `tsvector` field automatically: +```sql +CREATE TRIGGER tsvectorupdate BEFORE INSERT OR UPDATE +ON messages FOR EACH ROW EXECUTE FUNCTION +tsvector_update_trigger(tsv, 'pg_catalog.english', title, body); +``` +This CREATE TRIGGER statement should execute before the migration created and after the target table created. + +To make sure the trigger to compute the tsvector field for every row. Since this trigger will not compute the + +tsvector field for existing rows, you can run the following UPDATE statement after the migration is complete: +```sql +UPDATE messages SET tsv = to_tsvector('english', CONCAT(title, ' ', body)) WHERE tsv IS NULL; +``` ### Incorrect failure messages -The `postgresConfigValidation.sh` script referenced in [Preparing Postgres database sources](/edb-postgres-ai/hybrid-manager/using_hybrid_manager/migration/migration_guides/self_managed_pg_to_hm_managed/preparing_source_pg_db/) incorrectly reports failures for the `max_wal_size` database parameter check and for the replication slot check. +The `postgresConfigValidation.sh` script referenced in [Preparing Postgres database sources](/edb-postgres-ai/hybrid-manager/using_hybrid_manager/migration/migration_guides/self_managed_pg_to_hm_managed/preparing_source_pg_db/) incorrectly reports failures for the `max_wal_size` database parameter check and for the replication slot check. -The script reports a failure for the `max_wal_size` check, if the parameter is set to a value lower than 8 GB. Although this is a recommended setting for many production workloads, setting the parameter to a lower value doesn't prevent a migration from being performed. +The script reports a failure for the `max_wal_size` check, if the parameter is set to a value lower than 8 GB. Although this is a recommended setting for many production workloads, setting the parameter to a lower value doesn't prevent a migration from being performed. In addition, the Postgres configuration validation script also performs a check to see if a replication slot with a slot name matching the migration user/role was created. This is an outdated check that is no longer needed because the EDB DMS Reader automatically creates and manages the required replication slot. You can ignore the failure message related to the replication slot. The check will be removed in a future version of the `postgresConfigValidation.sh` script. From 21398c227efefffbdfc69e79bb060c4e859bff5d Mon Sep 17 00:00:00 2001 From: lingyangma Date: Mon, 25 Aug 2025 18:32:05 +0800 Subject: [PATCH 3/3] improve oracle preparing doc --- .../getting_started/index.mdx | 2 +- .../getting_started/installing/index.mdx | 4 + .../getting_started/installing2/index.mdx | 3 + .../installing_agent/index.mdx | 3 + .../preparing_oracle_source_databases.mdx | 340 ++++++++++++------ .../data-migration-service/index.mdx | 3 +- .../data-migration-service/terminology.mdx | 32 +- 7 files changed, 264 insertions(+), 123 deletions(-) diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/index.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/index.mdx index dfd33b2edcc..a540ae0b4e5 100644 --- a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/index.mdx +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/index.mdx @@ -4,9 +4,9 @@ navTitle: Overview navigation: - create_database - prepare_schema + - installing_agent - installing - installing2 - - installing_agent - preparing_db - config_reader - config_writer diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing/index.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing/index.mdx index fe0600fa4d9..372e3b203db 100644 --- a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing/index.mdx +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing/index.mdx @@ -7,6 +7,10 @@ navigation: - linux_x86_64 --- + +!!!note + EDB DMS Reader is only compatible with Hybrid Manager versions 1.2 and below. For Hybrid Manager 1.3 and above, please use EDB DMS Agent. + Select a link to access the applicable installation instructions: ## Linux [x86-64 (amd64)](linux_x86_64) diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing2/index.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing2/index.mdx index 084c7e85bf9..39819123a69 100644 --- a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing2/index.mdx +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing2/index.mdx @@ -7,6 +7,9 @@ navigation: - linux_x86_64 --- +!!!note + EDB DMS Writer is only compatible with Hybrid Manager versions 1.2 and below. For Hybrid Manager 1.3 and above, please use EDB DMS Agent. + Select a link to access the applicable installation instructions: ## Linux [x86-64 (amd64)](linux_x86_64) diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/index.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/index.mdx index 1e2b0c61956..7c2fb9c8a50 100644 --- a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/index.mdx +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing_agent/index.mdx @@ -7,6 +7,9 @@ navigation: - linux_x86_64 --- +!!!note + EDB DMS Agent is only compatible with Hybrid Manager versions 1.3 and above. For Hybrid Manager 1.2 and below, please use EDB DMS Reader and EDB DMS Writer. + Select a link to access the applicable installation instructions: ## Linux [x86-64 (amd64)](linux_x86_64) diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/preparing_db/preparing_oracle_source_databases.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/preparing_db/preparing_oracle_source_databases.mdx index 3341cfd3ff9..cb81de9f9b8 100644 --- a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/preparing_db/preparing_oracle_source_databases.mdx +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/preparing_db/preparing_oracle_source_databases.mdx @@ -6,46 +6,38 @@ navTitle: Preparing Oracle source +## Oracle configuration + Configuring Oracle for EDB Data Migration Services (EDB DMS) requires `sysdba` privileges. Configure an Oracle source database to: - Enable archive log mode. -- Enable supplemental logging for the database and table columns of interest. -- Ensure adequate redo log space is available. -- Create a user with limited privileges to carry out the data migration. +- Enable supplemental logging for the database and tables of interest. +- Prepare tablespaces for migration user +- Create a migration user and grant necessary privileges to carry out the data migration. +- Grant SELECT on source tables. +- (Optional) Ensure adequate redo log space is available for better performance. +- Validate configuration. + +### Step1: Enable archive log mode -Execute SQL statements with `sqlplus` or a similar client. +Debezium captures database changes by reading the database's redo logs. To ensure a complete and reliable history of all transactions without risk of data loss, Oracle must be configured in Archive Log Mode. This process saves a copy of every redo log before it is reused, providing Debezium with an unbroken stream of changes. -This command prompts you for the password for ``: +Log in to oracle **CDB** as sysdba: ```shell -sqlplus @:/ as sysdba +sqlplus /@:/ as sysdba ``` Where: - - `` is the Oracle DB hostname. - - `` is the Oracle DB port. - - `` is the Oracle System ID for the DB or CDB/PDB combination. - - `` is an Oracle DB username with sysdba privileges. - -## Oracle configuration - -To perform Oracle configuration: - -1. [Enable archive log mode](#enable-archive-log-mode). -1. [Enable database supplemental logging](#enable-database-supplemental-logging). -1. [Enable supplemental logging for table columns](#enable-supplemental-logging-for-table-columns). -1. [Verify redo logs for adequate count and size](#verify-redo-logs-for-adequate-count-and-size). -1. [Create a user with limited privileges for data migration](#create-a-user-with-limited-privileges-for-data-migration). -1. [Grant `SELECT` on source tables](#grant-select-on-source-tables). -1. [Validate configuration](#validate-configuration). - -### Enable archive log mode +- `` is an oracle user that can log in to the cdb as sysdba. +- `` is the password of dba_user. +- `` is the oracle db hostname. +- `` is the oracle db port. +- `` is the oracle CDB name. -Oracle databases can operate in `ARCHIVELOG` or `NOARCHIVELOG` mode. In `ARCHIVELOG` mode, filled redo logs are archived rather than put back into log rotation to be overwritten. This mode is needed for the change data capture (CDC) process to use LogMiner and produce a complete history of changes after an initial consistent snapshot. - -To see the database mode: +Run the following sql to check whether archive mode is enabled: ```sql archive log list; @@ -54,143 +46,178 @@ archive log list; The returned content indicates the database mode: ```sql -Database log mode Archive Mode -...or +-- below output indicates archive log mode is enabled +Database log mode Archive Mode -- (or "Archive Log Mode" in new versions of oracle) +-- or below output indicates archive log mode is not enabled Database log mode No Archive Mode ``` -If `ARCHIVELOG` mode is enabled, confirm with your DBA that the size of your recovery file destination is appropriate for your workload. +If your archive mode is already enabled, you can skip this step. Otherwise, log out from previous sql session first, then run the following command: -When enabling archive log mode, you need to enable a fast recovery area. For more information on enabling an Oracle fast recovery area, see [Enabling the Fast Recovery Area](https://docs.oracle.com/en/database/oracle/oracle-database/19/bradv/configuring-rman-client-basic.html#GUID-233338E2-3EE6-4248-A2B6-16A7899DB14F) in the Oracle documentation. +```shell +ORACLE_SID= sqlplus /nolog +``` -To enable archive logging: +Where -```sql -ORACLE_SID= sqlplus /nolog +- `` is the oracle CDB name. + +After entering sql session, run the following SQLs one by one: -CONNECT /_PWD AS SYSDBA +```sql +CONNECT / AS sysdba; +-- Set the location and size for the Fast Recovery Area (FRA) alter system set db_recovery_file_dest_size = ; alter system set db_recovery_file_dest = '' scope=spfile; +-- Restart the database in the correct state to enable archiving shutdown immediate startup mount +-- Enable archive log mode and open the database for use alter database archivelog; alter database open; +-- Verify the change was successful archive log list; exit; ``` -Where: - - `` is the Oracle DB system ID. - - `` is the name of a user with sysdba privileges. - - `` is the password for ``. - - `` is the size allowed for the recovery behavior, for example, `100G` for 100 gigabytes. - - `` is the file system path for an Oracle fast recovery area. This path can be a directory, file system, or Oracle Automatic Storage Management. Consult your DBA for guidance. +Where -The `archive log list` output shows the database is now in archive log mode. +- `` is an oracle user that can log in to the cdb as sysdba. +- `` is the password of dba_user. +- `` is the directory path of your FRA, where oracle redo logs will locate. +- `` is the space allocated to FRA, for example, 100G means 100 gigabytes. -### Enable database supplemental logging +!!!note + db_recovery_file_dest and db_recovery_file_dest_size configure Oracle's **Fast Recovery Area (FRA)**, a vital storage area for the archived logs that Debezium needs. Correctly setting them is essential for database stability, therefore, **always consult your DBA for production settings.** + +### Step2: Enable supplemental logging for the database and tables of interest Supplemental logging refers to the capture of additional information in Oracle redo logs, such as "before" state. This extra redo log information is needed for some log-based applications, such as EDB DMS, to capture change events. See [Supplemental Logging](https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/oracle-logminer-utility.html#GUID-D857AF96-AC24-4CA1-B620-8EA3DF30D72E) in the Oracle documentation for more information. -You can enable supplemental logging at the database and table level. The following command enables minimal supplemental logging required for LogMiner to function at the database level: +You need to enable supplemental logging at the database level as well as for those tables you want to migrate, they're crucial for debezium to run data migrations: +* Database level supplemental logging is the foundation. It ensures every CDC message has a Tracking Number (the Primary Key), so you can uniquely identify which order was changed. +* Table level Logging is the detail. It ensures the CDC message includes a full copy of the original order form (the "before" image of all columns), not just the item that was updated. + +First Log in to oracle **CDB** as sysdba: + +```shell +sqlplus /@:/ as sysdba +``` + +Where: + +- `` is an oracle user that can log in to the cdb as sysdba. +- `` is the password of dba_user. +- `` is the oracle db hostname. +- `` is the oracle db port. +- `` is the oracle CDB name. + +Run the following SQL to enable supplemental logging at the entire database level: ```sql ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; ``` -### Enable supplemental logging for table columns - -For every table you want to migrate, you must enable supplemental logging. To do -so for all columns in a table, apply the following statement: +Then switch to the ***PDB*** you want to migrate: ```sql -ALTER TABLE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; +ALTER SESSION SET CONTAINER = ; ``` -Where `
` is the identifier for the table to migrate. +Where: -Use `ALTER` with all table columns you want to migrate. +- `` is the PDB you want to migrate -### Verify redo logs for adequate count and size +And enable supplemental logging for all the tables you want to migrate: -The migration process involves two phases. The first is a consistent snapshot. The second is continuous streaming of database changes. This stream of database changes is powered by LogMiner and the Oracle DB redo logs. +```sql +ALTER TABLE . ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; +``` -Database changes have a limited lifetime on the redo logs before the change is no longer present in the log history. This lifetime depends on the size of the redo logs, the number of redo logs, and the change throughput to the database. Also, undersized logs cause frequent log switching and affect migration performance. +Where: -To examine the state of the database redo logs: +- `` is the schema to which your table belongs +- `` is the name of a table you want to migrate -```sql -SELECT GROUP#, TYPE, MEMBER FROM V_$LOGFILE; +!!!note + In oracle, a schema is identical to a user, so remember to set schema name correctly when running the above sql, otherwise data migration might encounter errors. - GROUP# TYPE MEMBER ----------- ------- -------------------------------------------------- - 1 ONLINE /opt/oracle/oradata/ORCLCDB/redo03.log - 2 ONLINE /opt/oracle/oradata/ORCLCDB/redo01.log - 3 ONLINE /opt/oracle/oradata/ORCLCDB/redo04.log +### Step3: Prepared tablespaces for migration user -SELECT GROUP#, ARCHIVED, BYTES/1024/1024 MB, STATUS FROM V_$LOG; +DMS and the underlying library it relies on(Debezium) requires a ***common user*** to carry out data migration. But before creating such a migration user, we need to first create dedicated tablespaces for it. Two tablespaces are needed: +* A CDB level tablespace for debezium's internal log mining and management processes that operate at the container level. +* A PDB level tablespace for operations related to specific tables you're capturing within that PDB. - GROUP# ARC MB STATUS ----------- --- --- ---------------- - 1 YES 2000 INACTIVE - 3 YES 2000 INACTIVE - 3 NO 2000 CURRENT +First Log in to oracle **CDB** as sysdba: + +```shell +sqlplus /@:/ as sysdba ``` -This example uses three log groups of size 2000MB. Each group has one file member. This might be too -small for many production databases. You can safely adjust the redo logs with synchronous commands such as the following: +Where: + +- `` is an oracle user that can log in to the cdb as sysdba. +- `` is the password of dba_user. +- `` is the oracle db hostname. +- `` is the oracle db port. +- `` is the oracle CDB name. + +Run the following SQL to create a ***CDB*** tablespace: ```sql - ALTER DATABASE ADD LOGFILE GROUP 4 ('/opt/oracle/oradata/ORCLCDB/redo04.log') SIZE 8G; - ALTER DATABASE ADD LOGFILE GROUP 5 ('/opt/oracle/oradata/ORCLCDB/redo05.log') SIZE 8G; - ALTER DATABASE ADD LOGFILE GROUP 6 ('/opt/oracle/oradata/ORCLCDB/redo06.log') SIZE 8G; - ALTER DATABASE ADD LOGFILE GROUP 7 ('/opt/oracle/oradata/ORCLCDB/redo07.log') SIZE 8G; - ALTER SYSTEM ARCHIVE LOG CURRENT; - ALTER SYSTEM CHECKPOINT; - ALTER SYSTEM ARCHIVE LOG CURRENT; - ALTER SYSTEM CHECKPOINT; - ALTER SYSTEM ARCHIVE LOG CURRENT; - ALTER SYSTEM CHECKPOINT; - ALTER DATABASE DROP LOGFILE GROUP 1; - ALTER DATABASE DROP LOGFILE GROUP 2; - ALTER DATABASE DROP LOGFILE GROUP 3; +CREATE TABLESPACE DATAFILE '.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; ``` -These commands result in four new 8GB log groups. Each group has a single log file. +Where: -Consult your DBA for appropriate production sizing. +- `` is the name of the CDB tablespace, any valid name is ok +- `.dbf` is the file path of this tablespace. Consult your DBA for the correct value. -### Create a user with limited privileges for data migration +Then switch to the ***PDB*** you want to migrate: -#### Tablespace preparation +```sql +ALTER SESSION SET CONTAINER = ; +``` -Provide a database user with adequate roles to carry out the CDC process. +Where: -Then, we recommend creating a tablespace for the CDC user. For container databases, you need to create a pluggable database as well. +- `` is the PDB you want to migrate -This example creates a tablespace and datafiles for CDC migration. Your database settings might vary, but a common configuration with `SMALLFILE` tablespaces and an 8kB database block size results in a maximum of 32GB of storage avaiable per `MAXSIZE` tablespace datafile. Therefore, you might need to add multiple `AUTOEXTEND` datafiles when this limit might be exceeded. +And create a tablespace for PDB: ```sql --- Create the tablespace, or in the case of a CDB/PDB, create the CDB tablespace -CREATE TABLESPACE DATAFILE '/opt/oracle/oradata/ORCLCDB/logminer_tbs.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; - --- For CDB/PDB deployments we must specify at least one tablespace datafile for the PDB -CREATE TABLESPACE DATAFILE '/opt/oracle/oradata/ORCLCDB/ORCPDB1/logminer_tbs_1.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; --- Additional data files can be added with as follows -ALTER TABLESPACE ADD DATAFILE '/opt/oracle/oradata/ORCLCDB/ORCLPDB1/logminer_tbs_2.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; +CREATE TABLESPACE DATAFILE '.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; +-- continue adding more datafiles if necessary +-- CREATE TABLESPACE DATAFILE '.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; ``` Where: - - `` is the tablespace name for the CDC migration user to use. -#### User creation and access grants +- `` is the name of the PDB tablespace, for convenience we use the same name as the CDB tablespace. +- `.dbf` is the file path of this tablespace. Consult your DBA for the correct value. -With the tablespace files in place, you can create a user with appropriate access grants for CDC migration. +If you're using a non-CDB database, the PDB realted tablespace setup can be omitted. -For a CDB/PDB database setup, note the tablespace default and quota: +### Step4: Create a migration user and grant necessary privileges to carry out the data migration + +As previous step mentioned, DMS requires a common user. First log in to ***CDB*** as sysdba: + +```shell +sqlplus /@:/ as sysdba +``` + +Where: + +- `` is an oracle user that can log in to the cdb as sysdba. +- `` is the password of dba_user. +- `` is the oracle db hostname. +- `` is the oracle db port. +- `` is the oracle CDB name. + +Run the following SQLs one by one to create a migration user and grant necessary permissions: ```sql - CREATE USER IDENTIFIED BY +CREATE USER IDENTIFIED BY DEFAULT TABLESPACE QUOTA UNLIMITED ON CONTAINER=ALL; @@ -217,7 +244,19 @@ For a CDB/PDB database setup, note the tablespace default and quota: GRANT SELECT ON V_$TRANSACTION TO CONTAINER=ALL; ``` -For a non-CDB database: +Where: + +- `` is the name of the common user you want to create for DMS to use. +- `` password of migration user. +- `` is the name of the CDB tablespace you created in previous step. Remember for convenience we use the same tablespace name for CDB and PDB. + +If in step3, you used different names for CDB and PDB, then apart from the above SQLs, you'll have to log in to the ***PDB*** and run the following SQL: + +```sql +ALTER USER DEFAULT TABLESPACE +``` + +If you're using a non-CDB database, run following SQL instead: ```sql CREATE USER IDENTIFIED BY @@ -245,37 +284,108 @@ For a non-CDB database: GRANT SELECT ON V_$TRANSACTION TO ; ``` +### Step5: Grant SELECT on source tables + +First Log in to oracle **PDB** as sysdba: + +```shell +sqlplus /@:/ as sysdba +``` + Where: - - `` is the name of the user to create for CDC migration table access. - - `` is the password for the migration user. - - `` is the tablespace for ``. -### Grant `SELECT` on source tables +- `` is an oracle user that can log in to the cdb as sysdba. +- `` is the password of dba_user. +- `` is the oracle db hostname. +- `` is the oracle db port. +- `` is the oracle PDB name -The new `` needs `SELECT` access to source tables. Oracle doesn't support -granting access to an entire schema, so you need to do this for each table. +For each table you want to migrate under this PDB, grant its select permission to the MIGRATION_USER you just created: ```sql -GRANT SELECT ON TO +GRANT SELECT ON . TO ``` Where: - - `` is the name of the user to create for CDC migration table access. - - `` is the name of an individual table to migrate. -### Validate configuration +- `` is the schema to which your table belongs. +- `` is the name of a table you want to migrate. +- `` is the name of the common user you created in previous step. + +!!!note + In oracle, a schema is identical to a user, so remember to set schema name correctly when running the above sql, otherwise data migration might encounter errors. + +### (Optional)Step6: Ensure adequate redo log space is available for better performance + +The migration process involves two phases. The first is a consistent snapshot. The second is continuous streaming of database changes. This stream of database changes is powered by LogMiner and the Oracle DB redo logs. + +Database changes have a limited lifetime on the redo logs before the change is no longer present in the log history. This lifetime depends on the size of the redo logs, the number of redo logs, and the change throughput to the database. Also, undersized logs cause frequent log switching and affect migration performance. + +To examine the state of the database redo logs, log in to ***CDB*** first: + +```shell +sqlplus /@:/ as sysdba +``` + +Then run following sqls: + +```sql +SELECT GROUP#, TYPE, MEMBER FROM V_$LOGFILE; + + GROUP# TYPE MEMBER +---------- ------- -------------------------------------------------- + 1 ONLINE /opt/oracle/oradata/ORCLCDB/redo03.log + 2 ONLINE /opt/oracle/oradata/ORCLCDB/redo01.log + 3 ONLINE /opt/oracle/oradata/ORCLCDB/redo04.log + +SELECT GROUP#, ARCHIVED, BYTES/1024/1024 MB, STATUS FROM V_$LOG; + + GROUP# ARC MB STATUS +---------- --- --- ---------------- + 1 YES 2000 INACTIVE + 3 YES 2000 INACTIVE + 3 NO 2000 CURRENT +``` + +This example uses three log groups of size 2000MB. Each group has one file member. This might be too +small for many production databases. You can safely adjust the redo logs with synchronous commands such as the following: + +```sql + ALTER DATABASE ADD LOGFILE GROUP 4 ('/opt/oracle/oradata/ORCLCDB/redo04.log') SIZE 8G; + ALTER DATABASE ADD LOGFILE GROUP 5 ('/opt/oracle/oradata/ORCLCDB/redo05.log') SIZE 8G; + ALTER DATABASE ADD LOGFILE GROUP 6 ('/opt/oracle/oradata/ORCLCDB/redo06.log') SIZE 8G; + ALTER DATABASE ADD LOGFILE GROUP 7 ('/opt/oracle/oradata/ORCLCDB/redo07.log') SIZE 8G; + ALTER SYSTEM ARCHIVE LOG CURRENT; + ALTER SYSTEM CHECKPOINT; + ALTER SYSTEM ARCHIVE LOG CURRENT; + ALTER SYSTEM CHECKPOINT; + ALTER SYSTEM ARCHIVE LOG CURRENT; + ALTER SYSTEM CHECKPOINT; + ALTER DATABASE DROP LOGFILE GROUP 1; + ALTER DATABASE DROP LOGFILE GROUP 2; + ALTER DATABASE DROP LOGFILE GROUP 3; +``` + +These commands result in four new 8GB log groups. Each group has a single log file. + +Consult your DBA for appropriate production sizing. + +### Step7: Validate configuration -The EDB DMS Reader installation (packaged as `cdcreader`) comes with a helper script that validates the Oracle configuration and helps you identify any issues. After you configure the database, we recommend running the script to ensure all checks pass. +The EDB DMS Reader installation (packaged as `cdcreader`) and DMS Agent installation (packaged as `cdcagent`) come with a helper script that validates the Oracle configuration and helps you identify any issues. After you configure the database, we recommend running the script to ensure all checks pass. Run the script without arguments to print the usage: ```shell +# run this command if you're using cdcagent +/opt/cdcagent/reader/oracleConfigValidation.sh +# run this command if you're using cdcreader: /opt/cdcreader/oracleConfigValidation.sh ``` ## SSL configuration -Ensure you configure your source database server to accept SSL connections to allow the EDB DMS Reader to connect to it. You must create a server certificate and a server private key, for example, with OpenSSL, to enable this configuration. +Ensure you configure your source database server to accept SSL connections to allow the EDB DMS Reader or DMS Agent to connect to it. You must create a server certificate and a server private key, for example, with OpenSSL, to enable this configuration. ## More information diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/index.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/index.mdx index 81e9cf79d43..01827e62efb 100644 --- a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/index.mdx +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/index.mdx @@ -22,9 +22,8 @@ navRootedTo: /edb-postgres-ai/migration-etl **EDB Postgres® AI migrations powered by EDB Data Migration Service** -!!!note Notice +!!!note Data Migration Service capabilities are only available as part of the EDB Postgres AI Hybrid Manager (HM), which is currently in tech preview. -!!! EDB Data Migration Service (DMS) offers a secure and fault-tolerant way to migrate database data. Using change data capture or CDC and event streaming, source database row changes are replicated to the migration destination. You can select a subset of your schemas' tables to migrate including support for schema, table, and column name remapping. diff --git a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/terminology.mdx b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/terminology.mdx index e62e9ad1d89..836b26cd664 100644 --- a/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/terminology.mdx +++ b/advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/terminology.mdx @@ -8,23 +8,45 @@ This terminology is important to understand EDB Data Migration Service (DMS) fun ## Change Data Capture (CDC) -CDC is a set of software design patterns used to determine and track changes in data sets by calculating *deltas*. EDB Data Migration Service uses CDC to stream changes from a database cluster to another. +CDC is a set of software design patterns used to determine and track changes in data sets by calculating *deltas*. EDB Data Migration Service uses CDC to stream changes from a database cluster to another. + + +## EDB DMS Agent + +The [EDB Data Migration Service Agent](getting_started/installing_agent), packaged as `cdcagent`, combines legacy DMS Reader and DMS Writer's function. Starting from Hybrid Manager 1.3, this new agent should be used. + +When running in reader mode, it uses Debezium to perform CDC operations on the source database and Kafka/Object storage to produce messages containing the change events. + +When running in writer mode, it consumes CDC records from Kafka/Object storage to replicate and apply changes on the destination database. + +Install the DMS Agent on your source machine, configure and launch it in reader mode when performing any type of migrations, as it allows you to connect to and read the source. + +If the destination is another self-managed database, you need to install the DMS Agent on your destination machine, configure and launch it in writer mode as it allows you to connect to self-managed destinations. If the destination for your migration is an Hybrid Manager-managed (HM-managed) database, you don't need to install and launch it. + +!!!note + EDB DMS Agent is only compatible with Hybrid Manager versions 1.3 and above. For Hybrid Manager 1.2 and below, please use EDB DMS Reader and EDB DMS Writer. ## EDB DMS Reader -The [EDB Data Migration Service Reader](getting_started/installing), packaged as `cdcreader`, uses Debezium to perform CDC operations on the source database and Kafka to produce messages containing the change events. +The [EDB Data Migration Service Reader](getting_started/installing), packaged as `cdcreader`, uses Debezium to perform CDC operations on the source database and Kafka/Object storage to produce messages containing the change events. Install and configure the DMS Reader when performing any type of migrations, as it allows you to connect to and read the source. If the destination for your migration is an Hybrid Manager-managed (HM-managed) database, you don't need to install the [DMS Writer](terminology.mdx#edb-dms-writer). If the destination is another self-managed database, install the [DMS Writer](terminology.mdx#edb-dms-writer). +!!!note + EDB DMS Reader is only compatible with Hybrid Manager versions 1.2 and below. For Hybrid Manager 1.3 and above, please use EDB DMS Agent. + ## EDB DMS Writer -The [EDB Data Migration Service Writer](getting_started/installing2), packaged as `cdcwriter`, uses CDC records to replicate and apply changes on the destination database. +The [EDB Data Migration Service Writer](getting_started/installing2), packaged as `cdcwriter`, consumes CDC records from Kafka/Object storage to replicate and apply changes on the destination database. Install and configure the DMS Writer on your destination machine when performing migrations to self-managed destination databases, as it allows you to connect to self-managed destinations. Additionally install the [DMS Reader](terminology.mdx#edb-dms-reader) for any type of migrations. +!!!note + EDB DMS Writer is only compatible with Hybrid Manager versions 1.2 and below. For Hybrid Manager 1.3 and above, please use EDB DMS Agent. + ## Auxiliary machine -An auxiliary machine is a Linux-based server (bare metal, VM, or local) dedicated to running the EDB DMS Reader and Writer for database migrations. This machine acts as the migration engine, capable of connecting to various source and destination databases, whether local or remote. +An auxiliary machine is a Linux-based server (bare metal, VM, or local) dedicated to running the EDB DMS Reader and Writer for database migrations. This machine acts as the migration engine, capable of connecting to various source and destination databases, whether local or remote. While it can be the same host as where you have a source or destination database, as well as the same host where you have the HM running, its only requirement is a Linux operating system, since it is the only platform currently supporting the EDB DMS Reader and Writer. @@ -46,7 +68,7 @@ When [creating migrations](getting_started/create_migration.mdx), you can choose ### Managed Postgres -Managed Postgres destinations are databases created and managed through the HM. All database clusters created within the project where you are performing the migration can be selected as possible migration destinations. +Managed Postgres destinations are databases created and managed through the HM. All database clusters created within the project where you are performing the migration can be selected as possible migration destinations. ### External Postgres