feat: Oracle replication

[chubei]

Setup Oracle Server

https://github.com/oracle/docker-images/blob/main/OracleDatabase/SingleInstance/README.md#running-oracle-database-enterprise-and-standard-edition-2-in-a-container

After building the 19.3.0-ee docker image, run:

docker run --name oracle -p 1521:1521 -p 5500:5500 -e ORACLE_PWD=123 -e ENABLE_ARCHIVELOG=true -v ./data:/opt/oracle/oradata oracle/database:19.3.0-ee

Populate data

First login as sysdba to the PDB:

./sqlplus sys/123@//localhost:1521/ORCLPDB1 as sysdba

Create a new user named CHUBEI (this is hardcoded in the test binary) and insert some data:

CREATE USER chubei IDENTIFIED BY 123;
GRANT CREATE SESSION TO chubei;
GRANT CREATE TABLE TO chubei;
ALTER USER CHUBEI QUOTA UNLIMITED ON USERS;
CREATE TABLE employees (
  employee_id NUMBER,
  first_name VARCHAR2(50),
  last_name VARCHAR2(50),
  hire_date DATE,
  salary NUMBER
);
INSERT INTO employees (employee_id, first_name, last_name, hire_date, salary)
VALUES (1, 'John', 'Doe', TO_DATE('2022-01-01', 'YYYY-MM-DD'), 50000);

INSERT INTO employees (employee_id, first_name, last_name, hire_date, salary)
VALUES (2, 'Jane', 'Smith', TO_DATE('2022-02-15', 'YYYY-MM-DD'), 60000);

Create the replication user

Login to CDB as sysdba:

./sqlplus sys/123@//localhost:1521/ORCLCDB as sysdba

Enable supplemental logging, create replication user C##DOZER (this is hardcoded in test binary) and grant permissions:

ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
CREATE USER C##DOZER IDENTIFIED BY 123;
ALTER USER C##DOZER SET CONTAINER_DATA=ALL CONTAINER=CURRENT;
GRANT CREATE SESSION TO C##DOZER;
GRANT SELECT ON SYS.V_$LOG TO C##DOZER;
GRANT SELECT ON SYS.V_$LOGFILE TO C##DOZER;
GRANT SELECT ON SYS.V_$ARCHIVED_LOG TO C##DOZER;
GRANT SELECT ON SYS.V_$LOGMNR_CONTENTS TO C##DOZER;
GRANT LOGMINING TO C##DOZER;
GRANT EXECUTE ON DBMS_LOGMNR TO C##DOZER;

Login to PDB again and grant more permissions:

./sqlplus sys/123@//localhost:1521/ORCLPDB1 as sysdba

GRANT CREATE TYPE TO C##DOZER;
GRANT SELECT ON SYS.ALL_TABLES to C##DOZER;
GRANT SELECT ON SYS.ALL_TAB_COLUMNS TO C##DOZER;
GRANT SELECT ON SYS.ALL_CONSTRAINTS TO C##DOZER;
GRANT SELECT ON SYS.ALL_CONS_COLUMNS TO C##DOZER;
GRANT EXECUTE ON SYS.DBMS_FLASHBACK TO C##DOZER;
GRANT SELECT ON CHUBEI.EMPLOYEES TO C##DOZER;

Setup Oracle Client

Download client from https://www.oracle.com/database/technologies/instant-client/downloads.html.

Run snapshot and replication

RUST_LOG=info DYLD_LIBRARY_PATH=/Users/chubei/Downloads/instantclient_19_16/ cargo test --target x86_64-apple-darwin -p dozer-ingestion-oracle -- --ignored --nocapture

This is what I use on apple silicon, adjust path and target accordingly.

You should see log like:

[dozer-ingestion/oracle/src/connector/mod.rs:459] message = OperationEvent {
    table_index: 1,
    op: BatchInsert {
        new: [
            Record {
                values: [
                    Decimal(
                        101,
                    ),
                    String(
                        "John",
                    ),
                    String(
                        "Doe",
                    ),
                    Date(
                        2022-01-15,
                    ),
                    Decimal(
                        50000,
                    ),
                ],
                lifetime: None,
            },
        ],
    },
    id: None,
}
[2024-02-03T10:16:24Z INFO  dozer_ingestion_oracle::connector] Replicating log /opt/oracle/oradata/ORCLCDB/redo02.log (1532833, 9295429630892703743), starting from 1534673
[2024-02-03T10:16:24Z INFO  dozer_ingestion_oracle::connector] Replicated all logs, retrying after 10s

Now login as CHUBEI to PDB:

./sqlplus chubei/123@//localhost:1521/ORCLPDB1

And insert some data:

INSERT INTO CHUBEI.EMPLOYEES (EMPLOYEE_ID, FIRST_NAME, LAST_NAME, HIRE_DATE, SALARY) VALUES (101, 'John', 'Doe', TO_DATE('2022-01-15', 'YYYY-MM-DD'), 50000);
COMMIT;

Now the test process should output something like:

[dozer-ingestion/oracle/src/connector/mod.rs:512] message = OperationEvent {
    table_index: 1,
    op: Insert {
        new: Record {
            values: [
                Decimal(
                    101,
                ),
                String(
                    "John",
                ),
                String(
                    "Doe",
                ),
                Date(
                    2022-01-15,
                ),
                Null,
            ],
            lifetime: None,
        },
    },
    id: Some(
        OpIdentifier {
            txid: 1595355,
            seq_in_tx: 0,
        },
    ),
}

Known working types

• NUMBER, converted to Decimal
• VARCHAR2, converted to String
• DATE, converted to Date, in %d-%b-%y format, example: 15-JAN-22
• TIMESTAMP(6), converted to Timestamp, in %d-%b-%y %I.%M.%S%.6f %p format, example: 04-FEB-24 12.19.18.326567 AM

NULLs are already handled.