Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 21 additions & 21 deletions HANDOFF.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,43 +2,43 @@

## Finished

- Added a hand-rolled `netsuite_suiteql` customer source using the existing OAuth1 TBA, retry, raw-schema, watermark, and SCD1 runtime.
- Added a stateful six-operation FastAPI simulator with invented customers, real HMAC-SHA256 signature verification, offset paging, and named auth/throttle/permission/malformed failures.
- Proved stateful create, update, duplicate-free replay, monotonic watermarks, transport-link removal, and proof-fixture cleanup over loopback HTTP.
- Added the tracked OpenAPI contract, customer staging model, and explicit “simulator-validated, not NetSuite-validated” documentation.
- Preserved completed Odoo work unchanged on `codex/odoo-json2` at `14244cd`.
- Merged the simulator-validated NetSuite SuiteQL foundation through protected PR #52.
- Added a read-only Odoo 19+ JSON-2 `res.partner` source, connector template, staging model, and operator guide.
- Proved Odoo against ephemeral official Odoo/PostgreSQL containers, then removed them.
- Rebased the Odoo work onto the NetSuite-enabled `main` while preserving both enterprise sources.
- Preserved the original Odoo commit on `backup/odoo-json2-pre-netsuite`.

## Try It

```bash
uv sync --extra dev
uv run python -m dander.dev.netsuite_simulator
uv run pytest tests/integration/test_netsuite_simulator.py
cp connectors/odoo.example.yaml connectors/odoo.yaml
export ODOO_API_KEY='YOUR_API_KEY'
uv run dander run odoo --dry-run --sandbox --project YOUR_PROJECT
```

## Checks

- Ruff lint and format passed; strict mypy passed for all 150 source files.
- All 664 tests passed.
- Wheel/sdist build and inspection passed; NetSuite contract/docs/fixtures are packaged.
- A source-free wheel installed with the documented `dev` extra and exposed all six simulator operations.
- No NetSuite tenant, GCP resource, Terraform state, public package, or remote branch was changed.
- Ruff lint/format and strict mypy passed; all 672 tests passed after the rebase.
- NetSuite, Odoo, Workday, and generic dlt source routing pass together through one adapter helper.
- The earlier Odoo live JSON-2 acceptance passed with bounded two-row pages, bearer auth, database routing, null normalization, and watermark replay.
- No Odoo deployment, retained-project change, version bump, or package publication occurred.

## Decisions

- SuiteQL replaces the old record-list example because the latter returns only IDs and links; the query is uniquely ordered by customer ID.
- The first slice is a full read with bounded pages and idempotent SCD1 replay; 100,000-row and concurrent-offset limits remain explicit.
- OAuth1 TBA is compatibility coverage only. Current OAuth2 and one real-tenant proof gate any supported future release.
- Odoo targets the current JSON-2 API; deprecated XML-RPC/JSON-RPC is not supported.
- The first slice is read-only `res.partner`; no retained-project Odoo pipeline was added.
- Salesforce remains memory-bounded but still rereads Accounts through synchronous QueryAll.

## Remaining

- Push `codex/netsuite-simulator` and open a focused PR only when requested.
- Push the rebased Odoo branch and open the focused PR.
- Let protected CI repeat Linux tests, packaging, scans, and Terraform validation.
- Obtain an authorized NetSuite/SDN sandbox for the narrow acceptance in `docs/netsuite-simulator.md`.
- Keep the connector out of the supported `0.2.0` surface; consider `0.3.0` only after tenant acceptance.
- Treat Odoo offset paging during concurrent source mutation as a documented first-slice limit.
- Scope Salesforce Bulk API 2.0 plus server-filtered SOQL as separate scale work.
- Continue reviewing the daily operator soak in issue #26.

## Review First

- `src/dander/ingestion/enterprise.py`
- `src/dander/dev/netsuite_simulator.py`
- `tests/integration/test_netsuite_simulator.py`
- `connectors/odoo.example.yaml`
- `docs/odoo.md`
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,11 @@ client credentials, primitive internal values, stable offset paging, and a decla
The first slice performs a full read and idempotent SCD1 publication; it does not claim unsafe
timestamp-plus-offset incrementality. See [`docs/servicenow.md`](docs/servicenow.md).

`connectors/odoo.example.yaml` reads Odoo 19+ contacts and companies through the current JSON-2
API using a bearer API key, bounded pages, and an inclusive `write_date` watermark. Odoo Online
requires a Custom plan for external API access; the official Odoo Community Docker image provides
a free local development target. See [`docs/odoo.md`](docs/odoo.md).

`connectors/netsuite.example.yaml` is a **simulator-validated, not NetSuite-validated** customer
SuiteQL slice. It uses bounded offset paging, stable ordering, declared schemas, and the existing
OAuth1 TBA signer. It is not part of the public `0.2.0` support surface; real-tenant acceptance and
Expand Down
40 changes: 40 additions & 0 deletions connectors/odoo.example.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Read-only Odoo 19+ JSON-2 connector. The API key remains a secret reference; for an Odoo host
# that serves only one database, remove auth_options.database and the X-Odoo-Database header is
# omitted. Odoo Online requires a Custom plan for external API access. Odoo Community can be run
# locally with the official Odoo and PostgreSQL images for no-license-cost development.
name: odoo
engine: odoo_json2
base_url: http://localhost:8069
auth_strategy: api_key_bearer
auth_ref: ODOO_API_KEY
auth_options:
database: dander_odoo
rate_limit:
requests_per_second: 4
burst: 1
backoff: exponential
max_retries: 4

endpoints:
- name: partners
path: /json/2/res.partner/search_read
pagination:
kind: offset
page_size: 100
primary_key: [id]
incremental_cursor: write_date
field_types:
id: INT64
active: BOOL
is_company: BOOL
raw_schema:
- {name: id, type: INT64, mode: REQUIRED}
- {name: name, type: STRING}
- {name: email, type: STRING}
- {name: phone, type: STRING}
- {name: city, type: STRING}
- {name: country_code, type: STRING}
- {name: is_company, type: BOOL, mode: REQUIRED}
- {name: active, type: BOOL, mode: REQUIRED}
- {name: create_date, type: STRING}
- {name: write_date, type: STRING, mode: REQUIRED}
10 changes: 10 additions & 0 deletions docs/decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -393,3 +393,13 @@
- OAuth1 TBA is retained only to exercise Dander's existing compatibility strategy. Oracle's
announced 2027.1 restriction on new TBA REST integrations makes current OAuth2 acceptance a
release gate; until a real tenant passes, the connector is not NetSuite-validated or supported.

## 2026-08-04 — Odoo starts on JSON-2 against Community

- Odoo 19+'s JSON-2 API is the connector contract. New work does not depend on the deprecated
XML-RPC/JSON-RPC endpoints.
- The first vertical slice reads only `res.partner` through the existing Source/runtime/writer
boundary, using API-key bearer auth, bounded offset pages, and declared raw fields.
- Official Odoo Community and PostgreSQL containers are the free acceptance target because Odoo
Online exposes its external API only on the Custom plan. Concurrently mutating large tables
need later snapshot/keyset paging before this slice is described as scale-ready.
42 changes: 42 additions & 0 deletions docs/odoo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Odoo JSON-2 connector

Dander's first Odoo slice reads contacts and companies (`res.partner`) through the Odoo 19+
JSON-2 API. It uses bearer API-key authentication, bounded offset pages, declared raw fields,
idempotent SCD1 publication, and an inclusive `write_date` watermark boundary.

## Free development environment

Odoo Online's free and Standard plans do not expose the external API. For development without an
Odoo license fee, run Odoo 19 Community and PostgreSQL locally using the official Docker images.
The official image instructions show the required PostgreSQL service and Odoo port `8069`.

After creating a database and admin user, create an API key under **Preferences → Account
Security → New API Key**. Copy `connectors/odoo.example.yaml` to `connectors/odoo.yaml`, set the
database name, and expose only the key to Dander:

```bash
cp connectors/odoo.example.yaml connectors/odoo.yaml
read -rs ODOO_API_KEY && printf '\n'
export ODOO_API_KEY
uv run dander run odoo --dry-run --sandbox --project YOUR_NO_BILLING_GCP_PROJECT
uv run dander run odoo --sandbox --project YOUR_NO_BILLING_GCP_PROJECT \
--build-models --select-model stg_odoo__partners
```

For hosted execution, store the key in Secret Manager and map `ODOO_API_KEY` to that secret in the
pipeline manifest. Never commit the key or a populated local connector.

## Current boundary

This is intentionally one read-only model. It does not write to Odoo, discover arbitrary models,
or support Odoo 18's deprecated XML-RPC/JSON-RPC APIs. Offset pages are ordered by immutable Odoo
record ID. The inclusive `write_date` boundary may replay tied rows; Dander's SCD1 writer makes
that safe. Concurrent source mutations can still shift membership between offset pages, so this
first slice is intended for evaluation and modest tables until snapshot/keyset paging is added.
API keys expire and must be rotated according to the Odoo account's policy.

References:

- [Odoo 19 external JSON-2 API](https://www.odoo.com/documentation/19.0/developer/reference/external_api.html)
- [Official Odoo Docker image](https://hub.docker.com/_/odoo)
- [Odoo pricing](https://www.odoo.com/pricing)
29 changes: 29 additions & 0 deletions models/staging/stg_odoo__partners.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
-- Purpose: normalize Odoo contacts and companies for CRM analysis.
-- Grain: one row per Odoo res.partner id.
WITH source AS (
SELECT
id AS partner_id,
name AS partner_name,
email,
phone,
city,
country_code,
is_company,
active AS is_active,
SAFE.PARSE_TIMESTAMP('%F %H:%M:%S', create_date) AS created_at,
PARSE_TIMESTAMP('%F %H:%M:%S', write_date) AS updated_at
FROM {{ ref('raw_odoo_partners') }}
)

SELECT
partner_id,
partner_name,
email,
phone,
city,
country_code,
is_company,
is_active,
created_at,
updated_at
FROM source
56 changes: 56 additions & 0 deletions models/staging/stg_odoo__partners.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
model: stg_odoo__partners
description: Odoo contacts and companies normalized from the read-only JSON-2 contract.
owner: data-eng
materialization: incremental
dataset: staging
source_system: odoo
sensitivity: confidential
unique_key: [partner_id]
incremental_cursor: updated_at

columns:
- name: partner_id
type: INT64
description: Stable Odoo res.partner identifier.
- name: partner_name
type: STRING
description: Contact or company display name.
- name: email
type: STRING
description: Primary email address when present.
- name: phone
type: STRING
description: Primary telephone number when present.
- name: city
type: STRING
description: Contact city when present.
- name: country_code
type: STRING
description: Contact country code when present.
- name: is_company
type: BOOL
description: Whether this partner represents a company.
- name: is_active
type: BOOL
description: Whether this partner is active.
- name: created_at
type: TIMESTAMP
description: UTC record creation time when available.
- name: updated_at
type: TIMESTAMP
description: UTC record modification watermark.

tests:
- column: partner_id
not_null: true
unique: true
- column: is_company
not_null: true
- column: is_active
not_null: true

metrics:
- name: partner_count
description: Number of Odoo contacts and companies represented.
aggregation: count_distinct
field: partner_id
3 changes: 3 additions & 0 deletions src/dander/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
Endpoint,
IngestionEngine,
NetSuiteSuiteQLSource,
OdooJson2Source,
Source,
SourceConfig,
WorkdayRaasSource,
Expand Down Expand Up @@ -1675,6 +1676,8 @@ def _build_source_adapter(config: SourceConfig, auth: AuthStrategy) -> Source:
return WorkdayRaasSource(config, auth)
if config.engine is IngestionEngine.NETSUITE_SUITEQL:
return NetSuiteSuiteQLSource(config, auth)
if config.engine is IngestionEngine.ODOO_JSON2:
return OdooJson2Source(config, auth)
return DltRestSource(config, auth)


Expand Down
2 changes: 2 additions & 0 deletions src/dander/ingestion/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
EnterpriseSource,
EnterpriseSourceError,
NetSuiteSuiteQLSource,
OdooJson2Source,
WorkdayRaasSource,
)
from dander.ingestion.pagination import (
Expand Down Expand Up @@ -37,6 +38,7 @@
"NoPagination",
"NetSuiteSuiteQLSource",
"OffsetPagination",
"OdooJson2Source",
"PageNumberPagination",
"PaginationKind",
"PaginationStrategy",
Expand Down
Loading
Loading