Skip to content

feat: use clickhouse client config file for SSL configuration in clickhouse migrations#9

Merged
scotwells merged 3 commits intomainfrom
feat/support-secure-clickhouse-connections
Dec 23, 2025
Merged

feat: use clickhouse client config file for SSL configuration in clickhouse migrations#9
scotwells merged 3 commits intomainfrom
feat/support-secure-clickhouse-connections

Conversation

@scotwells
Copy link
Contributor

@scotwells scotwells commented Dec 22, 2025

I was still having issues applying the migrations built in datum-cloud/engineering#8 because the apply_migrations() function wasn't using the secure configuration. I also realized that the current approach to building up the client command wouldn't be very flexible for consumers.

I adjusted the script so users can now just leverage the CLICKHOUSE_CLIENT_EXTRA_ARGS environment variable to inject additional arguments into the migration commands.

I pointed the staging environment at the migrations file in this branch so I could confirm that migrations executed successfully:

$ kubectl -n activity-system logs job/clickhouse-migrate
[INFO] ClickHouse Migration Runner Starting...
[INFO] Target: clickhouse-activity-clickhouse:9440
[INFO] Database: audit
[INFO] Migrations Directory: /migrations

[INFO] Waiting for ClickHouse to be ready at clickhouse-activity-clickhouse:9440...
[SUCCESS] ClickHouse is ready!

[INFO] Initializing schema_migrations table...
[SUCCESS] Schema migrations table is ready

[INFO] Looking for migration files in /migrations...
[INFO] Applying migration 001_initial_schema...
[SUCCESS] Migration 001_initial_schema applied successfully
[SUCCESS] Migrations complete: 1 applied out of 1 total
[INFO] Current migration status:
   ┌─version─┬─name───────────┬──────────────applied_at─┬─checksum_short─┐
1. │       1 │ initial_schema │ 2025-12-22 21:35:48.887 │ 8b52391bc576   │
   └─────────┴────────────────┴─────────────────────────┴────────────────┘

[INFO] Verifying schema...
[SUCCESS] Schema verification passed
[INFO] Table structure:
    ┌─name──────────┬─type───────────────────┬─default_type─┬─default_expression───────────────────────────────────────────────────────────────────────────────────┬─comment─┬─codec_expression─┬─ttl_expression─┐
 1. │ event_json    │ String                 │              │                                                                                                      │         │ ZSTD(3)          │                │
 2. │ timestamp     │ DateTime64(3)          │ MATERIALIZED │ coalesce(parseDateTime64BestEffortOrNull(JSONExtractString(event_json, 'stageTimestamp')), now64(3)) │         │                  │                │
 3. │ scope_type    │ LowCardinality(String) │ MATERIALIZED │ coalesce(JSONExtractString(event_json, 'annotations', 'platform.miloapis.com/scope.type'), '')       │         │                  │                │
 4. │ scope_name    │ String                 │ MATERIALIZED │ coalesce(JSONExtractString(event_json, 'annotations', 'platform.miloapis.com/scope.name'), '')       │         │                  │                │
 5. │ user          │ String                 │ MATERIALIZED │ coalesce(JSONExtractString(event_json, 'user', 'username'), '')                                      │         │                  │                │
 6. │ audit_id      │ UUID                   │ MATERIALIZED │ toUUIDOrZero(coalesce(JSONExtractString(event_json, 'auditID'), ''))                                 │         │                  │                │
 7. │ stage         │ LowCardinality(String) │ MATERIALIZED │ coalesce(JSONExtractString(event_json, 'stage'), '')                                                 │         │                  │                │
 8. │ verb          │ LowCardinality(String) │ MATERIALIZED │ coalesce(JSONExtractString(event_json, 'verb'), '')                                                  │         │                  │                │
 9. │ api_group     │ LowCardinality(String) │ MATERIALIZED │ coalesce(JSONExtractString(event_json, 'objectRef', 'apiGroup'), '')                                 │         │                  │                │
10. │ resource      │ LowCardinality(String) │ MATERIALIZED │ coalesce(JSONExtractString(event_json, 'objectRef', 'resource'), '')                                 │         │                  │                │
11. │ namespace     │ LowCardinality(String) │ MATERIALIZED │ coalesce(JSONExtractString(event_json, 'objectRef', 'namespace'), '')                                │         │                  │                │
12. │ resource_name │ String                 │ MATERIALIZED │ coalesce(JSONExtractString(event_json, 'objectRef', 'name'), '')                                     │         │                  │                │
13. │ status_code   │ UInt16                 │ MATERIALIZED │ toUInt16OrZero(JSONExtractString(event_json, 'responseStatus', 'code'))                              │         │                  │                │
14. │ bucket        │ UInt8                  │ MATERIALIZED │ cityHash64(audit_id) % 16                                                                            │         │                  │                │
    └───────────────┴────────────────────────┴──────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────┴──────────────────┴────────────────┘

[SUCCESS] All migrations completed successfully!

Relates to datum-cloud/enhancements#536

This change introduces the ability for the activity-apiserver to connect
to a clickhouse database using a secure connection. A client certificate
is expected to be used to connect to clickhouse.
@scotwells scotwells changed the title feat: use clickhouse client config file for SSL configuration feat: use clickhouse client config file for SSL configuration in clickhouse migrations Dec 22, 2025
This change introduces the ability for the
[activity-apiserver](https://github.com/datum-cloud/activity/blob/main/docs/components/apiserver-architecture.md)
to connect to a clickhouse database using a secure connection. A client
certificate is expected to be used to connect to clickhouse.

See [Clickhouse's Golang library
docs](https://clickhouse.com/docs/integrations/go#using-tls) for more
information.

---

Relates to https://github.com/datum-cloud/engineering/issues/90
@scotwells scotwells merged commit a5e269e into main Dec 23, 2025
4 checks passed
@scotwells scotwells deleted the feat/support-secure-clickhouse-connections branch December 23, 2025 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants