Add StarRocks as a first-class data source (correct SQL dialect)

[inaboishchikov]

Summary

I'd like to add StarRocks as a first-class data source in core/wren (CLI/SDK). I have a working implementation ready as a PR and would like to align on direction before opening it, as suggested in CONTRIBUTING.

Problem

StarRocks speaks the MySQL wire protocol, so today it can be connected through the doris data source (port 9030). Connection, models, joins and cubes work. However, SQL is transpiled with the sqlglot doris dialect, which differs from StarRocks. Tested against a live StarRocks cluster with wrenai 0.13.3 / 0.15.0:

Query	doris dialect	StarRocks
date_trunc('month', d)	emitted as DATE_TRUNC(d, 'MONTH') → "date_trunc requires first parameter must be a string constant"	needs DATE_TRUNC('MONTH', d)
array_agg(x)	rewritten to COLLECT_LIST(x) → "No matching function"	ARRAY_AGG(x)
group_concat(x)	results with doubled separators (a,,b,,c)	GROUP_CONCAT(x SEPARATOR ',')

Using mysql instead doesn't help: the same driver, but date_trunc turns into a STR_TO_DATE(CONCAT(...)) workaround and array_agg is unsupported.

Also, wren context import dbt fails for dbt-starrocks projects with Unsupported dbt adapter 'starrocks'.

sqlglot already ships a starrocks dialect. Overriding the dialect map for doris → starrocks (a one-line monkeypatch) made all three queries above return correct results, with the same connection and driver.

Proposal

Add a starrocks data source that reuses the existing MySQL/Doris connector but transpiles with sqlglot's starrocks dialect:

• core/wren: DataSource.starrocks, StarRocksConnectionInfo (same fields as Doris, port 9030), connector factory/registry, field registry, valid dialects, dbt adapter mapping (username/schema as in dbt-starrocks)
• core/wren-core-base + core/wren-core: DataSource::StarRocks using the existing MySQLDialect
• core/wren-mdl/mdl.schema.json: STARROCKS / starrocks
• docs + unit tests

Existing doris behaviour is unchanged.

Questions

1. Is a separate starrocks data source acceptable, or would you prefer something else (e.g. a dialect option on doris)?
2. The change touches both the Rust crates and the Python package. Is it fine to send it as a single PR, or would you rather split Rust (enum + dialect + schema) from Python?
3. Anything else you'd want covered (e.g. connector integration tests with a StarRocks container)?

I have the branch ready (unit tests pass; I could not compile the Rust part locally, so I'd rely on CI there) and can open the PR as soon as the direction is agreed.

[goldmedal]

Hi @inaboishchikov, we are happy to accept the PR for StarRock. Feel free to submit a PR.

Is a separate starrocks data source acceptable, or would you prefer something else (e.g. a dialect option on doris)?

I think a separate data source is fine.

The change touches both the Rust crates and the Python package. Is it fine to send it as a single PR, or would you rather split Rust (enum + dialect + schema) from Python?

I guess you only add the enum for Rust. It's not a big change for the Rust core module. It's fine to send a single PR for the StartRock data source.

Anything else you'd want covered (e.g. connector integration tests with a StarRocks container)?

It's better to add some integration tests on the Python side.

Thanks for your contribution 🚀

[inaboishchikov]

Thanks @goldmedal! The PR is up: #2790

It follows what we agreed on:

• separate starrocks data source, reusing the MySQL/Doris connector and transpiling with sqlglot's starrocks dialect;
• on the Rust side only the DataSource::StarRocks enum variant (mapped to the existing MySQLDialect) plus the mdl.schema.json entry;
• one PR, with unit tests and a testcontainers integration test (just test-starrocks, image starrocks/allin1-ubuntu:3.5.21) that runs the shared query suite and checks the three dialect cases from the description (date_trunc, array_agg, group_concat).

Two things I could not verify locally: the integration test has not been run, and the Rust change has not been compiled, as I don't have a Rust toolchain at hand. Since core/wren depends on the published wren-core-py, the integration test will need a wren-core-py build that includes the new enum. I would be glad to follow your usual process for that, and I'll fix whatever CI reports.

[inaboishchikov]

The PR is merged (#2790), thank you @goldmedal for the quick decisions and especially for running the integration test locally and sending the fixes, that made the difference.

For anyone who lands here looking for StarRocks support: once it is part of a release, use the new data source instead of doris:

datasource: starrocks   # connection fields are the same as for doris (host, port 9030, database, user, password)

It uses the same MySQL driver, but SQL is transpiled with the StarRocks dialect, so date_trunc, array_agg and group_concat work as expected. The dbt starrocks adapter is also supported in wren context import dbt (default catalog only for now).

[goldmedal]

StarRock has been released. https://github.com/Canner/WrenAI/releases/tag/wren-v0.16.0
Thanks @inaboishchikov 👍