Repository navigation
Replies: 1 comment 5 replies
|
Hey @jiangxt2:
|
5 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Summary
daft-dorisis an independently maintained Python package that connects Daft to Apache Doris. It provides reads from Doris physical tables and batch writes through HTTP Stream Load.The proposal is to list
daft-dorisunder Daft's Community Extensions.daft-dorisis a third-party, independently maintained connector for Apache Doris.Proposed directory entry
daft-dorisDataSource/DataSink)jiangxt2/daft-dorisWhy this connector
Apache Doris has connector-specific capabilities that are not captured by a generic SQL reader alone:
The package provides these capabilities through Daft's custom connector extension points while keeping Doris-specific dependencies, compatibility testing, and release cadence in the connector repository.
Usage
Version 1.0 is available from PyPI. Install it with the Doris extra:
pip install "daft-doris[doris]"From a source checkout:
pip install -e ".[doris]"Read a Doris table:
Write a Daft DataFrame through Stream Load; connection setup is omitted here:
Initial scope
split="single"is the default.split="auto"can use Doris tablet planning. Projection and the supported subset of Daft filters are pushed down; unsupported predicates remain as residual filters.load/upsert, and line-delimited JSON forpartial_update.upsertrequires a Unique Key table.partial_updaterequires a Merge-on-Write Unique Key table.Important boundaries
Fail,Label Already Exists, and malformed HTTP 4xx responses are known failures. Response loss and invalid post-transmission counters are generally ambiguous. The connector does not replay a payload under a new label.unsafe_where_sqlis a trusted predicate fragment, not an injection defense.Compatibility and validation
The current Alpha profile is intentionally narrow: Python 3.12/3.13, Daft 0.7.23, PyArrow 19.0.1, PyMySQL 1.2.0, optional ADBC dependencies for Flight, and Doris 4.0.6 as the certified real-service baseline. Python 3.12 is the primary full-suite baseline; Python 3.13 has a separate native Stream Load/MySQL smoke test.
The latest Doris 4.0.6 integration run passed 62 tests covering MySQL and Flight reads, tablet planning, Ray read coexistence, the Ray writer boundary, key-model write paths, redirects, labels, error handling, resource cleanup, least-privilege boundaries, and certified scalar/temporal/decimal/nullable and JSON value paths. The separate Python 3.13 smoke test also completed native Stream Load and MySQL readback. These results establish the current Alpha profile; they do not imply compatibility with every Daft, PyArrow, Python, or Doris version.
Why it fits the community model
DataSource,DataSourceTask, andDataSinkextension interfaces.If accepted, I can submit a small documentation PR with the proposed directory entry, installation command, supported scope, and Alpha limitations.
Questions for Daft maintainers
daft-dorisa good fit for the Community Extensions list as a third-party customDataSource/DataSinkconnector?References
daft-dorison PyPIdaft-dorisrepositoryREADME.md,architecture.md,writing.md, andtests/it-ledger.mdAll reactions