-
Notifications
You must be signed in to change notification settings - Fork 1.8k
feat(new sink): add Apache Doris sink support #23117
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Created Jira card for Docs Team review. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some small suggestions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 for docs
Hi @bingquanzhao, thank you for this PR. Please rebase on master and fix merge conflicts. There are 12k affected lines right now. |
- Update socket2 from 0.5.8 to 0.5.10 - Update sqlx from 0.8.3 to 0.8.6 - Keep mysql support in sqlx features for Doris sink
Thanks for your reminder, I have executed rebase master |
Still 12k lines added, find the offending commit and revert it or git checkout affected files from master |
1720078
to
ffe54be
Compare
Thank you for your help. I think I previously executed an incorrect command for generating the document, which resulted in modifying the source files of other parts. |
Summary
This PR introduces a new Apache Doris sink for Vector, enabling users to send log data directly to Apache Doris databases using the Stream Load API. The implementation includes:
Apache Doris is a modern MPP analytical database that provides sub-second query response times on large datasets, making it ideal for real-time data warehouses and log analysis scenarios.
Change Type
Is this a breaking change?
How did you test this PR?
Local Testing
cargo test
vector validate
make generate-component-docs
./scripts/check_changelog_fragments.sh
Test Configuration Used
Environment Setup
Does this PR include user facing changes?
Notes
Implementation Details
format
: Data format specification (json, csv, etc.)read_json_by_line
: JSON line-by-line reading modestrip_outer_array
: Array handling configurationcolumns
: Column mapping specificationDocumentation
CI=true make check-docs
)Dependencies
Code Quality
cargo fmt
Testing Strategy
References