You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Moved from apache/iggy#2756, which @amuldotexe opened on 2026-02-17. Its 6 comments stay there.
Paths in this text use the apache/iggy layout. core/connectors/<path> is <path> in this repository.
A key ingredient for adoption for Iggy will be relevant connector coverage.
This is still a draft, but I want it to be useful as a live tracker before someone picks up new work.
Last updated: April 9, 2026
Connector Ecosystem Roadmap: tracking connector work, gaps, and active PRs
Overview
Apache Iggy's connector ecosystem is growing. The main purpose of this discussion is to help contributors avoid duplicate work and find the next useful thing to pick up.
The live status in the next two sections should be trusted first. I refreshed those sections on April 9, 2026. Since the April 3 refresh there have not been new connector merges, but there have been connector-adjacent reliability/test fixes and more concrete progress signals in the active work below. The long target map below is still the broader draft and may lag behind merges by a few days.
External Java connector exists. Proper Java integration tests are still open in apache/iggy#2598 and are currently being re-verified after repo sync issues.
Generic HTTP sink shipped in apache/iggy#2925. HTTP source and header-forwarding follow-ups are still discussion-led and have no PR yet.
Active work right now
Please check both open issues and open PRs before starting something new. In a few cases, the issue is open and there is already active PR work behind it.
Open issue. Maintainer discussion leans toward a simple aws-sdk-s3 implementation first, with any richer OpenDAL-based variant as separate follow-up work.
Base InfluxDB connectors shipped in apache/iggy#2933; v3 support is the follow-up. Discussion suggests shared v2/v3 core plus version-specific adapters is feasible.
HTTP sink follow-ups are already taking shape. The merged HTTP sink work in feat(connectors): add generic HTTP sink connector iggy#2925 clearly points toward two next steps: per-message HTTP header forwarding (discussion #3029) and an HTTP source / webhook gateway (discussion #3039). The PR discussion also noted two current trade-offs in the shipped sink: middleware retries do not directly honor Retry-After as the exact sleep duration, and retry attempts are currently more visible in tracing than in connector-level counters.
InfluxDB v3 is still the obvious follow-up after the shipped v2 connector, and the latest issue discussion adds a design hint. In the merged InfluxDB PR (feat(connectors): implement InfluxDB Sink and Source iggy#2933), reviewers explicitly asked about v3 and the author confirmed it should come as a separate follow-up. That is now tracked in Extend InfluxDB Source/Sink connector to V3 iggy#3062, where the latest discussion suggests a shared v2/v3 core plus version-specific adapters rather than a forked implementation.
MongoDB source now has visible proof-of-work, which also reinforces the source DoD gap. The merged MongoDB sink work was intentionally sink-only, and the separate source issue (MongoDB Source Connector #11) now has a working basics update with state/polling proof and a demo video. The same thread also shows uncertainty about the exact source DoD, which is part of why the Pareto source suite still matters.
JDBC / MySQL is further along than the older table wording suggested. The issue comment on JDBC Source and Sink Connectors for Iggy #5 says the JDBC Source is largely implemented and tests are in progress, with the sink planned as a separate PR. That makes JDBC a more active lane than a plain “open issue” label implies.
There is still appetite for extracting more shared sink utilities into the connector SDK. During the HTTP sink review (feat(connectors): add generic HTTP sink connector iggy#2925), maintainers specifically called out duplicate conversion logic and asked for reuse of shared helpers in existing sinks too. That suggests small cross-connector cleanup PRs can be welcome alongside new connector work.
The Pareto source/sink test-suite issues are really about behavior contracts, not just coverage.Create Pareto Source Connector Test Suite #7 and Create Pareto Sink Connector Test Suite #8 were opened to capture a small shared baseline for connector behavior: when progress moves forward, what happens after restart, how replay or duplicate windows are documented, and how partial external failures are surfaced. That makes them a good on-ramp for contributors who want to improve connector correctness without inventing a whole new connector first.
9.6k stars. Wide-column store optimized for high-write workloads. Bloomberg serves 20B+ requests/day on 1,700+ nodes. Used by Netflix, Apple. Essential for time-series and IoT at massive scale.
12.4k GitHub stars. Elasticsearch fork, AWS-aligned. Linux Foundation project. May work as thin adapter over existing ES connector with compatibility layer.
15k customers. 89 of Fortune 100. Acquired by Cisco for $28B. HTTP Event Collector (HEC) API makes integration straightforward. Enterprise observability standard.
Universal data lake landing zone. 350T+ objects stored, 100M+ req/sec. 90%+ Fortune 100 via AWS. Enables Snowflake/Redshift S3 staging. Build with abstract storage interface so GCS/Azure/MinIO become thin adapters.
Planned apache/iggy#2700. 31k GitHub stars. #1 time-series DB by community. IoT standard. Line protocol write API is universal. InfluxDB 3 is built in Rust.
THE IoT protocol. De facto standard with 50%+ enterprise IoT adoption. Billions of connected devices. Iggy's "runs on Raspberry Pi" story is incomplete without MQTT. The single most important missing connector.
Universal integration. HTTP source receives webhooks from any service; HTTP sink pushes to any API. The "connect to anything" fallback. Every web service speaks HTTP.
2nd largest CNCF project. 10k contributors. 1,200 companies. OTLP becoming universal observability standard. Makes Iggy an observability pipeline backbone for logs, traces, and metrics.
(See #9.) De facto IoT standard. 50%+ enterprise adoption. $17.6B IoT protocol market. Billions of devices. The single most important missing connector.
IEC 62541 standard. 40%+ market share in industrial data acquisition. 1,000+ manufacturers. Industry 4.0 standard. Bridges factory floor into streaming.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
A key ingredient for adoption for Iggy will be relevant connector coverage.
This is still a draft, but I want it to be useful as a live tracker before someone picks up new work.
Last updated: April 9, 2026
Connector Ecosystem Roadmap: tracking connector work, gaps, and active PRs
Overview
Apache Iggy's connector ecosystem is growing. The main purpose of this discussion is to help contributors avoid duplicate work and find the next useful thing to pick up.
The live status in the next two sections should be trusted first. I refreshed those sections on April 9, 2026. Since the April 3 refresh there have not been new connector merges, but there have been connector-adjacent reliability/test fixes and more concrete progress signals in the active work below. The long target map below is still the broader draft and may lag behind merges by a few days.
Shipped today
Active work right now
Please check both open issues and open PRs before starting something new. In a few cases, the issue is open and there is already active PR work behind it.
DIRTY.BEHIND.@Luishfs. Working basics, state handling, and polling proof were shared, including a demo video. No PR yet. See also discussion #3043.aws-sdk-s3implementation first, with any richer OpenDAL-based variant as separate follow-up work.BEHIND.Additional insights from merged PR discussions
These are not roadmap commitments, but they did come up explicitly in merged PR discussion threads and linked follow-up issues:
Retry-Afteras the exact sleep duration, and retry attempts are currently more visible in tracing than in connector-level counters._idkeys and replay-aware duplicate handling, but the review discussion also spun out the larger design question around partial external writes and replay-safe progress into Discussion: how should sinks handle partial external writes and replay-safe progress? #10. That broader contract is still open.consume()return value being discarded (bug(connectors): consume() return value discarded in sink runtime iggy#2927) and offsets being auto-committed before sink processing completes (PollingMessages auto-commit commits offsets before sink processing #9). The first now has an implementation PR (fix(connectors): check consume() return value in sink runtime iggy#3061), but the second still looks like an important missing piece for stronger sink delivery semantics.1. Relational Databases
sqlx(54.9M downloads) ormysql_asynctiberius(native TDS protocol)rusqlitesqlx2. NoSQL / Document Databases
mongodb(official, 9.5M downloads)scylla(Cassandra-compatible, pure Rust)aws-sdk-dynamodb(official AWS SDK)reqwestazure_data_cosmosscylla(official Rust driver)reqwest(CouchDB REST API)3. Search & Analytics Engines
opensearch(official Rust client)reqwest(Solr REST API)reqwest(HEC is HTTP POST)meilisearch-sdkreqwest(Typesense REST API)4. Cloud Object Storage
aws-sdk-s3(official AWS SDK, v1.123)cloud-storageorgoogle-cloud-storageazure_storage_blobsaws-sdk-s3(S3-compatible)aws-sdk-s3(S3-compatible)aws-sdk-s3(S3-compatible)5. Data Lakehouse Formats
deltalake(Rust-native, delta-rs project)hudi(Rust-native, apache/hudi-rs)6. Data Warehouses
reqwest(Snowpipe REST API); or S3 stagingclickhouse;klickhouse(native protocol)aws-sdk-s3+sqlx(PG wire protocol)gcp-bigquery-clientdeltalakecrate or REST APItiberius(TDS/SQL Server protocol)reqwest(Stream Load HTTP API)reqwest(Stream Load HTTP API)reqwestduckdb(official Rust bindings)7. Time-Series Databases
reqwest(HTTP API; no mature Rust client)remote_writeAPI (protobuf/snappy over HTTP) for pushing metrics from Iggy.prost+reqwestreqwest(REST API)remote_write)sqlx8. Graph Databases
neo4rs(async Bolt protocol client)arangorsreqwest(Neptune REST API)reqwest9. Message Queues & Streaming Bridges
rumqttc(237K downloads/month, async, MQTT v5)lapin(async AMQP 0.9.1)rdkafka(18.6M sys downloads)aws-sdk-kinesis(official AWS SDK)aws-sdk-sqsgoogle-cloud-pubsubpulsar(async Rust client)async-nats(official async client)azure_messaging_servicebus; orfe2o3-amqprdkafka(Kafka-compatible protocol)10. Caches & Key-Value Stores
redis(official);fred(high-level async)rediscrate (Redis-compatible API)memcacheaerospikereqwestetcd-client(async gRPC)11. File Systems & Formats
tokio::fs;notify(file watcher)suppaftp;russh-sftphdfs-nativeparquet(Apache Arrow, very mature)apache-avro(official)bson(official, 14.2M downloads)csv(BurntSushi's, very popular)rmp-serde12. HTTP & API Protocols
reqwest(client);axum(server)tonic(gRPC);prost(protobuf)graphql_clienttokio-tungstenite13. Observability & Monitoring
opentelemetry;opentelemetry-otlpreqwestreqwestprost+reqwest(protobuf/snappy over HTTP)reqwestopentelemetry-otlp(Jaeger now prefers OTLP)reqwestor OTLP connector14. AI/ML & Vector Databases
qdrant-client(official gRPC client)reqwest(REST API)pinecone-sdkorreqwestmilvus-sdk; or gRPC viatonicreqwest(REST API)reqwest(REST API)15. SaaS & Enterprise
reqwest(REST/Bulk API)reqwest(REST API)reqwest(Webhook HTTP POST)lettrereqwest(webhooks)reqwest(REST API)16. Stream Processing Integration
17. Workflow & Orchestration
reqwest(REST API)temporal-sdk-core(official Rust SDK)reqwest(REST API / webhooks)reqwest(REST API)18. IoT Protocols
rumqttccoap;coap-liteopcua(pure Rust client/server)tokio-modbus(async RTU/TCP)coapcrate (LwM2M builds on CoAP)Codec / Serialization Format Status
apache-avrocratebsoncrate (14.2M downloads)parquetcratecsvcratermp-serdecrateSummary Counts
How to Contribute
Sink,Source,Transform) in the SDK cratecore/connectors/All reactions