Repository navigation
Release v1.0.0
Description
The first stable release. Before it, the whole engine went through a production-readiness review: the core, the structural endpoints, every built-in endpoint and the plugin host. Its findings are in this release: 143 fixes and 83 behaviour changes, most of them from the review. New are endpoints for LLMs and vector search. Code written against 0.4 compiles unchanged, with two small exceptions; several defaults and error cases behave differently, so read Behaviour changes in the changelog before upgrading.
- What 1.0 promises
- The Rust API follows semantic versioning: the crate root and the modules
models,traits,errors,outcomes,extensions,plugin(the SDK),type_handler,publisher,routeandcheckpoint. endpoints,middleware,supportandtest_utilsare implementation and may change in a minor release. So may the major version of a re-exported driver crate (mq_bridge::sqlx,mq_bridge::rdkafka, ...).- A minor release may add a field to a configuration struct or a variant to a configuration or error enum, and raise the minimum Rust version. Write config structs with
..Default::default()and give amatchon these enums a_arm.
- The Rust API follows semantic versioning: the crate root and the modules
- Upgrading from 0.4
- No type, field, variant or path of the public API was renamed or removed.
- Several configuration structs gained a field (among them
ObjectStoreConfig,PostgresCdcConfig,WebSocketConfig,KafkaConfig,FileConfig,CsvConfigandRedisStreamsConfig). A struct literal that lists every field needs the new one or..Default::default(). Middleware::Filterholds aFilterMiddlewareinstead of aString. Code that builds the variant adds.into(); code that matches it readsfilter.expression. Config files are not affected.- The minimum Rust version is 1.89. The features
aws,sqlxandpostgres-cdc, and sofull, need 1.94.1;zeromq-omqneeds 1.93. The crate is built with edition 2024. mqb: the cargo featuremeilisearchis removed.mongodb: the queue mode is namedconsume: delete;consume: consumeris still read, but a config saved by 1.0 is not read by 0.4. The queue is read in insert order, and the consumer creates an index oninsert_orderat startup, which takes a moment on a large collection.http_bulkand its presets (meilisearch,typesense,elasticsearch,qdrant,openai_chat,openai_embeddings): a request fails after 300 s without a response byte whenrequest_timeout_msis unset.- Building
fullfrom source needs perl and make, for the new featurekafka-ssl.
- What the review fixed
- Messages lost or delivered twice. A nacked message is delivered again on
postgres_cdc,mongodbchange streams and snapshots, and theclickhouseandsqlxcursor sources; a later acked batch used to move the position past it. A batch that is dropped without a commit is read again (http_bulk,clickhouse,sqlx) or backed out (ibm_mq).deduplicationno longer lets a duplicate through when the sink takes longer than five seconds, and akeytemplate with a missing field no longer drops messages. A redelivered message keeps itsmessage_idonibm_mqandaws(SQS), andmongodbchange capture stores its position before the first change, socapture_newno longer loses a change made between two runs. - TLS that was asked for and not used.
amqpandnatswithtls.required: trueno longer connect in the clear, awss://WebSocket output and anmqtts://URL connect with TLS, andamqpusestls.ca_file.tls.requiredwith a plain-text URL is an error onclickhouseandgrpc.kafkawith TLS or SASL works in the prebuiltmqb, the Python wheels and the Node packages; it failed at start withOpenSSL not available at build time. - Secrets in output.
Debugof a config no longer prints passwords, tokens and keys. A store URL in an error no longer shows its password, and credentials in a URL query string are blanked in logs and errors.mqblogs a URI with a user name and no password, the NATS form for a token, as***@host. - Hangs and panics. A route with
concurrencyabove 1 no longer hangs when every worker fails at once. Afilterordeduplicationthat drops every message no longer stalls the source. A NATS metadata entry that cannot be a header, a very smalllimiterrate and a missing rustls provider ingrpcno longer panic. - Errors at start instead of at runtime.
retryanddlqon an input, route options that cannot run, arequestwhose target can never reply and an ambiguous config file are refused at start. They used to log a warning or fail later. - One bad message fails alone. A row ClickHouse refuses, a statement
sqlxrejects, a document MongoDB refuses, a Kafka record over the size limit and a NATS message over the server's payload limit fail only their own message, as non-retryable, instead of the whole batch.
- Messages lost or delivered twice. A nacked message is delivered again on
- New features
- LLM calls in a pipeline:
openai_chatandopenai_embeddingswork with any API that speaks the OpenAI protocol (OpenAI, Ollama, vLLM and similar). Use them as an output or under alookupto classify, summarise or embed each message. See LLM calls in a pipeline. - Several messages in one LLM request:
messages_per_requestonopenai_chatsends that many messages as numbered inputs of one completion and hands each message its own answer, so the system prompt is paid for once per request. Answers are matched by number. A response that cannot be matched, or a request the API refuses for its content, is asked again in halves until the message at fault stands alone. Off by default. It was run against Ollama, not against OpenAI itself. qdrantendpoint: upserts points, deletes them by id and reads a collection. See Keep a vector index in sync with Postgres.qdrant,openai_chat,openai_embeddings,typesense,elasticsearchandmeilisearchare endpoint types of their own, with a URI form formqb(for exampleqdrant://host:6333/<collection>).http_bulk:query.per_message,query.split_on_reject,upsert.split_on_rejectandread.cursor.end_when_missing.websocket: atlsblock,basic_authandmax_connections.redis_streams: atlsblock.filesink:fsync(off,batchorperiodic).kafka:create_topic: false.postgres_cdc:split_key_change.object_store:on_decode_error.- Prebuilt C library: each release carries an
mq-bridge-c-<target>archive (shared library and header, every endpoint) for Linux x86_64, macOS arm64 and Windows x86_64. set_route_display_name(route, display)gives a route that runs under a generated name a readable one in logs, errors and DLQ metadata.CanonicalMessage::get_metadata,set_metadataandremove_metadata, and constructors for config structs withoutDefault.- docs/CAPABILITIES.md lists for every endpoint what it can do as input and output, whether it replies, and how it is secured.
- LLM calls in a pipeline:
The full list is in the changelog.
What's Changed
- Core production readiness review & fixes by @marcomq in #150
- Production readiness audit for structural endpoints by @marcomq in #151
- updated md files by @marcomq in #152
- production readiness fixes by @marcomq in #153
Full Changelog: v0.4.20...v1.0.0