Skip to content

Release v1.0.0

Choose a tag to compare

@marcomq marcomq released this 12 Oct 21:01
· 268 commits to main since this release
v1.0.0
a77c2f0

Description

The first stable release. Before it, the whole engine went through a production-readiness review: the core, the structural endpoints, every built-in endpoint and the plugin host. Its findings are in this release: 143 fixes and 83 behaviour changes, most of them from the review. New are endpoints for LLMs and vector search. Code written against 0.4 compiles unchanged, with two small exceptions; several defaults and error cases behave differently, so read Behaviour changes in the changelog before upgrading.

  • What 1.0 promises
    • The Rust API follows semantic versioning: the crate root and the modules models, traits, errors, outcomes, extensions, plugin (the SDK), type_handler, publisher, route and checkpoint.
    • endpoints, middleware, support and test_utils are implementation and may change in a minor release. So may the major version of a re-exported driver crate (mq_bridge::sqlx, mq_bridge::rdkafka, ...).
    • A minor release may add a field to a configuration struct or a variant to a configuration or error enum, and raise the minimum Rust version. Write config structs with ..Default::default() and give a match on these enums a _ arm.
  • Upgrading from 0.4
    • No type, field, variant or path of the public API was renamed or removed.
    • Several configuration structs gained a field (among them ObjectStoreConfig, PostgresCdcConfig, WebSocketConfig, KafkaConfig, FileConfig, CsvConfig and RedisStreamsConfig). A struct literal that lists every field needs the new one or ..Default::default().
    • Middleware::Filter holds a FilterMiddleware instead of a String. Code that builds the variant adds .into(); code that matches it reads filter.expression. Config files are not affected.
    • The minimum Rust version is 1.89. The features aws, sqlx and postgres-cdc, and so full, need 1.94.1; zeromq-omq needs 1.93. The crate is built with edition 2024.
    • mqb: the cargo feature meilisearch is removed.
    • mongodb: the queue mode is named consume: delete; consume: consumer is still read, but a config saved by 1.0 is not read by 0.4. The queue is read in insert order, and the consumer creates an index on insert_order at startup, which takes a moment on a large collection.
    • http_bulk and its presets (meilisearch, typesense, elasticsearch, qdrant, openai_chat, openai_embeddings): a request fails after 300 s without a response byte when request_timeout_ms is unset.
    • Building full from source needs perl and make, for the new feature kafka-ssl.
  • What the review fixed
    • Messages lost or delivered twice. A nacked message is delivered again on postgres_cdc, mongodb change streams and snapshots, and the clickhouse and sqlx cursor sources; a later acked batch used to move the position past it. A batch that is dropped without a commit is read again (http_bulk, clickhouse, sqlx) or backed out (ibm_mq). deduplication no longer lets a duplicate through when the sink takes longer than five seconds, and a key template with a missing field no longer drops messages. A redelivered message keeps its message_id on ibm_mq and aws (SQS), and mongodb change capture stores its position before the first change, so capture_new no longer loses a change made between two runs.
    • TLS that was asked for and not used. amqp and nats with tls.required: true no longer connect in the clear, a wss:// WebSocket output and an mqtts:// URL connect with TLS, and amqp uses tls.ca_file. tls.required with a plain-text URL is an error on clickhouse and grpc. kafka with TLS or SASL works in the prebuilt mqb, the Python wheels and the Node packages; it failed at start with OpenSSL not available at build time.
    • Secrets in output. Debug of a config no longer prints passwords, tokens and keys. A store URL in an error no longer shows its password, and credentials in a URL query string are blanked in logs and errors. mqb logs a URI with a user name and no password, the NATS form for a token, as ***@host.
    • Hangs and panics. A route with concurrency above 1 no longer hangs when every worker fails at once. A filter or deduplication that drops every message no longer stalls the source. A NATS metadata entry that cannot be a header, a very small limiter rate and a missing rustls provider in grpc no longer panic.
    • Errors at start instead of at runtime. retry and dlq on an input, route options that cannot run, a request whose target can never reply and an ambiguous config file are refused at start. They used to log a warning or fail later.
    • One bad message fails alone. A row ClickHouse refuses, a statement sqlx rejects, a document MongoDB refuses, a Kafka record over the size limit and a NATS message over the server's payload limit fail only their own message, as non-retryable, instead of the whole batch.
  • New features
    • LLM calls in a pipeline: openai_chat and openai_embeddings work with any API that speaks the OpenAI protocol (OpenAI, Ollama, vLLM and similar). Use them as an output or under a lookup to classify, summarise or embed each message. See LLM calls in a pipeline.
    • Several messages in one LLM request: messages_per_request on openai_chat sends that many messages as numbered inputs of one completion and hands each message its own answer, so the system prompt is paid for once per request. Answers are matched by number. A response that cannot be matched, or a request the API refuses for its content, is asked again in halves until the message at fault stands alone. Off by default. It was run against Ollama, not against OpenAI itself.
    • qdrant endpoint: upserts points, deletes them by id and reads a collection. See Keep a vector index in sync with Postgres.
    • qdrant, openai_chat, openai_embeddings, typesense, elasticsearch and meilisearch are endpoint types of their own, with a URI form for mqb (for example qdrant://host:6333/<collection>).
    • http_bulk: query.per_message, query.split_on_reject, upsert.split_on_reject and read.cursor.end_when_missing.
    • websocket: a tls block, basic_auth and max_connections. redis_streams: a tls block.
    • file sink: fsync (off, batch or periodic). kafka: create_topic: false. postgres_cdc: split_key_change. object_store: on_decode_error.
    • Prebuilt C library: each release carries an mq-bridge-c-<target> archive (shared library and header, every endpoint) for Linux x86_64, macOS arm64 and Windows x86_64.
    • set_route_display_name(route, display) gives a route that runs under a generated name a readable one in logs, errors and DLQ metadata.
    • CanonicalMessage::get_metadata, set_metadata and remove_metadata, and constructors for config structs without Default.
    • docs/CAPABILITIES.md lists for every endpoint what it can do as input and output, whether it replies, and how it is secured.

The full list is in the changelog.

What's Changed

Full Changelog: v0.4.20...v1.0.0