v24.2.3

Features

• Schema Registry: added support for the "verbose" query parameter on the schema compatibility checker endpoint by @pgellert in #23011

Bug Fixes

• Avoids a crash in Redpanda's layer that could be triggered by empty offset commits. by @andrwng in #22830
• Fix New segment does not line up with previous segment error for compacted topics. by @Lazin in #22924
• Fixes a bug in which empty offset commit requests could result in a crash. by @andrwng in #22895
• Fixes timequeries not being served to remote read replica topics with remote.read=false. by @WillemKauf in #22885
• Made EndTxnRequest fully idempotent by @mmaslankaprv in #22941
• #22936 Fixes a bug in metrics reporting where label values containing special characters (", \, \n) could lead to a malformed metrics response. by @pgellert in #22939
• fixes problem of a node being reported as alive after its node_id has changed by @mmaslankaprv in #22835
• fixes rare assertion that might have been triggered in the raft::heartbeat_manager by @mmaslankaprv in #23056

Improvements

• Added support for Metadata API v8 by @michael-redpanda in #23039
• Added vectorized_kafka_rpc_connections_rejected_rate_limit metric which counts incoming Kafka connections rejected due to the connection rate limit (if set), analogously to the existing vectorized_kafka_rpc_connections_rejected metric which counts rejected connections due to the hitting the open connection limit. by @travisdowns in #22816
• Allows DeleteRecords requests from Kafka clients or rpk topic trim-prefix to be called with truncation_offset <= start_offset without returning an error. The request is instead treated as a no-op. by @WillemKauf in #22945
• Allows the self-test to be completely compatible with a mixed version cluster, in the case of a rolling upgrade. by @WillemKauf in #22922
• Reduce the memory overhead of many small segments. by @rockwotj in #22966
• Return core assignments from health report in /v1/cluster/partitions admin API output. by @ztlpn in #22773
• Schema Registry: 5 new compatibility checks are added for protobuf (ONEOF_FIELD_REMOVED, MULTIPLE_FIELDS_MOVED_TO_ONEOF, REQUIRED_FIELD_{ADDED,REMOVED}, FIELD_NAMED_TYPE_CHANGED, MESSAGE_REMOVED) by @pgellert in #22858
• fixed spurious warning log entries that may be altering users by @mmaslankaprv in #22771
• optimized memory usage with large core count nodes by @mmaslankaprv in #22982

Full Changelog: v24.2.2...v24.2.3

v24.1.15

Bug Fixes

• Avoids a crash in Redpanda's layer that could be triggered by empty offset commits. by @andrwng in #22829
• Fix New segment does not line up with previous segment error for compacted topics. by @Lazin in #22926
• Fixes a bug in which empty offset commit requests could result in a crash. by @andrwng in #22896
• #20135 Fix an issue where transforms would miscalculate their initial start offset, leading to consuming the whole input topic. by @oleiman in #22736
• fixes problem of a node being reported as alive after its node_id has changed by @mmaslankaprv in #22910
• PR #22778 [v24.1.x] cluster: add serde_fields() to start_test_request by @WillemKauf
• PR #22781 [backport] [v24.1.x] miscellaneous idempotency fixes #22687 by @bharathv

Improvements

• Allows DeleteRecords requests from Kafka clients or rpk topic trim-prefix to be called with truncation_offset <= start_offset without returning an error. The request is instead treated as a no-op. by @WillemKauf in #22944
• Made leadership changes related with reconfiguration faster and less disruptive by @mmaslankaprv in #22769
• Reduce the memory overhead of many small segments. by @rockwotj in #22967
• Schema Registry: 5 new compatibility checks are added for protobuf (ONEOF_FIELD_REMOVED, MULTIPLE_FIELDS_MOVED_TO_ONEOF, REQUIRED_FIELD_{ADDED,REMOVED}, FIELD_NAMED_TYPE_CHANGED, MESSAGE_REMOVED) by @pgellert in #22859
• fixed spurious warning log entries that may be altering users by @mmaslankaprv in #22770
• PR #21620 [v24.1.x] [CORE-5632] cluster: Reset cloud storage metric in the cluster::partition::remove_persistent_state by @Lazin
• PR #22684 [v24.1.x] Logging improvements in cluster members subsystem by @mmaslankaprv
• PR #22703 [v24.1.x] schema_registry: Sync Swagger with docs by @BenPope
• PR #22759 [v24.1.x] Disable log cleanup policy for controller log by @mmaslankaprv
• PR #22768 [v24.1.x] CORE-5686 Decouple different raft group shutdown sequences by @mmaslankaprv
• PR #22789 [v24.1.x] config: update docs for cloud_storage_azure_hierarchical_namespace_enabled by @nvartolomei
• PR #22795 [v24.1.x] CORE-6479 dt/random_node_ops: suppress method_not_found log by @pgellert
• PR #22887 [v24.1.x] cluster: fix partition::may_read_from_cloud() for remote read replicas (manual backport) by @WillemKauf
• PR #22943 [backport] [24.1.x] tx: do not throw USE on expired transaction_ids by @bharathv
• PR #22947 [v24.1.x] [CORE-5542] s/segment_utils: pass abort source into compacted index reader by @mmaslankaprv
• PR #22959 [v24.1.x] c/members_table: fixed typo in applying remove node command by @mmaslankaprv
• PR #22960 [v24.1.x] k/group: do not replicate empty offsets commit batch by @mmaslankaprv
• PR #22987 [v24.1.x] kafka: deal with truncation_offset < 0 cases in prefix_truncate() by @WillemKauf
• PR #22993 [backport] [v24.1.x] rm_stm: fix race during leadership transfer by @bharathv

Full Changelog: v24.1.14...v24.1.15

v23.3.20

Bug Fixes

• fixes problem of a node being reported as alive after its node_id has changed by @mmaslankaprv in #22886
• PR #21587 [v23.3.x] cloud storage: Fix deleted_count in clean_up_at_start by @jcipar
• PR #22888 [v23.3.x] cluster: fix partition::may_read_from_cloud() for remote read replicas (manual backport) by @WillemKauf

Improvements

• Schema Registry: 5 new compatibility checks are added for protobuf (ONEOF_FIELD_REMOVED, MULTIPLE_FIELDS_MOVED_TO_ONEOF, REQUIRED_FIELD_{ADDED,REMOVED}, FIELD_NAMED_TYPE_CHANGED, MESSAGE_REMOVED) by @pgellert in #22860

Full Changelog: v23.3.19...v23.3.20

v24.2.2

Features

• rpk: Add rpk registry mode to manage the schema registry mode. by @r-vasquez in #22697

Bug Fixes

• Schema Registry: Fix Swagger for POST /compatibility/subjects/{subject}/versions/{version} by @BenPope in #22646
• PR #22685 [v24.2.x] cloud_storage: minor fixes in topic_mount_handler by @WillemKauf
• PR #22692 [v24.2.x] tests/availability: fix resuming of suspended nodes by @bashtanov
• PR #22757 [backport] [v24.2.x] miscellaneous idempotency fixes #22687 by @bharathv
• PR #22760 [v24.2.x] Disable log cleanup policy for controller log by @mmaslankaprv

Improvements

• Allow rpk cluster self-test start to run, even in a cluster with mixed versions of redpanda (before and after cloudcheck addition in 24.2.x). by @WillemKauf in #22698
• #22649 Schema Registry: Remove spurious log entry: No syntax specified for the proto file by @BenPope in #22650
• rpk topic describe-storage can be used now with internal topics. by @r-vasquez in #22490
• PR #22655 [v24.2.x] Shard placement table: add metrics by @ztlpn
• PR #22658 [v24.2.x] [CORE-6750] schema registry exclusiveMinumum/exclusiveMaximum double support by @andijcr
• PR #22659 [v24.2.x] rpk: update cloudcheck self-test docs by @WillemKauf
• PR #22671 [v24.2.x] cluster: remove cloud storage self-test checks by @WillemKauf
• PR #22682 [v24.2.x] cloud_storage: add topic_mount_manifest (manual backport) by @WillemKauf
• PR #22704 [v24.2.x] schema_registry: Sync Swagger with docs by @BenPope
• PR #22751 [v24.2.x] gha/transform-sdk: pin and centralize language versions by @rockwotj

Full Changelog: v24.2.1...v24.2.2

v24.1.14

Features

• #22603 Schema Registry: Support normalize=true by @BenPope in #22634

Bug Fixes

• Fixes a bug in the purger, which when used with ABS may attempt to download a directory Blob as a manifest. This would completely block the purger from purging partitions for a deleted topic. by @WillemKauf in #22541
• Fixes a bug in which the abs_parse_impl used in xml_sax_parser would throw an exception when processing a List Blobs response with a <BlobPrefix> element within <Blobs>, but without any <Blob> elements. by @WillemKauf in #22539
• Fixes a rare race condition triggering an assert (crash) with idempotent producers. by @bharathv in #22738
• Kafka client used by audit system will now properly handle DNS lookup failure errors by @michael-redpanda in #22525
• Schema Registry: Fix Swagger for POST /compatibility/subjects/{subject}/versions/{version} by @BenPope in #22645
• PR #21589 [v24.1.x] cloud storage: Fix deleted_count in clean_up_at_start by @jcipar
• PR #22322 [v24.1.x] Increased tolerance in fast addition tests by @mmaslankaprv
• PR #22537 [v24.1.x] cloud_storage: add error_outcome::operation_not_supported by @WillemKauf
• PR #22694 [v24.1.x] tests/availability: fix resuming of suspended nodes by @bashtanov
• PR #22700 [v24.1.x] admin: remove throw on unknown self test type in admin_server (manual backport) by @WillemKauf

Improvements

• #22603 Schema Registry: Improve AVRO Normalization by @BenPope in #22634
• #22628 rpk debug bundle: include the result of uname -a by @JFlath in #22630
• #22651 Schema Registry: Remove spurious log entry: No syntax specified for the proto file by @BenPope in #22652

Full Changelog: v24.1.13...v24.1.14

v23.3.19

Features

• #22600 Schema Registry: Support normalize=true by @BenPope in #22635

Bug Fixes

• Fix timequery failing when requested timestamp is in the future and local log is empty. by @nvartolomei in #21321
• Fixes a bug in the purger, which when used with ABS may attempt to download a directory Blob as a manifest. This would completely block the purger from purging partitions for a deleted topic. by @WillemKauf in #22543
• Fixes a bug in which the abs_parse_impl used in xml_sax_parser would throw an exception when processing a List Blobs response with a <BlobPrefix> element within <Blobs>, but without any <Blob> elements. by @WillemKauf in #22540
• Kafka client used by audit system will now properly handle DNS lookup failure errors by @michael-redpanda in #22526
• Schema Registry: Fix Swagger for POST /compatibility/subjects/{subject}/versions/{version} by @BenPope in #22647
• #21253 Fixed a crash during raft snapshot application (prefix truncation of the log). A race condition between raft snapshot application and lagging state machine apply fibers caused the state machine offset to move backwards incorrectly. by @bharathv in #21255

Improvements

• Schema Registry: remove the schema from memory when the last subject version referencing it is deleted. by @pgellert in #21258
• #21296 HTTP Proxy: Avoid large allocations during JSON serde of requests and responses by @BenPope in #21469
• #21296 Schema Registry: Avoid large allocations during JSON serde of requests and responses by @BenPope in #21469
• #21467 Schema Registry: Avoid large contiguous allocations whilst handling schema. by @BenPope in #21557
• #22600 Schema Registry: Improve AVRO Normalization by @BenPope in #22635
• #22626 rpk debug bundle: include the result of uname -a by @JFlath in #22627
• #22653 Schema Registry: Remove spurious log entry: No syntax specified for the proto file by @BenPope in #22654
• rpk: --mechanism flag is now required to update users when using rpk security user update. by @r-vasquez in #21515
• PR #18319 [v23.3.x] offline_log_viewer: fix get_control_record_type by @bharathv
• PR #20747 [v23.3.x] Do not set transaction timeout in tests that do not need it by @mmaslankaprv
• PR #20826 [v23.3.x] cpu_profiler: prealloc result buffers by @StephanDollberg
• PR #21290 [v23.3.x] pandaproxy: fix spelling of operation by @andijcr
• PR #21363 [v23.3.x] tests: increase tolerance in local retention movement test by @mmaslankaprv
• PR #21377 [v23.3.x] rptest: tolerate leadership changes in chunk read tests by @nvartolomei
• PR #21464 [v23.3.x] Maintenance mode test fixes by @ztlpn
• PR #21526 [v23.3.x] producer_state: add request::set_error and make it idempotent by @bharathv
• PR #21568 Backport for "Merged Guard from concurrent elections" #21361 by @bashtanov
• PR #22318 [v23.3.x] Fix http client segfault in metrics_reporter by @ztlpn
• PR #22536 [v23.3.x] cloud_storage: add error_outcome::operation_not_supported by @WillemKauf

Full Changelog: v23.3.18...v23.3.19

v24.2.1

Deprecation notice

We recommend v24.2.2 or newer. For more context, see this Technical Service Bulletin.

New Features

• Redpanda is now compliant with FIPS 140. To put Redpanda into FIPS mode, you will need to install the redpanda-fips package and set the fips_mode node config to true. For more information about Redpanda's security features and it's FIPS compliance, please see our documentation. by @michael-redpanda in #17616
• rpk: Add support to JSON Schema to rpk, now supported in rpk registry and rpk topic produce/consume by @r-vasquez in #20820
• PR #19796 [CORE-3180] json schema registry validate input schema by @andijcr
• PR #19818 [CORE-4180] schema_registry/json: JSON Schema support by @BenPope
• Support JavaScript and TypeScript as languages for Data Transforms. by @rockwotj in #18078
• rpk: now you can manage client quotas using rpk cluster quotas. by @r-vasquez in #18711
• rpk: introduce a command to import cluster client quotas: rpk cluster quotas import. by @r-vasquez in #21311
• Adds new cluster config tls_min_version that allows users of Redpanda to specify the minimum version of TLS Redpanda will support. By default, the value is TLS v1.2. by @michael-redpanda in #21372
• Schema Registry: Support /mode endpoints for READONLY by @BenPope in #17952
• Schema Registry: Support subjectPrefix for GET /subjects by @BenPope in #20145
• Schema Registry: Support for deleted=true query parameter on POST /subjects/<subject>. by @BenPope in #18391
• #17418 rpk: ability to transfer partition leadership by @daisukebe in #18026
• #2183 Adds path-style addressing to S3 client requests. by @WillemKauf in #17806
  This option is configurable through rpk cluster config edit: by @WillemKauf in #17806
• Adds a cloud storage check as part of the cluster's self test. by @WillemKauf in #17586
• Allow decreasing core count if node-local core assignment is enabled. by @ztlpn in #20312
• If the node_local_core_assignment flag is enabled, Redpanda will try to maintain balanced distribution of partition replicas across cores. by @ztlpn in #19864
• KIP-518 (ListGroups v4): adds redpanda and rpk support for printing consumer group states with rpk group list and filtering for specific group states with --states. by @pgellert in #21347
  #7867 rpk container start: now starts a Redpanda Console container connected with the cluster. by @r-vasquez in #18066
• rpk container now has a set of flags to specify ports for node to start on. by @r-vasquez in #17908
• rpk: rpk cluster partitions list now supports filtering with broker IDs. by @daisukebe in #17945
• Adds a read distribution histogram to the internal metrics that can be visualized using Prometheus and Grafana: by @WillemKauf in #18745
  • vectorized_kafka_fetch_read_distribution_bucket
  • vectorized_kafka_fetch_read_distribution_count
  • vectorized_kafka_fetch_read_distribution_sum

Improvements

• Adds a new fetch_read_strategy option non_polling_with_debounce. This option will add a fixed delay specified by fetch_reads_debounce_timeout to the start of every fetch. by @ballard26 in #20744
• Adds configuration options to trigger cache trim before the cache reaches its maximum size. by @jcipar in #18756
  • cloud_storage_cache_trim_threshold_size
  • cloud_storage_cache_trim_threshold_objects
    These mirror the options for controlling maximum size: cloud_storage_cache_size and cloud_storage_cache_max_objects
• Re-adds the fetch_read_strategy cluster config property to select between polling and non-polling fetch implementations. Uses the non-polling fetch implementation by default. by @StephanDollberg in #18090
• Split cache into buckets using cloud_storage_cache_num_buckets configuration parameter. by @Lazin in #18762
• The new default behavior, if these are not set, is to trigger a trim when the cache is 100% full. by @jcipar in #18756
• #2183* * --cloud-backoff-ms uint, the backoff in milliseconds for a cloud storage request. by @WillemKauf in #17586
• --cloud-timeout-ms uint, the timeout in milliseconds for a cloud storage request. by * redpanda will now self configure its cloud_storage_url_style for use with the internal S3 Client on start-up if it is left unspecified. by @WillemKauf in #18107
• Add an optional TLS config field for CRL (Certificate Revocation List) by @oleiman in #18708
• Add more cases to the rpk disk self-test to better probe write performance at various IO depths, and at 16K block sizes. Return more information about the specifics of the test in the output. by @travisdowns in #20590
• Add output batch compression for Data Transforms (configurable per deployed transform) by @oleiman in #18514
• Add schema registry support to experimental Data Transforms C++ SDK by @oleiman in #21292
• Adds the ability to pause a deployed data transform w/o removing it from the system by @oleiman in #18236
• Adds the ability to start transform processing from an arbitrary offset on the input topic. by @oleiman in #19975
• Client qroup quotas now support dynamic configuration of the rate. by @BenPope in #18693
• Data transforms better respect overall system memory usage and go faster when more memory is available. by @rockwotj in #17810
• Enable controller log eviction - now the snapshotted part of the controller log will be deleted. by @ztlpn in #18836
• Golang Transforms SDK (Serde): Adds the ability to specify subject name strategy for output topics. Transforms can also provide a function to be called after the subject name is derived - e.g. to ensure that the subject is created before emitting records. by @oleiman in #19823
• HTTP Proxy: Avoid large allocations during JSON serde of requests and responses by @BenPope in #20827
• Implemented node-local core assignment - nodes decide themselves on which core to put partitions instead of using global assignments. This feature is enabled by default for new clusters. by @ztlpn in #18581
• Improve rpk cluster self-test status output for the cloud test. by @WillemKauf in #21353
• Improve cloud storage cache to prevent readers from being blocked during cache eviction. by @Lazin in #18056
• Improved logging around create_topics and alter_configs requests by @graphcareful in #18789
• It is now possible to use rpk group seek to set commits on a group that does not yet exist by @twmb in #18667
• Made electing a leader faster by @mmaslankaprv in #18394
• Made leadership changes related with reconfiguration faster and less disruptive by @mmaslankaprv in #19966
• Modify connection limit reached log line at WARN instead of INFO level by @graphcareful in #18773
• Run directory walking during cache trimming concurrently. On some deployments it was observed that it can take hours for 600K objects with busy reactor during which...

v24.1.13

Bug fixes

• PR #21565 [v24.1.x] Guard from concurrent elections by @bashtanov
• PR #21612 [v24.1.x] Fix http client segfault in metrics_reporter by @ztlpn
• PR #21613 [v24.1.x] cst/cache: fix in mem trim termination condition by @abhijat

Full Changelog: v24.1.12...v24.1.13

v24.1.12

Improvements

• rpk: --mechanism flag is now required to update users when using rpk security user update. by @r-vasquez in #21514

Bug fixes

• PR #21521 [v24.1.x] archival: keep locks until outcome is known by @nvartolomei
• PR #21524 [v24.1.x] producer_state: add request::set_error and make it idempotent by @bharathv
• PR #21570 [v24.1.x] cst/config: Disable trim carryover by @abhijat
• PR #21579 [v24.1.x] Fix http client segfault in metrics_reporter by @ztlpn

Full Changelog: v24.1.11...v24.1.12

v24.1.11

Bug Fixes

• #21421 Speeds up the shutdown time of partition replicas when compactions are happening. by @andrwng in #21422
• fixed handling of delayed snapshot requests that might lead to an assertion by @mmaslankaprv in #20148

Improvements

• PR #21383 [v24.1.x] Optimize replicated_partition::validate_fetch_offset by @ballard26
• PR #21434 [v24.1.x] cst/ducktape: Change assertion for deleted chunks by @abhijat
• PR #21471 [v24.1.x] [CORE-5672] crypto/bench: Only load context once per run by @michael-redpanda
• PR #21480 [v24.1.x] schema_registry: Remove extraneous log by @BenPope
• PR #21502 [v24.1.x] cst/cache: tracker based trim by @abhijat

Full Changelog: v24.1.10...v24.1.11