YTsaurus Server 25.4.0

Overview

This document summarizes the key changes, improvements, features, and bug fixes introduced in YTsaurus 25.4.0. Thank you to all contributors and maintainers for their work.

YTsaurus Server 25.4.0 is supported since 0.27.0 version of the Kubernetes operator.

Significant changes

• Row-Level Security (RLS) — end-to-end support. ACEs now accept an expression, restricting access to individual rows; full_read is extended row-wise (covers copy/concatenate/remote_copy). Wired through master, proxy read_table and controller agent in operations; CHYT 2.18+ already supports it, SPYT 2.9.0 is required and YQL/QT — 0.4.1.

• Bulk-insert under user transaction — now general use. Introduced as experimental in 25.3 (the prerequisite for REPLACE INTO //dynamic/table in YQL), it is promoted to general availability in 25.4 — safe to use in production workloads.

• Authorized node expiration. Master now remembers who set expiration_time/expiration_timeout and, with enable_authorized_expiration on, runs the removal under that user. Prevents accidental cross-account deletions of shared subtrees and gives an auditable lifecycle for auto-expiring nodes.

• Distributed write protocol for files. Files can now be written from many client processes in parallel, the same way tables already could. Removes the single-writer bottleneck for large artifacts and enables fan-out uploads from operations and pipelines.

• transfer_bundle_resources command. Atomically moves resource quotas (tablet count, tablet static memory) between two tablet cell bundles in a single mutation, making bundle rebalancing safe for automation and the bundle controller.

• Heavy hitters for lookup queries. Dynamic tables expose raw and data-weighted heavy hitters for lookups, giving operators a first-class way to find the keys behind cache pressure, hot tablets and skewed read patterns.

• check_operation_permission API. Lets users and orchestrators ask the scheduler whether a given user has a given permission on an operation, mirroring check_permission for Cypress objects.

• Chunk filtration via input_query in path attributes. Filters chunks based on a query directly in path attributes — useful for read optimizations, a12fb8b.

• Compressed-data-size constraints for sorted operations. Adds primary compressed-data-size limits and a consider_only_primary_size job-sizing option that accounts only for the primary size when slicing jobs, 1057b82.

• CPU overload handling on data node. Integrates CPU overload handling into the data node; adds dynamic adjustment of pending-disk-read / pending-disk-write memory limits and the maximum session count, 5a4fe1f.

• YTQL syntax v3 — expressivity upgrades. Several long-standing limitations are lifted:

  • Expressions allowed in IN and BETWEEN.
  • Integer index in GROUP BY / GROUP BY KEY (GROUP BY 1, 2).
  • New string functions make_ngrams and split.
  • Reads from tablet followers in SELECT queries.
  • Fetch join executed on proxy.
  • Array join with lists of composite and any values.

Breaking changes

• List nodes have been completely removed. Loading a snapshot that contains list nodes will crash master-server. To ensure that no list nodes exist before the update, the following steps can be taken:

1. If alert_on_list_node_load option in the master dynamic config is set to true, then validation should have produced an alert when loading from snapshot in 25.2 and a crash in 25.3. If no alerts or crashes were observed, then it should be safe to update.
2. Otherwise, it is advised to use the dry-run option --validate-snapshot with the master-server binary version 25.4 to validate that the snapshot can be loaded before updating, 0abb6d3.

• Pool tree config cleanup. Removed enable_scheduling_tags (the feature it disabled is no longer relevant) and max_share_ratio from pool config (it was used to limit guarantees pre-vector-HDRF), d807f21.

• Deprecating list_queue_consumer_registrations. New disallow_list_all_registrations option in queue consumer registration manager config; this will become the default in subsequent releases after a migration phase, 3a49cef.

Full changelog

Scheduler and GPU

New Features & Changes:

• Introduce check_operation_permission API to verify a user's permission on an operation, 05236de.
• Add allow_children_guarantees option for scheduler pools to forbid child pools from having guarantees, 8b93622.
• Introduce list-job-traces API to retrieve a job's GPU traces, d495b01.
• Allow scheduling_tag_filter for operations spanning several pool trees, 5c17ea5.
• Increase default allocation preemption timeout to 30 seconds, f3d1c5a.
• Alert when an operation's spec is too large, 9067544.

Fixes & Optimizations:

• Removed enable_scheduling_tags from pool tree config and max_share_ratio from pool config (both obsolete), d807f21.
• Removed obsolete checks that blocked preemption when an ancestor pool was starving, 721def2.
• Allow resource limits overcommit during preemptive scheduling to mitigate resource distribution issues, 273bbc6.

Queue Agent

New Features & Changes:

• Add a profiling metric exposing whether a queue agent instance is banned, so external systems can alert on instances banned for too long, 8095e03.
• Server-side support for the queue_agent_controller_liveness Odin check in multi-cluster environments, c960ce5.

Fixes & Optimizations:

Proxy

New Features & Changes:

• New disallow_list_all_registrations option in queue consumer registration manager config — a transitional step toward removing list_queue_consumer_registrations, 3a49cef.
• Support LinkNode with attributes via RPC, 570c83f.
• Support RLS (Row-Level Security) in the read_table command, 9ee66dd.
• Register chyt_http(s)_server in Cypress, 69595e0.

Fixes & Optimizations:

• Fix datetime mapping to protobuf integral types, c04424a.
• HTTP server now starts listening only after Start() is called, 8af9288.

Dynamic Tables

New Features & Changes:

• New transfer_bundle_resources command — atomically transfers resources between two tablet cell bundles, e2479e2.
• A configurable share of compaction tasks is reserved for the most starving {table, compaction reason} pairs, 5f2ad34.
• Heavy hitters and data-weighted heavy hitters for lookup queries, 75e474e.
• YTQL make_ngrams function, c848ed3.
• YTQL split function — splits a string by a delimiter into a list, d355e0b.
• Expressions in IN and BETWEEN (syntax v3), 128294e.
• Integer index in GROUP BY / GROUP BY KEY (syntax v3), 359dfb3.
• Aggregating columns are now allowed for unfolding indices; if the unfolded value contains repeats (e.g. [1;2;3;1]), table values are aggregated, 8b5954a.
• Reads from followers enabled in select queries, f8c5fde.
• Proxy subqueries supported in explain-query, 0f2799f.
• Fetch join on proxy, 3c3515c.
• Array join with lists of composite and any values, be8bc8c.

Fixes & Optimizations:

• Sample keys are now tracked under the tablet_background category, 009a33d.
• Query builder produces more balanced ASTs for predicates, 913c370.
• Avoid excessive memory footprint from memory-tracking in-memory chunks within a memory pool, 8f7f624.
• Fix compaction timestamp digest for min_data_versions=1, 7a2a576.
• Improved performance of aggregate column evaluation, 7ffe4d2.
• Slimmer QL codegen cache, ba32890.
• Fix crashes from incorrect ids produced by range inference on is_prefix(...) expressions, 9c64c4a.
• Add missing /overload_controller prefix to profiling sensors, 01f2324.
• Fix CASE operator ignoring the use_canonical_null_relations option, 5b48f8b.
• Fix array join when unpacking null lists, 3f8ca76.
• Minor use-after-free fix, 2e99e6d.
• Fix segfault in YTQL codegen for queries with nested subqueries under certain conditions, 63fcaa9.
• Add missing casts for the CAST operator in YTQL, 910cf8b.

MapReduce

New Features & Changes:

• Job API in job proxy — exposed to user jobs over Unix domain sockets. Initial endpoint OnProgressSaved bumps a job's preemptible-progress start time, 50011e3.
• Use a code-generated comparator in partition reduce when the schema is known, for higher performance, 984cee5.
• Exec node chunk cache renamed to artifact cache; metrics moved to yt/exec_node/artifact_cache, 70fb3e0.
• Chunk filtration via input_query in path attributes — useful for read optimizations, a12fb8b.
• Compressed-data-size constraints for sorted operations, including primary compressed-data-size limits and a consider_only_primary_size job-sizing option, 1057b82.

Fixes & Optimizations:

• Fix readers in MR operations not accounting for nested tables, ea0179f.

Master Server

New Features & Changes:

• Permission check during removal of expired nodes (authorized expiration). New attributes expiration_time_user, expiration_timeout_user, expiration_time_last_reset_time, expiration_timeout_last_reset_time and config entries enable_authorized_expiration, expiration_attempt_limit, expiration_persist_period, 5bc8e76.
• Skip node disposal when a node was restarted, 8616a6b.
• List nodes have been completely removed. Loading a snapshot containing list nodes will crash master-server, 0abb6d3.
• "Lie" about full_read permission in authorization errors. When a user lacks even basic read, the error mentions only basic read, so they don't unnecessarily request full_read, 655e1a4.
• Removed duplicate alerts about misconfigured roles on master cells in multicell manager — alerts now only fire on the primary master, 1e5aeff.

Fixes & Optimizations:

• Fix requests like create //a/b& previously failing with "no such child 'b'", 28cecce.
• Secondary index objects now receive a proper ACD (matching the indexed table), 57f8185.

Node

New Features & Changes:

• Option to skip waiting for preceding blocks in PutBlocks requests, 5b23e49.
• Metrics for huge-pages I/O operations, 320f828.
• Integrate CPU overload handling into data node; allow dynamic adjustment of pending-disk-read / pending-disk-write memory limits and the maximum session count, 5a4fe1f.

Fixes & Optimizations:

• Fix race in node lease handling: when a node is banned and its lease tx is aborted while a registration is in flight, the new registration could reuse the already-aborted lease, fe7b03f.

Other

• Make TAsyncExpiringCache sharded, dd06cc5.
• Time at which a memory profile was collected and its collection duration are added to the profile dump, 9aa42a7.
• Fix extra trace contexts when reading columnar chunks outside the schemaless multi-chunk reader context, f0f0103.
• Fix races on memory guard (was not thread-safe, unlike memory tracker), 640e137.