Skip to content

YTsaurus 25.2.0 release notes draft

Jump to bottom
H0plessFerret edited this page Sep 5, 2025 · 15 revisions

YTsaurus 25.2.0 release notes:

To install YTsaurus Server 25.2.0 update the k8s-operator to version 0.22.0. (TODO: поправить ссылку и версию)

Significant changes

We are continuing the deprecation of list_node. Master-servers will now produce an alert after loading a snapshot if it contains a list node. This behaviour can be turned off using alert_on_list_node_load option. Please consider switching to other types and removing or replacing all remaining list nodes. Failure to do so will result in master-server not starting in the next major update. With this release, we've bundled the script that should help you migrate in the vast majority of cases. You can find it at yt/yt/scripts/master/replace_list_nodes. We've published a blog post explaining our reasoning for deprecating this type and suggesting other migration methods. !!(TODO: Add a link to the blogpost!)!!

Full changelog

Scheduler and GPU

New Features & Changes:

  • Add validation of exe node resource limits. If a node does not meet configured limit scheduler raises an alert, c4b5dcd.

  • The scheduler event log now includes the amount of accumulated fair resource usage and the accumulated usage deficit, along with the previously collected resource usage, 6f1dd5e.

  • Allow dot by default in pool name validation regexp, 93cf17a.

  • Added type, user and title to operation orchid (i.e. //sys/scheduler/orchid/scheduler/scheduling_info_per_pool_tree/default/fair_share_info/operations/<op_id>/type), ffda123.

  • Add alert for unrecognized options in pool tree config, 9770034.

  • Add option to scheduler config and operation spec that fails operation if specified pools don't exist, 30eebf3.

  • Add option to allow starting gang operations only in FIFO pools, edcac9a.

  • Add operations permission validation via access control objects, bd64281.

Fixes & Optimizations:

  • Remove legacy method to configure pool trees with explicit attributes, 7573bd6. Other:

Queue Agent

New Features & Changes:

  • Add enable_verbose_logging option in the dynamic config which enables verbose logging for specific objects from verbose_logging_objects, 67efedf.
  • Add flag to select queue export implementation, 29f63ac.
  • Add sum aggregation for lag metrics for consumer partitions, e37cd27.
  • Take queue exports into account when trimming for replicated table and chaos replicated table queues, c016ca9.

Fixes & Optimizations:

  • Fix init_queue_agent_state in case directory already exists (occurs in k8s operator), 8aca0e2.
  • Fix write_data_weight_rate for empty partitions, 3e1dd6d.
  • Change cypress synchronizer logic such that all tables are taken into account, e.g. replicated_table_mapping table rows are checked to be correct, 8e4d941.
  • Add retry backoffs to slow down misconfigured exports, 619c7dd.
  • Prevent catastrophic split-brain between queue agent instances leading to orchid redirect loop by limiting the number of redirection and add retries, 39b014d.
  • Previously in case queue was re-created, and export directory was re-used without clearing export progress, queue agent would crash, because export progress would advance backwards. The behavior is fixed in this commit, 133b109.
  • Fix potential data loss in case of multiple exports per queue. Due to incorrect merging of queue export progresses, f51f9fa.

Proxy

New Features & Changes:

  • Support MutationId in CreateQueueProducerSession, 591d500.
  • Add the ability to configure a user limit for a specific proxy role, 14d49a0.
  • Extend impersonation functionality in HTTP protocol. It is now allowed for all unbanned superusers, 9eee43d.
  • Add memory tracker for http user traffic, 3951df4.
  • Allow sending/reading values larger than 16MB to RPC proxy via wire protocol to/from methods dealing with static tables, df8fb64.
  • Add to_lower and to_upper option to OAuth login_transformations, 0245614.
  • Support native date types in arrow parser, ecef747.
  • Start measuring the memory usage of write-table queries, 2f2fb5d.
  • Adding the ability to track memory and drop requests for specific users, 5685b25.
  • Support signature generation and validation via Cypress public keys, 12c8532.

Fixes & Optimizations:

  • Fix heap use after free in arrow encoder, 7bb2920.
  • Optimizations in concatenate method, 0f943c5.
  • Changed caching options in config of CypressUserManager for OAuthAuthenticator. Migrate to options compatible with AsyncExpiringCache ("expire_after_*_time"). Older options ("cache_ttl", "optimistic_cache_ttl") are deprecated and will be removed in future versions, 1ecabbc.
  • Fix for CVE-2023-33460: Memory leak in yajl 2.1.0 with use of yajl_tree_parse function. f7b9064
  • Fix passing mutating options in CreateQueueProducerSession in RPC proxy, a706adb.
  • Fix possible deadlock in the chunk meta cache, 7c68dbe.
  • Fixing the calculation of state_counts and type_counts in list_jobs, 84d7713.
  • Set attribute treat_as_queue_producer=%true during queue_producer creation, 88eac20.
  • Send the cumulative CPU metric during the request, 0a1b5fc.
  • Fixing the reading of tables with date type columns in arrow format, 1a23993.
  • Fix memory tracking per user, 35bbf61.
  • Use compression thread pool for compression of streams in HTTP proxy, 4679206.
  • Support JSON in arrow writer, 9f3a349.
  • If there is not enough memory to handle RPC responses, retryable error (Unavailable) will be returned (instead of non-retryable error *MemoryPressure), db04463.
  • Add the ability to read tables with different number of columns in chunk meta in arrow format, 6210035.
  • Make HTTP proxy work when master in read-only mode, ccb0228.
  • Introduce get_current_user method in rpc and http proxies, ab3f903.
  • Multiproxy mode in rpc-proxy. Client can use multiproxy of one cluster to operate other connected clusters, e0f98d7.
  • Support dynamic reconfiguration of signature subsystem in HTTP and RPC proxy, 029f6ce
  • Drop only heavy request during memory overcommit, 646071a.

Dynamic Tables

New Features & Changes:

  • Consider primary key prefix constrained by predicate in ORDER BY and use ordered execution if it is sufficient, f8dbc00.
  • Add bundle controller mode without spare node assignments, 51c6b39.
  • Allow background compaction and partition tasks to be executed within the two-level-fair-share thread pool, 55c2dfd.
  • cardinality_state and cardinality_merge functions for QL, a267e20.
  • Timestamp functions for arbitrary time zones are added to QL, 55178fe.
  • Support RegisterChunkReplicasOnStoresUpdate for ordered tables. Reduces the number of master requests required for reading flushed chunks via tablet node API, 155fe69.
  • Casts double->(unsigned) integer now avoid undefined behavior and function the same regardless of execution engine by clamping values and converting NaNs to zero, 8c492fa.
  • Optimized writes into tables with secondary indices under certain conditions, 4066880.
  • Enable decommission_through_extra_peers by default. This option significantly reduces downtime of tablet node maintenance, c34ff21.
  • Implemented array_agg function for QL, d656eec.
  • Hydra changelogs may now optionally be checksummed before being sent to storage layer, 6e4a84a.
  • Allow columns with type Any to be unfolded via indices. Their contents are checked in runtime, d0eb7ad.
  • Use key bounds instead of legacy keys in reshard implementation, de94ef1.
  • Smooth movement for tablets with hunks, d5357ea.
  • Add option to decrease serialization time by serializing transaction within each lock group in each row separately, fe90e97.
  • An optimization - SELECT queries with GROUP BY and JOIN group rows BEFORE joining when possible, 75e8e64.
  • Add a method to return freezing or unmounting tables back to mounted state, bbf1101.
  • Add ability to run bundle controller without instance allocator service, 46ad173.
  • Simple subqueries for YTQL, a7e0701.
  • Tablet nodes are now susceptible to Query memory limit, 13b82fe.
  • Introduce a protocol that allows writing to the tablet during smooth movement, e72b84d.
  • Allowed bundle controller to control Query memory limits, c6c6de4.
  • Add profiling counters in tablet nodes for pull_queue/pull_queue_consumer commands, [6aebfc1](https://github.com/ytsaurus/ytsaurus/commit/6aebfc1a758092864dae6b2517 7ec4339bc1c2db).
  • Select queries now properly choose a random in-sync replica even if candidates belong to the same cluster, efdf083.
  • Added total_qrouped_row_count to QL statistics, e37b81f.
  • Offload building replication card outside of main chaos slot thread. This should allow hosting large collocations of large tables inside single chaos cell, aad9921.
  • Add log drop tracker in overload controller, 8cd772d.
  • //sys/@config/security_manager/enable_tablet_resource_validation = %false, 4f78e41.
  • mirrored with @resource_limits. The usage of the latter is preferable, 27ed1ef.
  • Support remote copy for dynamic table with compression dictionaries, 0791aea.
  • Introduced "evaluatable schema" extension for secondary indices, which allows indexation of expressions, bf45155.
  • Enable hunks remote copy by default, 9f2c5f4.
  • Use lookup join when left subplan is selective, 03d25a9.

Fixes & Optimizations:

  • Fix per-category tablet dynamic memory accounting at followers, d147efd.
  • Signal handler stack enlarged. Enabled memory protection for signal handler stack to avoid memory corruption due to stack overflow during signal handler execution, 7ba96aa.
  • Fixed accounting of row cache memory, bcadf28.
  • Bundle controller releases spare nodes on gnd clusters correctly, 907525b.
  • Fix incorrect labels in bundle controller alerts, 4e778a7.
  • Bundle controller does not fail even if some bundles are configured incorrectly, 71da9d3.
  • Fixed to_any function - cast of EValueType::Composite to EValueType::Any now works as expected. Allowed some functions to work with both of these types, 20f22b7.
  • Fixed a bug involving unfolding secondary indices that lead to crashes when query predicate contained list_contains(expr) where expr was not a reference, 61b2b4e.
  • Fixed crashes in proxies when selecting from a table with a malformed computed column, 913edd3.
  • Fixed select_rows not waiting on locks when reading via lookup, 336ab90.
  • Fixed bad invoker choice when migrating to query thread pool, 8d93be7.
  • Misconfiguration of row cache no longer leads to OOM, 7db2519.
  • Fixed a bug with incorrect meta size estimation in scan format which caused overzealous chunk fragmentation during compaction, 11a59e3.
  • Fixed the bug where range inference produced mishapen keys and ranges, aa7597d.
  • Improved performance of timestamp_floor_week() function, 06a643c.
  • Fixed data corruption of floating point values in dynamic tables in scan format, 8fcbf50.
  • Just noticed pdgb doesn't work for replicated tables, cb64762.
  • Timestamp_floor_*_localtime functions did not respect non-Moscow timezones. That is now fixed, 07f30b0.
  • Use physical chunk count instead of logical one in ordered dynamic tablet chunk lists, 96c1a7f.
  • Eliminate a memory leak caused by cancelled selects, 75521e1.
  • Join-predicate is now used in range inference for subqueries that fetch data from dictionaries, d62d232.
  • Fixed bug leading to segfaults in table with nested columns, 45cb542.
  • bundle controller ignores faulty bundles to make progress, dc149b2.
  • Fixed incompatibility of AVG function in QL for the following case
  1. old tablet nodes
  2. new proxies
  3. zero non-null aggregated values, ef7e062.

MapReduce

New Features & Changes:

  • Support dynamic reconfiguration of signature subsystem in exec node, 029f6ce

  • When the job count is explicit and an unordered map is used, early completion of job input building based on the max_data_slice_count, max_data_weight_per_job, or max_compressed_data_size_per_job constraints was only possible for all jobs except the last one. For the last job, these constraints were not applied because the unordered chunk pool had to guarantee the specified job count and could not form more jobs. As a result, the last job could become significantly larger, and situations where the number of data slices exceeded max_data_slice_count were silently ignored. This PR skips early completion of job input building by the max_data_slices_per_job constraint when the job count is explicit, which helps to balance job sizes more evenly. Additionally, an alert is now raised if the number of data slices exceeds max_data_slice_count, correcting the previous silent behavior for the last job. Fixes:

  • Finally, this PR also skips the checks for max_data_weight_per_job and max_compressed_data_size_per_job when the job count is explicit, as there is no reason to fail the operation later when the last job is formed, 3bb3594.

  • Add option for lower bound of user job CPU limit, c4675d4.

  • Support ability to create separate volume to run GPU check. Add ability to configure GPU checks through operation options, 438023e.

  • Add support for NBD network disks to disk_request, a83d345.

  • Add various job splitter options to spec, dbcf4f2.

  • Remote operations are now allowed for everyone from any remote cluster, 5b24a57.

  • This commit unifies the logic for slicing by compressed data size and by data weight in the unordered chunk pool. Previously, max compressed data size per job was used to slice by compressed size, and its semantics differed from that of max data weight per job, 73ab4d8.

  • Support extra jobs for gang operations. Introduce gang ranks, d45f1f7.

  • Add new delivery fenced connection that works on vanilla linux kernel. It may be used for CPU intensive or GPU jobs to prevent job abort on interruption, d7861cd.

  • Add new delivery fenced connection that works on vanilla linux kernel. It may be used for CPU intensive or GPU jobs to prevent job abort on interruption, a2c5ef1.

  • Enhance input compressed size estimation for operations with columnar statistics, 68474e7.

  • CAs now fetch schema from external cells by default, b5932ab.

  • Exec node can not start multiple jobs in one allocation if told so by the spec option. Note that this feature must also be enabled in exec node's dynamic config to work, ee037ac.

  • Hostname in containers is now built using slot-{slot_index}.{exec_node_hostname} format, fe028ee.

  • RemoteCopy operations now set some system attributes on destination table, even if copy_attributes is set to false in spec. These attributes are copied: compression_codec, erasure_codec, optimize_for, 25be378.

Fixes & Optimizations:

  • Scheduler does not publish nested_transaction_ids in operation's attributes. Please, use input_transaction_ids instead, f230430.
  • Pass specific error instead of Job failed by external request , bc9b656.

Master Server

New Features & Changes:

  • Support bulk insert under a user transaction, 3fe8c73.
  • Make the default optimize_for configurable for static and dynamic tables, 9e5d4a8.
  • Introduce TCompactTableSchema, which holds the wire protobuf schema representation and consumes less memory than TTableSchema, 666861a.
  • Introduce a hard limit on the response size for read requests to //sys/chunks. Configure via virtual_chunk_map_read_result_limit. Previously, accessing this virtual map could crash the master server due to excessive creation of fibers and subsequent memory allocation. This is a temporary mitigation while we work on a better fix, 61eb1f5.
  • Change copy/move commands to require an explicit flag to dropp secondary indices; previously this was the default behaviour, 0668453.
  • Allow users to authenticate with their aliases. This is a preliminary compatibility step toward migrating system users to a new naming scheme (e.g. job -> yt-job), b52bbb5.
  • Annotate read requests to external attributes (typically served at secondary cells) with the proper user identity, 49fd63b.
  • Add per-chunk replica throttling of data node heartbeat processing, 5ca5f62.
  • Include more information about secondary indices in table attributes, cd9ebbe.
  • Move operation locking output dynamic tables from the controller to the native protocol, 9ac2870.
  • Prohibit chunk_host role revocation from cells with chunks & cypress_node_host role revocation from cells with native Cypress nodes, 5d802d7.

Fixes & Optimizations:

  • Stop generating mutation IDs for read requests. Mutation IDs are considered to be a part of the request message and generating a unique mutation ID for every request previously broke the object service cache, 3d1d9eb.
  • Use the persistent response keeper for commit/abort requests for Cypress transactions, b6da7ab.
  • Fix crash in HydraUpdateMasterCellChunkStatistics when ChunkScanExecutor_ called OnChunkScan more than once before the execution of the committed HydraUpdateMasterCellChunkStatistics mutation, 0c2d095.
  • Fix "list node creation is forbidden" error when attempting to set a list to an attribute of a nonexistent node, bca5761.
  • Remove the broadcast from DoGetMulticellOwningNodes, 75a99bd.
  • Fix potential missing inherited attributes when a node created with the "force" flag overwrote another node, 86d07f9.
  • Fix revision validation when revision paths differ from execution paths, f39e9e8, 1663a23.
  • Forbid creation of tables that are indices of themselves, f6404a8.
  • Fix creation of secondary indices beyond portal, 96ad84e.
  • Fix careful chunk requisition update in chunk merger, 996e11c.
  • Fix statistics checks during the removal of a master-cell, b558b10.
  • Fix job heartbeat processing on a yet non-registered new cell, 4733cbb.
  • Fix the chunk replicator not respecting medium-specific replication factor override in certain scenarios, d6bab2f.
  • Fix the ID of the admins built-in group, 29529ec.
  • Fix nullptr dereference in HydraCreateForeignObject, a30d422.
  • Drop legacy ZooKeeper shard, 558b7b5.
  • Fix master-server not changing the reliability of the exec nodes, 4e9becf.
  • Fix a race between the transaction coordinator committing a transaction and a cell with an exported object unreferencing said object, 5b72aad.
  • Fix manual Cypress node merging for Scheduler transactions, 8a80023.
  • Fix a master crash when setting a YSON dictionary with duplicated keys into a custom attribute, 867354b.
  • Fix row comparison in shallow merge validation so that it does not fail the job, 404a790.
  • Fix a crash triggered by reading @local_scan_flags attribute, d8743cb.
  • Fix non-deterministic error caused by the non-deterministic order of YSON struct field loading when two or more required fields are missing. Since error message is a part of the response for a master mutation, this could lead to "state hashes differ" alert on the master, d907ada.
  • Fix TAttributeFilter handling, 488b343.
  • Fixed locking for concatenation in append mode, c1f5c7e.
  • Fix a bug related to the compatibility patch for the imaginary chunk locations, f591951.

Data Node

New Features & Changes:

  • Support erasure encoding in read size estimation, 4b3a28e.
  • Add flag enable_read_size_estimation to disable read size estimation (true by default), 4b3a28e.

Fixes & Optimizations:

  • Fixed chunk meta extensions absorption in meta aggregated writer, 677d2d4.
  • Fix bug in computing compression ratio in read size estimation based on heavy columnar statistics, add unit tests to address such kind of bugs, 4b3a28e.

Other

New Features & Changes:

  • Sensors and logs for the result of mlock calls, 085a74c.
  • Implemented stockpile relative to user-jobs memory limit (necessary for exe nodes in dynamic-tables-oriented clusters), 8b7c91b.

Fixes & Optimizations:

  • Fix yt.exec_node.rpc_proxy_in_job_proxy_count metric, add host label, 416d8bf.
  • Add config for master cell directory synchronizer into cluster node dynamic config, d049363.
  • Fixed start of node heartbeats before actual registration, 58e442b.
  • Fix crash in meta aggregated writer on corrupted chunks, 5653dfb.
  • Reusal of node lease transaction during re-registration, d0eb92b.
  • Stop aborting node lease transaction when re-register, 6da69bd.
  • Fixed crash of node disconnecting while starting heartbeat report, 66efd89.
  • Tablet nodes used to not validate erasure coding. This is now fixed, ae194de.
  • Data nodes used to not validate block checksums before writing them on disk. This is now fixed, f9cf7c0.
  • Fix UB in chunked memory pool, ec99700.
  • Fix UB in logging zstd compression, 870ca53.
  • Fix a bug in RPC service where a heavy request which had been queued would use propagating storage (e.g. a trace context) from another request, 7745b84.
  • Improved tracking of memory used for row cache, 122fd89.
  • Store 64-bit counters for histogram buckets, 94fe6d3.

Clone this wiki locally