1.23.0

🚀 What's New

Batch Kafka Processing

Introduced two new stable nodes for reliable, batch-oriented Kafka processing:

• Read Kafka Messages — Reads a finite snapshot of a Kafka topic into a lazy DataFrame.
  • Multiple starting offset modes: committed, earliest, latest, timestamp, and explicit.
  • Configurable limits on message count and total bytes.
  • Transaction-aware reading with read_committed isolation.
• Commit Kafka Offsets — Explicitly commits consumed offsets after all designated destinations have successfully completed processing.

The Kafka Connection node has also been promoted from experimental to stable.

Multi-Column Transformation Rules

Several transformation nodes now support configurable rule lists, making it easier to apply operations across multiple columns.

Supported nodes:

• Replace Values
• Regex Replace
• Set Timezone
• Truncate to Period
• Add Time Delta
• Split Column
• Add Column by Expression
• Insert Column into DataFrame

Rule capabilities:

• Select columns by exact name, wildcard patterns (*, ?), or all columns.
• Filter selections by data type and exclusions.
• Apply the first matching rule to each column.

The node editors have been redesigned to support rule-based configuration.

Backward compatibility: Existing saved graphs are automatically migrated to the new format during the upgrade and continue working without modification.

Database Table and Column Comments

Database comments are now available throughout the data workflow:

• View source comments in the database catalog and column selectors.
• Preserve column comments in the output metadata of read nodes.
• Add and edit column comments when modifying database table structures.

Column Nullability Management (NULL / NOT NULL)

Table structure operations now support safely changing the nullability of existing columns.

• Switch between nullable and non-nullable columns.
• Validate existing data before applying a NOT NULL constraint.
• Supported databases: PostgreSQL, MySQL/MariaDB, MS SQL Server, Oracle, and ClickHouse.

Built-in Node Documentation

Documentation is now bundled directly with DVT and available from the pipeline editor.

• Available in English and Russian.
• Covers most built-in nodes.
• Accessible without relying on external documentation services.

Enhanced AI Assistant (MCP)

The AI assistant now supports additional project management and pipeline design operations:

• Project creation — Create new projects directly through the assistant.
• Schedule management — Create, modify, enable, and disable execution schedules (administrator access required).
• Automatic node layout — Arrange pipeline nodes from left to right on the canvas.
• Database schema preparation — Prepare target table structures with an SQL preview before applying changes.

Offline Installation and Upgrades

DVT can now be installed and upgraded in environments without internet access.

• New scripts for installing and upgrading from local archives.
• Container images are loaded from local files.
• Existing configuration and secrets are preserved during upgrades.

---

⚡ Improvements

Data Preview

Improved readability and type awareness in the DataFrame preview:

• Binary values are displayed as <binary: N bytes> without exposing their contents.
• Nested data structures, including lists and structs, are presented in a readable format.
• Added type indicators for BINARY, LIST, and STRUCT columns.
• Column header tooltips now include detailed Arrow type information.

Refreshed Node Icons

Replaced emoji-based node icons with consistent vector icons across:

• The node library.
• Node headers on the canvas.
• Context menus.

Seamless Extension Runtime Updates

Execution workers now safely update extension runtimes between tasks, minimizing queue interruptions and preventing job loss during runtime updates.

Improved Node Descriptions

Refined parameter descriptions and tooltips across numerous nodes to provide clearer guidance for both manual configuration and AI-assisted workflows.

---

⚠️ Important Changes

Kafka Connection Migration

The Kafka Connection node is no longer experimental.

Its input has changed from a hidden numeric identifier to a string-based connection identifier from the connection catalog.

Action required: Graphs created with the experimental version of this node must be reconnected.

Kafka Offset Commit Semantics

Kafka message consumption never commits offsets automatically.

To advance the consumer group's offsets, add a Commit Kafka Offsets node after receiving successful completion signals from all destinations.

Delivery guarantees:

• Processing follows at-least-once semantics.
• If a failure occurs after writing to a destination but before committing offsets, messages may be processed and written again.
• Exactly-once delivery is not guaranteed.

Irreversible Extension Identity Migration

This release introduces an irreversible database migration affecting extension identities.

Required upgrade procedure:

1. Wait for all active executions to complete.
2. Apply the database migrations before starting the updated services.
3. Start the updated DVT services.

Additional details:

• Extensions do not need to be reinstalled.
• Existing extension data and state are preserved.
• External extension API URLs now use canonical package names.
• Legacy extension names remain supported internally.

---

🐛 Bug Fixes

• ClickHouse: Fixed incorrect nullability metadata and schema representation for LowCardinality(Nullable(...)) columns.
• Execution Workers: Fixed task queue stalls during extension runtime updates.
• Realtime Connections: Fixed duplicate UI realtime connections that could create unnecessary WebSocket connections.
• Installation Manager: Corrected the configuration download URL in the installation script.

1.22.0

1.22.0