Yak Ops v1.4.0

Yak Ops v1.4.0

Released: 2026-10-24

Overview

v1.4.0 重点完善 MySQL CDC 的 Schema Evolution、Snapshot/Binlog 恢复与 JDBC 目标表结构处理，并改善 Data Sync 及管理页面的可靠性。该版本包含较多运行时重构；升级前必须核对现有任务、Checkpoint 和元数据库的兼容性。

Highlights

• 实时表结构演进： MySQL CDC 可将受支持的结构变化独立于行数据进行协调；MySQL / PostgreSQL JDBC 目标支持经过校验的 DDL，运行时拒绝不受支持或结果不确定的变更。
• 安全删除字段： 管理范围内的非主键 DROP COLUMN 需要显式开启 allowDropColumns；默认禁止破坏性 DDL，不自动删除仅存在于目标端的字段。
• CDC 恢复可靠性： 修复 GTID / Binlog Offset 比较、空表 Bootstrap、Schema History 与历史 DDL 恢复问题；快照状态采用更紧凑的版本化编码并保留已实现的旧格式读取路径。
• 任务界面与工程化： Data Sync 列表、编辑器、详情、日志及 Data Source / Schedule / Management / Dashboard 完成职责拆分，改善查询生命周期和交互稳定性。

New Features

• 新建的 MySQL CDC → MySQL / PostgreSQL 实时任务提供“同步结构变更”配置；其启用行为以任务目标和实际保存参数为准。已存在而未包含该字段的任务不会被静默转换为启用。
• 提供独立 Schema Change Event、Schema Coordinator 与目标 JDBC MetadataApplier；支持经校验的列新增、部分类型变化、受控删除及 DDL 预览。具体支持边界以实际 JDBC Dialect / Schema 检查为准。
• 可显式允许管理范围内的 DROP COLUMN。该操作不可逆，应在启用前备份；非管理列、主键或不满足前置条件的目标会拒绝执行。

Improvements

• MySQL CDC Snapshot/Binlog 对齐、GTID 与偏移顺序、Schema History 持久化/恢复和 Snapshot Checkpoint 编码更加严格；不将全部历史状态格式视为自动兼容。
• JDBC 目标结构对账和 DDL 有界重试使用真实目标元数据作为判断依据，未确认的执行结果不自动当作成功。
• 前端使用更清晰的 Feature Ownership 和 TanStack Query 生命周期管理，减少跨用户/Workspace 缓存混用及旧轮询逻辑；日志查看器交互与主题表现得到调整。
• CI 按变更范围选择测试，正式 Release Gate 仍执行全量后端测试、前端验证、分发包与 Compose Smoke。

Bug Fixes

• 修复空 MySQL 表首次启动 CDC 时可能产生不完整 Schema History，导致后续 Binlog Row 无法找到表结构的问题（#1706）。
• 修复 Binlog/GTID 偏移比较边界及相关快照水位处理（#1707）。
• 修复 CDC 字段变更后的 Schema History / DDL 恢复与 VARCHAR 扩容处理（#1684、#1687、#1710）。
• 修复同步结构变更开关缺少 Field 上下文导致编辑器崩溃的问题（#1688）。

Breaking Changes

• 本版没有新增 Flyway SQL 脚本；v1.3.0 发布的 V1__baseline.sql 不应修改，也不允许通过 Flyway repair 绕过旧库差异。
• 新建实时任务与旧 Task Definition 的 Schema Evolution 默认行为可能不同：新建任务应核实开关状态，旧定义缺失开关时仍视为关闭。DROP COLUMN 保持默认禁止。
• 不承诺全部 v1.3.0 及更早的持久化 CDC Checkpoint / Offset 状态可无条件恢复。 尽管本版保留了部分旧格式读取路径，跨版本状态恢复必须使用真实生产版本的状态副本验证；失败时不得跳过校验或直接删除状态来强行恢复。
• 此前 v1.0.0–v1.2.0 的元数据库依旧不支持直接复用 v1.3.0 的新安装基线；v1.4.0 不提供该历史版本的自动迁移。

Compatibility

• 运行环境：Java 21；Yak Ops 元数据库为 MySQL。实际支持的数据库版本以 Release Gate 和受控环境测试为准。
• 数据同步范围：JDBC Source/Sink 涉及 MySQL、PostgreSQL、Oracle；实时 Source 为 MySQL CDC。不代表 PostgreSQL CDC 或 Oracle CDC 已实现。
• 实时目标 DDL 演进适用于受支持的 MySQL / PostgreSQL 路径；Oracle 目标不开放此运行时 DDL 开关。
• 不对未验证的跨版本状态转换、分布式 exactly-once、任意 DDL 兼容性作承诺。

Upgrade Guide

1. 备份现有 v1.3.0 MySQL 元数据库、任务定义、数据卷与 CDC 持久化状态，记录当前运行中的任务和版本。
2. 在备份副本上检查 flyway_schema_history 与 V1__baseline.sql 校验和；本次无 V2 迁移脚本。不得改写已发布的 V1，也不得凭校验通过推定所有任务状态可兼容。
3. 针对正在运行的 CDC 任务，记录 Checkpoint / Offset 并在隔离环境使用 v1.4.0 对真实旧状态执行恢复验收；如果失败，保留旧版本和原始状态，制定明确切换方案。
4. 核对实时任务“同步结构变更”和“允许删除字段”开关，特别是新建与历史 Task Definition 的默认差异。先使用测试表验证 DDL，不允许默认开启破坏性操作。
5. 使用明确的 1.4.0 分发包或 Docker Tag 在预发布环境完成启动、任务执行与取消/恢复、调度及数据正确性验收，再切换生产流量。
6. 回滚时恢复经过验证的旧版本镜像和备份数据。已执行的破坏性目标 DDL 不能通过镜像回滚自动撤销。

Known Limitations

• DROP TABLE、RENAME 等不属于已承诺的自动 Schema Evolution 范围；只有显式允许且满足条件的管理字段可以删除。
• Oracle JDBC 目标的实时 DDL 演进限制仍然存在；PostgreSQL / Oracle CDC Source 未在本版实现。
• v1.3.0 → v1.4.0 运行状态和历史任务配置应经过专门验收，不能因全新任务测试通过就视为兼容。
• 发行版正式测试结论需以该版本提交的 Release Gate、Manual E2E 和附件为准；本说明不预先宣称尚未执行的验收已通过。

Downloads

• 正式发布后：yak-ops-1.4.0.tar.gz、SHA256SUMS 和 release-evidence-v1.4.0.md 将随 GitHub Release v1.4.0 提供。
• 版本化 Docker 镜像：weifuwan/yak-ops:1.4.0 与 weifuwan/yak-ops-api:1.4.0，仅在 Release Publish 成功后可用；不能以 latest 代表版本凭证。

References

• v1.3.0 发布记录
• v1.3.0 到当前 main 的变更（正式发布后以 v1.3.0...v1.4.0 Tag 对比为准）
• Schema Evolution 与 DROP COLUMN #1689
• CDC Snapshot / State Recovery #1708
• CDC Base 边界 #1709
• CDC Schema ACK / Offset 清理 #1711
• 发布流程

Yak Ops v1.3.0

Yak Ops v1.3.0 Release Notes

Summary

Yak Ops v1.3.0 统一 Task 任务管理模型，完善 JDBC 数据同步、MySQL CDC、执行状态和调度能力，优化 MySQL 时区配置，并将 Flyway 整理为单一数据库初始化脚本。

Known Limitations

v1.3.0 仅支持空 MySQL 数据库全新安装，不支持从 v1.0.0–v1.2.0 原地升级。已有数据库需要单独迁移，不能直接复用旧数据卷。

What's Changed

• v1.3.0 Release Contract + Architecture Boundary by @weifuwan in #1496
• Data Sync Multi-Table Route Contract + Persistence by @weifuwan in #1497
• Multi-Table Definition Snapshot + Table Execution by @weifuwan in #1498
• Multi-Table Runtime + Per-Table Retry / Metrics by @weifuwan in #1499
• Multi-Table Sync Editor + Schema Preview by @weifuwan in #1500
• Data Sync Executor Structure + Type Boundary Refactoring by @weifuwan in #1501
• Data Sync Executor Shared Attempt Orchestration by @weifuwan in #1502
• Data Sync Execution Lifecycle Safety + Metrics Optimization by @weifuwan in #1503
• docs(core): 统一 Core 英文注释为中文 by @weifuwan in #1504
• refactor(runtime): 将本地 Pipeline 执行实现迁移到 yak-flow-runtime by @weifuwan in #1505
• PR0 — Core / Runtime Architecture Alignment Contract by @weifuwan in #1506
• PR1 — Core API Consistency + Compilation Closure by @weifuwan in #1507
• PR2 — Configuration + Graph Execution Contract by @weifuwan in #1508
• PR3 — TaskInfo + Runtime Environment Foundation by @weifuwan in #1509
• PR4 — Source Coordination + Operator Context Alignment by @weifuwan in #1510
• PR5 — Local Job Assembly + StreamTask Lifecycle by @weifuwan in #1511
• PR6 — StreamEdge Partitioning + Local Channel by @weifuwan in #1512
• PR7 — Checkpoint Coordination + State Recovery by @weifuwan in #1513
• PR1 — Legacy Offline / Realtime Sync Removal + Interface-Only Baseline by @weifuwan in #1514
• Restore Data Sync UI / Controller / Service / DAO; remove only legacy runtime by @weifuwan in #1515
• YakFlow Core / Runtime Ownership + Flink Architecture Alignment by @weifuwan in #1517
• Legacy Data Sync Execution + Connector Removal by @weifuwan in #1518
• YakFlow JobGraph / ExecutionGraph + Flink Execution Lifecycle Alignment by @weifuwan in #1519
• YakFlow StreamTask MailboxProcessor + Flink Mailbox Lifecycle Alignment by @weifuwan in #1520
• YakFlow StreamTaskInput + ResultPartition / InputGate Alignment by @weifuwan in #1521
• YakFlow Source Coordination + Execution Attempt Recovery Alignment by @weifuwan in #1522
• YakFlow StreamOperator + Sink V2 + KeyGroup Alignment by @weifuwan in #1523
• PR3 — YakFlow Aligned Barrier Checkpoint + Operator State / StateBackend by @weifuwan in #1524
• YakFlow Checkpoint Correctness + State Recovery Safety by @weifuwan in #1525
• Yak Ops CI Gate Consistency + Acceptance Status Cleanup by @weifuwan in #1526
• Restore Spotless Baseline + Quality Check by @weifuwan in #1527
• Yak Ops CI Consolidation + Conditional Execution by @weifuwan in #1528
• PR1 — YakFlow Core Configuration Tests + Contract Test Cleanup by @weifuwan in #1529
• PR2 — YakFlow Runtime Failure + Checkpoint Recovery Test Alignment by @weifuwan in #1530
• YakFlow Unified Sink API + SinkV2 Removal by @weifuwan in #1531
• YakFlow Legacy API Compatibility Cleanup by @weifuwan in #1532
• YakFlow Core / Runtime Javadoc Convention + Comment Alignment by @weifuwan in #1533
• YakFlow Core / Runtime Comment Convention Baseline Cleanup by @weifuwan in #1534
• YakFlow Connector Base Reader Foundation by @weifuwan in #1535
• YakFlow JDBC Source + Split Enumerator by @weifuwan in #1536
• YakFlow JDBC Source Integration + Architecture Cleanup by @weifuwan in #1537
• Fix YakFlow PR3 Core Schema Migration CI by @weifuwan in #1538
• YakFlow Logical Types + RowData Contract Alignment by @weifuwan in #1539
• YakFlow JDBC Factory + Dialect Converter Alignment by @weifuwan in #1540
• YakFlow JDBC Source Reliability + Checkpoint Alignment by @weifuwan in #1541
• YakFlow JDBC Catalog + Internal Package Alignment by @weifuwan in #1542
• YakFlow JDBC Source Split Recovery + Catalog Metadata + Projection by @weifuwan in #1543
• YakFlow Sink Foundation + Mailbox Batch Lifecycle by @weifuwan in #1544
• YakFlow JDBC OutputFormat + Statement Executor Alignment by @weifuwan in #1545
• YakFlow JDBC Multi-Table Changelog + Checkpoint Acceptance by @weifuwan in #1546
• JDBC Acceptance CI On-Demand + Release Gate Integration by @weifuwan in #1547
• YakFlow JDBC Sink Changelog Safety + Internal Structure Alignment by @weifuwan in #1548
• YakFlow JavaDoc Convention + JDBC Source/Sink Comment Alignment by @weifuwan in #1549
• YakFlow MySQL CDC Binlog Source + Debezium Foundation by @weifuwan in #1550
• YakFlow MySQL CDC Hybrid Snapshot + Binlog Handoff by @weifuwan in #1551
• MySQL Hybrid Bootstrap Schema History Readiness by @weifuwan in #1552
• YakFlow MySQL CDC Runtime Integration + Cross-Database Acceptance by @weifuwan in #1553
• YakFlow MySQL CDC Incremental Snapshot + Binlog Alignment by @weifuwan in #1554
• Data Sync Mapping + Auto Create Removal & Business Cleanup by @weifuwan in #1555
• Datasource Catalog Retirement + YakFlow JDBC Metadata Unification by @weifuwan in #1556
• YakFlow MySQL Multi-Table JDBC + CDC Runtime E2E Acceptance by @weifuwan in #1557
• Data Sync Business Service Refactoring + Single-Table Simplification by @weifuwan in #1558
• PR1 — Task Platform Contract + Plugin SPI by @weifuwan in #1559
• PR2 — Unified Task Definition + Data Sync Migration by @weifuwan in #1560
• PR3 — Task Instance + Schedule + Metrics Alignment by @weifuwan in #1561
• Task Definition Foundation Reset + Sync Plugin Consolidation by @weifuwan in #1562
• JavaDoc Convention Unification + Datasource / Task Comment Alignment by @weifuwan in #1563
• Sync Pipeline HOCON Configuration + Connector Boundary Alignment by @weifuwan in #1564
• Task Plugin API Simplification + Sync Configuration Cleanup by @weifuwan in #1565
• Data Sync Definition CRUD + Frontend Contract Alignment by @weifuwan in #1566
• Boot Controller Package Flattening + Task Definition API Unification by @weifuwan in #1567
• Task Definition JSON-Only Configuration + HOCON Removal by @weifuwan in #1568
• ServiceLoader SPI Registration via AutoService by @weifuwan in #1569
• PR1 — Flink Documentation Standards + FLIP Adoption by @weifuwan in #1570
• PR2 — Release Documentation Retirement + Release Gate Alignment by @weifuwan in #1571
• PR3 — Legacy Backend Documentation Removal by @weifuwan in #1572
• Flink CI Quality Alignment + Script Cleanup by @weifuwan in #1573
• docs: Adopt YOIP proposal process and preserve Flink template by @weifuwan in #1574
• [DISCUSS][YOIP-1] YakFlow Checkpoint and End-of-Input Semantics by @weifuwan in #1575
• [DISCUSS][YOIP-1] Converge Checkpoint and Finished-Operator Semantics by @weifuwan in #1576
• [DISCUSS][YOIP-1] Resolve Final Checkpoint Review Findings by @weifuwan in #1577
• docs: Accept YOIP-1 Checkpoint and End-of-Input Sem...

Yak Ops v1.2.0

Yak Ops v1.2.0 Release Notes

Status: Release Candidate — Awaiting Formal Release Gate

Theme: Schema-Aware Sync + Smart Runtime

Yak Ops v1.2.0 重点解决两个问题：

系统能否自己理解 Source / Target Schema，并把普通用户不应该维护的 Runtime / Retry 参数内化为系统策略？

本版本在 v1.1.0 的调度、Execution / Attempt、Realtime Desired State、Auto Recovery 与运行诊断基础上，进一步收口 Schema、字段映射、自动建表、运行规划与智能重试。

Highlights

Schema-Aware Sync

Data Sync 现在把真实数据库物理结构归一为产品级 Logical Schema：

Datasource Catalog
        ↓
LogicalTable
        ↓
YakTableSchema
        ↓
Target Table Plan
        ↓
Schema Compatibility
        ↓
Controlled Auto Create

主要能力：

• Source Metadata Introspection。
• Logical Type Normalization。
• Composite Primary Key 顺序保留。
• MySQL / PostgreSQL / Oracle Target Native Type Planning。
• Target Schema Compatibility。
• 缺失 Target 的显式 Auto Create。
• CREATE TABLE + Table / Column Comment DDL。
• Runtime 执行前重新读取真实 Catalog 并再次校验。
• Task Editor 提供 Schema Preview，DDL 通过目标表辅助入口按需查看。

Auto Create 只处理缺失目标表，不对已存在表执行 ALTER / DROP / DDL Sync。

Column Mapping

Task 现在拥有稳定、可冻结的字段映射：

Source Field
→ Target Field

支持：

• 字段子集。
• 字段重排。
• 字段改名。
• Mapping Editor 手工连接。
• Auto Create 时使用映射后的目标字段名建表。
• REALTIME Source PK 改名后继续完成 Snapshot / INSERT / UPDATE / DELETE。
• Mapping 冻结进 Execution definitionSnapshot。

不包含：

• Expression。
• CAST。
• Custom SQL。
• Transform。

Smart Runtime

普通用户不再直接维护：

fetchSize
readBatchSize
writeBatchSize
splitSize
sourceParallelism
checkpoint / queue / timeout tuning

OFFLINE 新任务默认使用 backend-owned AUTO Runtime Policy。

根 Execution 创建时，系统根据：

• Mapping 后 Source Logical Schema。
• 受控 Source Statistics。
• Target 数据库类型。

规划 Effective Runtime Config，并冻结到 Execution Snapshot。

同一个 Execution 的 Retry Attempt 继续复用该冻结配置，不在每次 Retry 时重新规划。

历史没有 policy 的 Runtime Config 继续按 FIXED 兼容，避免升级后覆盖已有调优值。

SMART Retry

新建普通 Task 省略 Retry Policy 时，后端物化：

mode = SMART
maxAttempts = 3
base backoff = 15s

SMART 只对明确瞬时失败自动 Retry，例如连接、recoverable / transient、timeout、SQLState 08 / 40 与部分网络错误。

以下错误不会因为“最多 3 次”而无条件重试：

• SQL / Schema 错误。
• 权限错误。
• 完整性约束。
• 产品校验失败。
• 非法参数。
• 能力不支持。
• 未知异常。

写入安全优先：

• OFFLINE APPEND / OVERWRITE 在 Runtime 已启动后不自动重放。
• OFFLINE UPSERT / REALTIME CHANGELOG 可对明确瞬时失败继续 SMART Retry。
• 本版本仍不声明 exactly-once。

Durable Retry Recovery

RETRY_WAITING 的 nextRetryTime 已持久化。

Yak Ops 在等待 Retry 时重启：

RETRY_WAITING
        ↓
restart Yak Ops
        ↓
same Execution Root
same taskVersion
same definitionSnapshot
same retryPolicy
        ↓
wait persisted nextRetryTime
        ↓
next Attempt

它不会因为进程退出直接把该 Execution 改成 LOST，也不会重新创建新的 MANUAL / AUTO_RECOVERY Root。

Realtime Task Lifecycle

Realtime Task List 已补齐：

启动
停止
重新启动

状态语义：

• PUBLISHED + STOPPED + 无 Active Execution → 启动。
• PENDING / RUNNING / RETRY_WAITING → 停止。
• desiredState=RUNNING + 无 Active Execution → 重新启动。
• UNPUBLISHED → 启动禁用。

用户主动 Stop 会把 desiredState 改为 STOPPED，因此后续按钮回到“启动”；它与 desired-state drift 下的“重新启动”不是同一个状态。

Database Upgrade

v1.2.0 使用一个 forward-only Release Migration：

V3__v1_2_0.sql

它按开发期原 SQL 顺序包含：

Data Sync Auto Create Table
→ Data Sync Column Mapping

新增：

• yak_ops_data_sync_task.auto_create_table
• yak_ops_data_sync_task.mapping_config

原开发期 V3 / V4 Draft Migration 已在 Release Freeze 前 squash，不属于最终用户 Flyway 历史。

v1.0.0 / v1.1.0 的 V1 / V2 Migration 保持不可变。

Compatibility

v1.2.0 保持：

• Product Version 与 Task definitionVersion 独立。
• 旧 Task autoCreateTable=false 默认行为。
• mapping_config = NULL 的隐式同名 Mapping。
• 历史 Runtime Config 无 policy 时按 FIXED。
• 历史 Retry Policy 无 mode 时按 FIXED。
• REALTIME CDC state 继续按 Workspace / Task / definitionVersion 隔离。
• Realtime continuation 继续是 at-least-once。

Known Limitations

v1.2.0 明确不包含：

• Logical Table Persistence。
• Catalog Refresh / Schema Diff。
• DDL Sync / Automatic Schema Evolution。
• Multi-table Task。
• Transform / Expression / Custom SQL Mapping。
• PostgreSQL CDC 或更多 CDC Source。
• 用户可配置 Runtime Tuning UI。
• 用户可配置 Retry Policy UI。
• Continuous Realtime Reconciliation / 常驻 Watchdog。
• Runtime Trace Retention / Object Storage / Full-text Search / REALTIME Trace。
• 通用 YakFlow Checkpoint 跨进程恢复。
• Distributed Runtime / Worker / HA / leader election / fencing。
• Exactly-once transaction coordination。

Release Acceptance

v1.2 Required Manual E2E 已冻结为：

• MAPPING-001 — Offline MySQL → PostgreSQL Mapping + Auto Create + Comment + AUTO Runtime。
• MAPPING-002 — Realtime MySQL CDC → PostgreSQL PK Rename + Auto Create + Start / Stop / Resume。
• RECOVERY-001 — Durable Retry Recovery。

三个 Required Manual E2E 已于 2026-10-07 完成人工执行并确认 PASS：MAPPING-001、MAPPING-002、RECOVERY-001。

正式发布身份与最终验收结果以 v1.2.0 Release Readiness 为准。

Yak Ops v1.1.0

Yak Ops v1.1.0 Release Notes

Automation & Recovery

Yak Ops v1.1.0 focuses on one question:

Can a published data-sync task keep operating with clear scheduling, retry and restart semantics instead of depending only on manual execution?

This release adds scheduling, retry attempts, realtime desired state / auto recovery, and Operations Center visibility while keeping the existing Data Sync / YakFlow responsibility boundary.

Highlights

Offline Scheduling

Published Offline Tasks can be scheduled with:

• Quartz Cron expressions.
• Explicit time zone.
• Enable / disable.
• Scheduler-computed next fire time.
• SCHEDULE Execution trigger source.
• SKIP_IF_RUNNING when the same Task already has an active Execution.
• Dedicated Offline Task configuration UI for Cron + IANA Time Zone.
• Explicit Schedule enable / disable actions in the Offline Task list after the Task is published; Operations Center remains observability-focused.

Yak Ops DB remains the Schedule source of truth. Quartz is runtime trigger infrastructure and is rebuilt from enabled Schedule definitions after application restart.

Retry / Attempt

A Data Sync Instance is now the stable Execution root.

One Execution can contain multiple Attempts:

Execution
  ├── Attempt #1 FAILED
  ├── Attempt #2 FAILED
  └── Attempt #3 SUCCEEDED

Retry behavior includes:

• maxAttempts, including Attempt #1.
• Fixed backoffSeconds.
• RETRY_WAITING Execution state.
• Independent Attempt metrics / timestamps / error history.
• Stop / Cancel prevents automatic Retry.
• Retry reuses the Execution's immutable taskVersion and definition snapshot.

Retry does not create a new root Execution and does not use Quartz as a Retry timer.

Realtime Desired State + Auto Recovery

Realtime Tasks now separate user intent from current process state:

Publication State
Desired State
Execution State

A user Start sets:

desiredState = RUNNING

A user Stop or Unpublish sets:

desiredState = STOPPED

After application restart:

• old process-local active Executions become LOST,
• a published Realtime Task whose desired state remains RUNNING gets a new AUTO_RECOVERY Execution,
• the new Execution keeps the same taskId + definitionVersion CDC state identity,
• Debezium persisted offset / schema history is reused.

Realtime continuation remains at-least-once; this release does not claim exactly-once.

Operations Center

Operations Center is the cross-Task observability surface for automation and recovery:

• OFFLINE / REALTIME aggregate Dashboard.
• Execution count and status distribution.
• Read / write totals and duration aggregates where persisted metrics support them.
• Realtime abnormal Task / Execution visibility.
• AUTO_RECOVERY trend / marker.
• Failure ranking and recent automation health.

Per-Task definition and runtime controls do not live in the Dashboard. Data Integration owns Task Definition / Publish; the OFFLINE Task List owns manual Run and explicit Schedule enable / disable; Task Detail owns Task-scoped read-only Execution / Attempt inspection.

Offline Execution Diagnostics

v1.1.0 also adds structured runtime diagnostics for bounded Offline JDBC sync:

Source Split
  → generated SELECT SQL
  → numeric Split range
  → rows read
  → end-to-end read duration

Target Batch
  → SQL Template
  → Batch rows
  → executeBatch duration
  → commit duration

The implementation is deliberately separated into three layers:

• YakFlow Runtime Trace Contract + JDBC diagnostic events.
• File-backed Execution Trace Store + read-only diagnostics API.
• OFFLINE Task Detail 执行诊断 UI.

Trace files are stored under:

${yak.ops.home}/data/data-sync/execution-traces

Important boundaries:

• Trace is best-effort and never changes Data Sync Runtime success/failure.
• Business row values and Sink bind parameters are not persisted in Trace.
• Trace error messages are sanitized before persistence.
• Large Source / Sink detail uses cursor pagination instead of loading the entire Trace file.
• Lifecycle product events remain available as auxiliary audit context.
• REALTIME Runtime Trace is not included in this release.

Release Acceptance

Backend Acceptance now has three independent layers:

JDBC Acceptance
MySQL CDC Cross-Database Acceptance
Data Sync Automation Acceptance

The new Automation Acceptance covers Scheduler, SKIP_IF_RUNNING, Retry/Backoff, Cancel no-retry and Auto Recovery identity semantics.

Two required v1.1 Manual E2E scenarios have been executed and recorded as PASS on 2026-10-04:

• AUTOMATION-001 — Offline Cron scheduling + disable — PASS.
• AUTOMATION-002 — Realtime process restart + Auto Recovery — PASS.

Formal publication still requires the exact 1.1.0 release commit to pass the Release Gate Full Sweep.

Database Upgrade

v1.1.0 has one forward-only Release Migration:

V2__v1_1_0.sql

V1__baseline.sql remains immutable from v1.0.0.

V2 groups the v1.1.0 schema changes in release order:

Offline Schedule
→ Retry / Attempt
→ Realtime Desired State / Auto Recovery
→ Execution Product Event

The feature-level V2 ~ V5 files used during development were unpublished Draft Migrations and were squashed before release. They are not part of the public Flyway history.

Upgrade behavior includes:

• existing Tasks default to no automatic Retry unless explicitly configured,
• active published Realtime Tasks are backfilled to desiredState=RUNNING during the v1.1.0 upgrade,
• historical Execution identity remains stable,
• Execution product-event persistence starts with v1.1.0; historical executions are not backfilled with synthetic lifecycle events.

Compatibility

v1.1.0 preserves:

• existing v1.0 Task identity.
• Task definitionVersion independence from product version.
• existing Workspace / Datasource ownership boundaries.
• existing persisted Realtime CDC state identity by Workspace / Task / definitionVersion.
• manual Offline / Realtime execution.

Known Limitations

v1.1.0 deliberately does not include:

• Distributed Runtime / Worker / HA.
• Leader election or distributed fencing.
• Exactly-once transaction coordination.
• QUEUE / PARALLEL Offline concurrency.
• Continuous Realtime recovery watchdog.
• Durable cross-process Retry timer.
• Auto Create Table.
• Logical Data Modeling.
• Multi-table Task.
• Transform.
• PostgreSQL CDC.
• Paimon / Hive / File connector expansion.
• Object-storage Runtime Trace backend, Trace retention policy, full-text Trace search or REALTIME Runtime Trace.

Additional UI boundary:

• Retry Policy does not yet have a dedicated configuration UI.

The corresponding backend contracts exist and Operations Center can observe their runtime state.

Deployment Note

Realtime restart continuation depends on the persisted Yak Ops data directory.

Production deployments must preserve:

${yak.ops.home}/data

Container replacement must keep the mounted data volume. Deleting that state removes the persisted Debezium offset/schema history required for continuation and the local Offline Execution diagnostic Trace history.

Release Evidence

Formal release identity and acceptance evidence are tracked in:

• v1.1.0 Release Readiness
• v1.1.0 Release Evidence

Yak Ops v1.0.0

Yak Ops v1.0.0 Release Notes

状态：Release Candidate

目标版本：1.0.0

目标 Tag：v1.0.0

Yak Ops v1.0.0 是第一个稳定发布版本，目标是提供一条可部署、可验收、可追溯的数据集成核心路径。

1. V1 核心能力

平台

• 用户登录 / 退出 / 当前身份。
• 用户管理。
• Workspace 生命周期、成员关系与请求隔离。
• 用户级 Workspace 选择保持。
• 用户偏好、收藏与常用数据源类型。

数据源

内置支持：

MySQL 5 / 8
PostgreSQL
Oracle

支持 Datasource CRUD、真实连接测试、批量删除 / 连通性测试、Catalog Metadata、Structured JDBC Connection、Advanced Properties，以及 MySQL 5 / 8 Driver 隔离。

离线同步

Source：

MySQL

Target：

MySQL
PostgreSQL
Oracle

写入方式：

APPEND
OVERWRITE
UPSERT

其中 UPSERT 使用目标数据库原生语义：

MySQL       → ON DUPLICATE KEY UPDATE
PostgreSQL  → ON CONFLICT DO UPDATE
Oracle      → MERGE INTO

支持字段自动映射、JDBC Source split、Source parallelism、手工运行、Stop、Instance 历史与 readRows / writeRows 指标。

实时同步

Source：

MySQL CDC

Target：

MySQL
PostgreSQL
Oracle

支持：

• Debezium initial snapshot。
• MySQL Binlog continuous streaming。
• INSERT / UPDATE / DELETE。
• JDBC CHANGELOG apply。
• Checkpoint。
• persisted offset / schema history。
• 同一 Task + 同一 definitionVersion 停止后从已完成 offset 继续。
• Instance 运行状态与事件指标。

V1 的实时恢复语义是：

at-least-once

不声明 exactly-once。

任务生命周期

Data Integration 负责：

• Task Definition。
• Save / Save & Publish。
• Publish / Unpublish。
• definitionVersion。

Operations Center 负责：

• 离线任务运行 / Stop。
• 实时任务 Start / Stop。
• Instance 列表 / 详情。
• 运行态观察。

2. 自动验收

正式 Release Gate 会完整执行：

Quality Gate
Full JDBC Acceptance
Full MySQL CDC Cross-Database Acceptance
Distribution Build + Verification
Backend / Frontend Docker Build
OCI Version / Revision Check
Compose Startup
Backend Health
Frontend HTTP
Nginx Login + Current User
Release Candidate Evidence

跨数据库自动 Acceptance 覆盖：

Offline
MySQL → MySQL
MySQL → PostgreSQL
MySQL → Oracle

Realtime
MySQL CDC → MySQL
MySQL CDC → PostgreSQL
MySQL CDC → Oracle

3. Manual E2E

v1.0.0 必选的 5 个 Manual E2E 已由项目维护者人工执行通过：

OFFLINE-001  APPEND
OFFLINE-002  OVERWRITE
OFFLINE-003  UPSERT
REALTIME-001 Snapshot + INSERT / UPDATE / DELETE
REALTIME-002 persisted-offset continuation

详细记录见 v1.0.0 Release Readiness。

4. Distribution / Docker

Distribution：

yak-ops-1.0.0.tar.gz
SHA256SUMS

Docker：

weifuwan/yak-ops:1.0.0
weifuwan/yak-ops-api:1.0.0

正式发布物必须来自同一个 Release Commit，并通过 Version Consistency 与 Distribution Verification。

5. 部署要求

• Java 21。
• Yak Ops 元数据库必须可用。
• 生产环境必须显式配置 YAK_OPS_DATASOURCE_MASTER_KEY。
• 示例默认密码只用于本地 / 演示环境，生产环境必须替换。
• Realtime CDC 状态目录必须持久化，容器替换 / 回滚不能清理 ${yak.ops.home}/data。
• MySQL CDC Source 必须满足 ROW Binlog 与连接器所需权限。

Compose 支持：

• compose.yaml：包含 MySQL。
• compose.without-mysql.yaml：使用外部 MySQL。

6. Known Limitations

以下限制属于 V1 明确边界，不是 Release Blocker：

• Realtime 为 at-least-once，Checkpoint 后少量未确认事件在恢复时可能再次消费。
• 不支持 exactly-once transaction coordination。
• Offline Source 当前只承诺 MySQL。
• Realtime Source 当前只承诺 MySQL CDC。
• 目标表需要预先存在，不支持 Auto Create Table。
• 不支持 DDL propagation / Schema Evolution。
• 不支持 Transform。
• 不支持 Multi-table Task。
• 不支持 Scheduler / Cron。
• 不支持自动重试策略。
• 不支持 Distributed Runtime / Worker / HA。
• 应用重启后旧的 active Instance 会标记为 LOST；Realtime 通过新 Instance + persisted connector state 继续，不复活旧 Instance。
• Offline OVERWRITE 会先清空目标表，再执行写入；后续写入失败不会自动恢复清空前的数据。
• 启用 Offline split 后，不承诺多个 split 共享一个整表一致性快照时点。

7. V1 Non-Goals

以下能力明确进入后续版本规划，不阻塞 v1.0.0：

• 自动建表与逻辑建模。
• 调度系统。
• 自动重试。
• 分布式执行。
• Exactly-once。
• DDL / Schema Evolution。
• Transform。
• Multi-table Sync。
• 通用 Enterprise RBAC。

8. Database Migration

v1.0.0 是 Flyway baseline 的冻结点。

正式 Tag 创建后：

V1__baseline.sql = immutable

后续数据库变化必须通过新的 Migration，例如：

V2__*.sql
V3__*.sql

不得回改已发布的 V1 baseline。

9. Release Readiness

当前 V1 Scope 内没有已知 P0 / P1 Release Blocker。

历史 Issue #1074 / #1079 不纳入本次 v1.0.0 Release Blocker 判定。

最终状态以 v1.0.0 Release Readiness 与正式 V1 Release Gate 为准。

v0.1.0

This is the first release of Yak Ops, and also just the beginning.

It’s not perfect yet, but that’s okay. We’ll keep improving it, little by little.

If no one is building it, I will.