Skip to content

OSS Transfer v4.1.41

Choose a tag to compare

@github-actions github-actions released this 09 Nov 15:33
· 127 commits to main since this release

对象比对全面改进

  • 不再限制 10,000 条差异:每条差异都保存在 Agent 数据目录下的文件中。按容量(默认 2 GiB)、保留天数(默认 30 天)、磁盘剩余空间(默认至少 1 GiB)和数量自动清理,可在 Agent 的 service.json 中用可选的 comparison 块调整。
  • 导出:ZIP(differences.csv、retry-source-keys.incr、summary.json)或 JSON Lines。retry-source-keys.incr 列出需要重新迁移的源 Key,可直接用于再次迁移;summary.json 说明导出是否完整。
  • 断点续比:取消后可以继续;Agent 重启或升级后自动从检查点继续。
  • 更快:按 Key 顺序并发比对,每个对象两端的 HEAD 同时发出。测试环境中 321,845 个对象的整桶比对从 1 小时 55 分钟缩短到 9 分钟。
  • 更准确:修复列举时间带毫秒、HEAD 时间只到秒导致的误判;列出不一致的字段;新增“ETag 不同”提示;读取失败时给出具体原因(HTTP 状态、错误码、请求 ID);两端都不存在的对象单独计数。
  • 内容(SHA-256)模式:大对象按 8 MiB 分段读取,失败的分段从中断处重试,60 秒收不到数据的读取会中止并重试,并显示已读取的字节数。
  • 控制台和 CLI:比对历史、按结果筛选、分页、导出 ZIP、继续比对;新增 oss-transfer-cli object list、object status --status、object export、object resume。
  • 行为变化:只指定 Key 的 object compare 现在会启动单对象比对,15 秒内完成时直接显示结果,否则显示比对 ID;--json 输出改为比对记录。
  • 兼容性:新 Master 可以管理旧版 Agent 上的比对(最多保存 10,000 条差异,不能继续比对)。如果在 Agent 的 service.json 中添加了 comparison 块,降级到旧版前请先删除。

Object comparison overhaul

  • No 10,000-difference limit: every difference is stored in files under the Agent's data directory. Results are cleaned up by size (2 GiB by default), age (30 days), free disk space (at least 1 GiB) and count; adjust these with the optional comparison block in the Agent's service.json.
  • Export as ZIP (differences.csv, retry-source-keys.incr, summary.json) or JSON Lines. retry-source-keys.incr lists the source keys to transfer again and can be used for another migration; summary.json states whether the export is complete.
  • Resume: continue a canceled comparison; after an Agent restart or upgrade a comparison resumes from its checkpoint by itself.
  • Faster: keys are compared concurrently in key order, with both HEADs of an object sent at once. A 321,845-object bucket in the test environment took 9 minutes instead of 1 hour 55 minutes.
  • More accurate: fixes false differences when listings report milliseconds and HEAD only seconds; names the properties that differ; adds an "ETag differs" hint; read failures show the HTTP status, error code and request ID; objects missing on both sides are counted separately.
  • Content (SHA-256) mode reads large objects in 8 MiB ranges, retries a failed range from where it stopped, aborts and retries a read that receives no data for 60 seconds, and shows the bytes read.
  • Console and CLI: comparison history, status filters, paging, ZIP export and resume; new oss-transfer-cli object list, object status --status, object export and object resume.
  • Behaviour change: object compare with only keys now starts a single-object comparison and prints the result when it finishes within 15 seconds, otherwise its ID; --json prints the comparison record.
  • Compatibility: a new Master manages comparisons on older Agents (at most 10,000 stored differences, no resume). If you added a comparison block to an Agent's service.json, remove it before downgrading.

Full Changelog: v4.1.40...v4.1.41