Repository navigation
OSS Transfer v4.1.42
v4.1.42 以一位新用户从安装到完成迁移的完整体验为目标:按评估中发现的问题改进了安装、加入节点、存储连接、增量迁移、失败恢复、报告和卸载,并重写了文档。
文档
- 新增规划指南:拓扑与执行节点数量、端口与防火墙、源端和目标端所需权限(附最小权限策略示例)、速度与容量、大规模核对,以及增量迁移与割接流程。
- 新增安全指南:网络暴露、用 NGINX 提供 HTTPS、凭据如何存储和传递、密码与会话、加入邀请、以 root 运行的组件。
- README 快速开始写明每一步成功时应看到什么;安装、运维(按症状排查问题)、CLI、Docker Compose 等页面全面更新,中英文内容一致。
存储连接
- 自动寻址(默认):Endpoint 是 IP、localhost、不含点的主机名、以 .internal/.local/.lan 结尾或带端口时用 Path-style,否则用 Virtual-hosted。MinIO、SeaweedFS、Ceph 等自建存储不再需要手动选择。明确选择过的连接不受影响。
- 连接检查在列举存储桶后,还会用实际寻址方式访问一个存储桶,所以寻址错误在保存时就能发现。只授权到单个存储桶的密钥只会得到警告,可用“检查指定存储桶”或
storage test --bucket检查。 - 常见错误(密钥无效、签名不匹配、存储桶不存在、拒绝访问、无法解析域名、连接被拒绝或重置、超时、TLS)先显示一句可操作的说明,原始错误可展开查看或用
--verbose查看。
迁移可靠性
- 存储重启或网络抖动时,连接被拒绝或重置、请求中途断开、超时、DNS 失败和 5xx 会重试约 10 次(1~2 分钟),不再立即判为失败;读取源对象时连接中断,会从断开处续读。约 40 秒的存储重启不再导致对象失败。
- 扫描后在源端被删除的对象记为“已跳过(源端已删除)”,不再计为失败(仅对象存储源)。
- 扫描后发生变化的源对象,重试时读取其当前版本。
- 中断后未清理的分片上传会在之后自动中止。
- 下载到本地前的预检会实际写入测试文件,并报告可用空间。
- 任务详情直接给出失败的主要原因和离线节点。
- 执行节点的 service.log 会记录每次子任务的开始和结束、失败对象(每次最多 20 个,其余计数)、存储重试和与 Master 的连接变化,便于排查。
增量迁移:覆盖有变化的对象
- 新冲突策略“覆盖有变化的对象”(CLI
--conflict changed):目标不存在时复制;大小不同、两端都是普通 MD5 ETag 且不同,或源端修改时间晚于目标端时覆盖;其余跳过。不读取对象内容。 - 使用“跳过同名对象或文件”且有对象被跳过时,任务详情会提示:这些对象只按名称判断,可用对象比对核对。
- 任务详情新增“比对源和目标”,直接打开预填好的对象比对。
报告
- 停止后继续、重试失败对象之后,任务报告会列出每个对象的最终结果,不再出现大量“结果未知”。
task report会显示报告是否完整、各类结果的数量和不完整的原因;报告不完整时退出码为 3。--output -可把报告写到标准输出。
加入节点与 Docker
- 添加节点的 Docker 选项可直接用于 macOS/Windows 的 Docker Desktop,无需 root。每个邀请使用独立的容器名和数据卷(
oss-transfer-agent-<编号>)。 - 当 Master 地址是本机回环地址时会给出警告,并建议
host.docker.internal或局域网地址。 - 原生加入命令在 root 和普通用户(sudo)下都能直接运行。
- 版本更新页为 Docker 执行节点提供可复制的更新命令;版本已一致时提供“重建命令”,可用来增加目录挂载。
- 移除节点前会提示:保存在该节点上的任务报告明细和比对结果将无法再读取,请先导出。
安装、升级与卸载
- 新增
install.sh --uninstall,可配合--role、--keep-data、--yes使用;也支持安装脚本创建的 Docker Compose 安装。Windows 的手动卸载步骤见安装文档。 - 重复安装同一版本时默认跳过,并以“无需更改”结束;需要时加
--force。 - 安装前会显示安装计划(角色、版本、目录、服务名)。如果要安装的版本比现有数据所用的版本旧,会先警告;
--keep-data卸载后会给出带--version的完整重装命令。 - 被移除的执行节点会自行停止,不再反复重启。
oss-transfer-cli update master不再需要--version,会自动安装最新正式版。
控制台与 CLI
- 控制台:
- 新密码至少 8 个字符,不能与用户名、当前密码或初始密码相同,也不能是常见密码、单一重复字符或连续的数字、字母。
- 新增首次使用清单。
- 异常记录默认只显示未解决的记录,并适配手机屏幕。
- 版本号统一显示为 v4.1.42。
- 时间筛选会注明按哪个时区解释。
- CLI:
- 退出码统一:0 成功,1 操作失败,2 命令行用法错误,3 结果不完整,130 被中断。
- 状态、策略等代码翻译为可读文字(
--json仍输出原始代码),数字不再使用科学计数法。 - 写操作只输出一句结果摘要,错误不再重复显示。
- 连不上 Master 时会给出检查地址、端口和服务的提示;重复执行 stop/resume 时提示“无需处理”。
兼容性与升级
- 请先升级 Master,再升级执行节点。
- 混合版本期间:
- 旧版执行节点会收到自动寻址解析后的结果。
- “覆盖有变化的对象”只能分配给 v4.1.42 执行节点。
- 重试时读取源对象当前版本,需要 Master 和执行节点都是 v4.1.42;否则行为与 v4.1.41 相同。
- 已知限制:
- SSE-KMS/SSE-C 加密对象的 ETag 不是 MD5,“覆盖有变化的对象”可能会重复复制这些对象。
- 源端的删除不会同步到目标端。
- Install.ps1 暂不支持
-Uninstall。
v4.1.42 is about the whole experience of a new user, from installation to a finished and verified migration. Based on a new-user evaluation, it improves installation, adding nodes, storage connections, incremental migrations, failure recovery, reports and uninstallation, and rewrites the documentation.
Documentation
- New planning guide: topology and Agent count, ports and firewall, the permissions needed on the source and target (with minimal policy examples), speed and capacity, large-scale verification, and an incremental migration and cut-over runbook.
- New security guide: network exposure, HTTPS with NGINX, how credentials are stored and passed, passwords and sessions, join invitations, and what runs as root.
- The README quick start says what success looks like at each step. The install, operations (troubleshooting by symptom), CLI and Docker Compose pages are rewritten; English and Chinese match.
Storage connections
- Automatic addressing (default): path-style when the endpoint is an IP address, localhost, a name without a dot, ends with .internal/.local/.lan, or has a port; virtual-hosted otherwise. Self-hosted storage such as MinIO, SeaweedFS and Ceph needs no manual choice. Connections with an explicit choice are unchanged.
- After listing buckets, the connection check also reaches one bucket with the actual addressing, so addressing mistakes show up when you save. Keys limited to one bucket only get a warning; check that bucket with "Check a bucket" or
storage test --bucket. - Common errors are shown first as an actionable sentence: invalid key, signature mismatch, missing bucket, access denied, DNS, refused or reset connection, timeout and TLS. The raw error can be expanded, or shown with
--verbose.
Reliability
- When storage restarts or the network blips, the transfer retries about 10 times over 1-2 minutes before an object fails. This covers refused or reset connections, connections dropped mid-request, timeouts, DNS failures and 5xx responses. A source read cut off mid-object continues from where it stopped. A storage restart of about 40 seconds no longer fails objects.
- Objects deleted at the source after the scan are skipped as "deleted at source", not failed (object storage sources only).
- Source objects that changed after the scan are read again at their current version when retried.
- Multipart uploads left behind by an interruption are aborted later automatically.
- The precheck before downloading to a local directory writes a probe file and reports the free space.
- The task detail names the main cause of failures and any offline nodes.
- The Agent's service.log records each child task attempt's start and end, failed objects (the first 20 per attempt plus a count of the rest), storage retries and Master connection changes.
Incremental migrations: Overwrite changed objects
- New conflict policy "Overwrite changed objects" (CLI
--conflict changed). It copies a missing object, and overwrites when the sizes differ, when both ETags are plain MD5 ETags and differ, or when the source was modified after the target. Everything else is skipped, and object content is never read. - When a task with "Skip existing objects or files" skipped objects, the task detail explains that they were matched by name only and can be checked with object comparison.
- The task detail has a new "Compare source and target" button that opens a prefilled object comparison.
Reports
- After stop and continue, or after retrying failed objects, task reports list every object's final result instead of many "unknown" rows.
task reportshows whether the report is complete, the counts per result and the reasons it is incomplete, and exits 3 when it is incomplete.--output -writes the report to standard output.
Adding nodes and Docker
- The Docker option of Add node works on Docker Desktop for macOS and Windows without root. Each invitation uses its own container and volume name (
oss-transfer-agent-<id>). - A loopback Master address shows a warning that suggests
host.docker.internalor a LAN address. - The native join command runs as root or, through sudo, as a regular user.
- Software updates shows copyable update commands for Docker Agents, and "Recreate commands" when the versions already match, for example to add a directory mount.
- Removing a node warns that the task report details and comparison results stored on it will no longer be available, so export them first.
Install, upgrade and uninstall
- New
install.sh --uninstall, with--role,--keep-dataand--yes. It also removes Docker Compose installations made by the installer. Windows manual removal steps are in the install guide. - Reinstalling the same version is skipped and ends with "Nothing changed" unless you add
--force. - The installer shows an installation plan (roles, version, directories, service name) before it starts. It warns before installing a version older than the one that last used the data. After
--uninstall --keep-datait prints the complete reinstall command, including--version. - A removed Agent stops by itself instead of restarting in a loop.
oss-transfer-cli update masterno longer needs--version; it installs the latest stable release.
Console and CLI
- Console:
- A new password needs at least 8 characters and must differ from the user name and from the current and initial passwords. It also must not be a common password, one repeated character, or a run of consecutive digits or letters.
- New first-run checklist.
- Exception records show unresolved records by default and fit phone screens.
- Versions are shown as v4.1.42.
- Time filters name the time zone used.
- CLI:
- Consistent exit codes: 0 success, 1 operation failed, 2 usage error, 3 incomplete result, 130 interrupted.
- States and policies are shown as readable words, while
--jsonkeeps the codes. Numbers are no longer printed in scientific notation. - Write commands print a one-line summary, and errors are no longer repeated.
- When the CLI cannot reach the Master it suggests checking the address, the port and the service. Repeating
task stoportask resumereports "No action needed".
Compatibility and upgrading
- Upgrade the Master first, then the Agents.
- While versions are mixed:
- Older Agents receive automatic addressing already resolved.
- "Overwrite changed objects" only runs on v4.1.42 Agents.
- Reading the current version of a changed source on retry needs both the Master and the Agent on v4.1.42; otherwise the behaviour is that of v4.1.41.
- Known limits:
- ETags of SSE-KMS/SSE-C encrypted objects are not MD5s, so "Overwrite changed objects" may copy such objects again.
- Deletions at the source are not propagated to the target.
- Install.ps1 has no
-Uninstallyet.