AVBTool MOD v1.2.8

English

Highlights

Large Images No Longer OOM — mmap-backed FEC

The fec tool now memory-maps ordinary input files instead of loading
the whole image into a private heap copy. A 4 GiB image that used to
be a guaranteed out-of-memory kill now completes in ~45 s with an RSS
peak around a third of the input size, on a machine whose available
memory is smaller than the image. Non-ordinary inputs (pipes, block
devices) fall back to the previous malloc + fread path; output is
byte-identical either way.

• Before (v1.2.0): fec read the entire image into heap memory.
  An 8 GiB image on an 8 GiB machine was OOM-killed ~100 s in.
• After (v1.2.8): fec maps the file read-only. The process
  peak RSS is ~33 % of the input (page-cache-backed, reclaimable),
  not 1:1 with the image. Cold start (never-read input) behaves the
  same; the kernel reclaims cold pages under pressure instead of
  killing the process.

OpenMP FEC Encoding — up to ~4x Faster

The interleaved RS-8 column loop is now compiled with OpenMP
(-fopenmp). Columns are independent, so the encode is
schedule(static) and the output is byte-identical regardless of
thread count (verified sha256 across 1 vs 4 threads).

• 8 GiB cold image: serial ~176 s → 4 threads ~45 s (~3.9x, on a 4-core
  i5).
• Thread count is controlled by the standard OMP_NUM_THREADS
  environment variable; no CLI flag needed — by default the fec
  tool automatically uses every logical core.

--threads N for Level-0 Hashtree Hashing

add_hashtree_footer accepts --threads N (default 1, serial, matching
upstream). For N > 1, level-0 block digests are computed in
multiprocessing workers and stitched back by block offset, bypassing
the GIL on large images. Output is byte-identical to the serial path
independent of N or scheduling.

• 128 MiB image: serial ~3.3 s → --threads 4 ~2.7 s end-to-end
  (the hashing stage itself runs ~3.9x faster; the overall speedup is
  smaller because RSA signing and the FEC write are serial and
  un-parallelisable).

Bug Fixes

• --threads N>1 no longer crashes. The multiprocessing branch
  passed image.name, but the image is an ImageHandler whose
  attribute is .filename. Any --threads N>1 run crashed with
  'ImageHandler' object has no attribute 'name'. Fixed. (The
  serial path was unaffected.)
• --parallel renamed to --threads for the CLI flag (and the
  internal parameter). --parallel was ambiguous next to the OpenMP
  thread count used by the fec tool; --threads names it directly.

60-Byte libfec Footer

The fec tool now writes exactly sizeof(struct fec_header) (60 bytes)
after the raw parity — not a 4 KiB block. avbtool reads the last
struct.calcsize(FEC_FOOTER_FORMAT) = 60 bytes, so the footer now
sits at the correct byte offset and verification passes.

Nuitka-Only CI

The build matrix is now Nuitka-only (amd64 + arm64, 2 jobs).
PyInstaller was dropped. CI triggers on push to both main and
edge.

Compatibility Notes

• --threads N (was --parallel N) is the only intentional CLI
  deviation from upstream avbtool 1.2.0. Default N=1 preserves
  upstream serial behaviour; N > 1 output is byte-identical.
• fec is automatically multi-threaded via OpenMP; set
  OMP_NUM_THREADS=1 to force a single thread.
• Operating requirements: unpack the tarball and put its bin/ on
  PATH; the three binaries (avbtool, fec, openssl) are
  static/self-contained with no runtime .so dependencies.

---

中文

主要更新

大镜像不再 OOM — mmap 化 FEC

fec 工具现在对普通输入文件使用内存映射（mmap），不再把整个镜像
读进私有堆内存。一个原本必然触发 OOM kill 的 4 GiB 镜像，现在在
"可用内存小于镜像"的机器上也能跑完（~45 秒，进程 RSS 峰值约为输入
的 1/3）。非普通输入（管道、块设备）自动退回原来的 malloc + fread
路径，两种路径输出逐字节一致。

• 之前（v1.2.0）：fec 把整个镜像读入堆内存。8 GiB 镜像在
  8 GiB 机器上会在 ~100 秒时被 OOM kill。
• 现在（v1.2.8）：fec 只读映射文件。进程峰值 RSS 约输入的
  33%（页缓存支撑、可被内核回收），不再与镜像大小 1:1。冷启动
  （从未读过的文件）表现一致，内存紧张时内核回收冷页，而不是杀进程。

OpenMP FEC 编码 — 最高约 4 倍速

交织的 RS-8 列循环现在用 OpenMP（-fopenmp）编译。各列彼此独立，
采用 schedule(static)，输出与线程数无关、逐字节一致（1 vs 4
线程已用 sha256 验证）。

• 8 GiB 冷启动镜像：串行 ~176 秒 → 4 线程 ~45 秒（~3.9x，4 核 i5）。
• 线程数由标准 OMP_NUM_THREADS 环境变量控制，无需 CLI 参数 —
  默认 fec 工具自动使用全部逻辑核心。

--threads N 加速 Level-0 Hashtree 哈希

add_hashtree_footer 接受 --threads N（默认 1，串行，与上游一致）。
N > 1 时用 multiprocessing worker 计算 level-0 块摘要，再按块偏移
拼回，绕过 GIL 处理大镜像。输出与串行路径逐字节一致，与 N 和调度
顺序无关。

• 128 MiB 镜像：串行 ~3.3 秒 → --threads 4 ~2.7 秒（全流程；哈希段
  本身约 3.9 倍快，因 RSA 签名和 FEC 写盘不可并行，全流程加速较小）。

缺陷修复

• --threads N>1 不再崩溃。 multiprocessing 分支原来传了
  image.name，但 image 是 ImageHandler，属性叫 .filename。任何
  --threads N>1 的运行都会崩
  （'ImageHandler' object has no attribute 'name'）。已修复
  （串行路径本来就没问题）。
• --parallel 改名为 --threads（CLI 参数及内部参数同步）。
  --parallel 与 fec 工具的 OpenMP 线程数语义混淆，--threads
  更准确。

60 字节 libfec Footer

fec 工具现在在原始 parity 后精确写入 sizeof(struct fec_header)
（60 字节）—— 不再是 4 KiB 块。avbtool 读取最后
struct.calcsize(FEC_FOOTER_FORMAT) = 60 字节，footer 现在位于正确
字节偏移，校验通过。

Nuitka-only CI

构建矩阵改为 Nuitka-only（amd64 + arm64，2 个 job），移除
PyInstaller。CI 在 main 和 edge 两个分支的 push 上都触发。

兼容性说明

• --threads N（原 --parallel N）是 CLI 相对上游
  avbtool 1.2.0 的唯一有意偏离。默认 N=1 保持上游串行行为；
  N > 1 输出逐字节一致。
• fec 默认通过 OpenMP 自动多线程；设 OMP_NUM_THREADS=1 可强制
  单线程。
• 运行要求：解包 tarball 并把其 bin/ 加到 PATH；三个二进制
  （avbtool、fec、openssl）静态自包含，无运行时 .so 依赖。

AVBTool MOD v1.2.0

English

Highlights

Fully Self-Contained — No External Dependencies

This avbtool is a standalone, self-contained build. All cryptographic operations and FEC generation run in-process, with no OpenSSL binary and no external fec tool required.

• In-process RSA — RSA private/public key parsing and signing use the bundled cryptography library instead of shelling out to openssl. The frozen single-file executable has no external binary dependencies.
• --pass-file support — Load encrypted RSA private keys in-process via --pass-file. The passphrase is held in memory for the duration of the invocation only; the decrypted key is never written to disk.
• In-process FEC — Reed-Solomon forward error correction codes are generated directly inside avbtool, emitting the same 4 KiB libfec footer (magic 0xfecfecfe) as AOSP. calc_fec_data_size() and on-disk layout remain consistent.

--dynamic_partition_size

New option for add_hash_footer and add_hashtree_footer:

• partition_size is derived automatically from the image size after any existing footer is truncated, so repeated runs remain idempotent.
• The AVB footer and metadata are placed immediately after the image payload, with no manual partition size calculation needed.
• Combining --dynamic_partition_size with --partition_size or --calc_max_image_size is now a hard error.

Python 3 Compatibility and CLI Hardening

• Binary commands that defaulted to sys.stdout (generate_test_image, extract_vbmeta_image, ATX certificate/metadata/credential commands) now use stdout.buffer, eliminating a TypeError on Python 3.
• Missing required arguments (--image, --partition_size) produce clear error messages instead of NoneType tracebacks.
• set_ab_metadata creates a missing misc image, matching its documented behavior.
• The no-subcommand case prints a usage error; other attribute errors are no longer swallowed silently.

Secure Defaults

• add_hashtree_footer now defaults to sha256 hash (previously sha1), meeting Android 10+ dm-verity requirements out of the box. --hash_algorithm sha1 remains available for legacy devices.

Frozen Binary Packaging

• CI builds single-file executables for amd64 and arm64 using both PyInstaller and Nuitka, packaged in a single workflow.
• Each binary is named with version, architecture, builder, and Shanghai date stamp.

Compatibility Notes

• --dynamic_partition_size places the AVB footer immediately after the image. If the target partition is smaller than image_size + metadata + footer, the operation will fail with a clear error.
• The in-process RSA loader supports PEM and DER formats for both private and public keys. Keys with exponents other than 65537 are not supported.
• Operating requirements: Python 3.8+, or use the pre-built frozen binaries (no Python required).

---

中文

主要更新

完全自包含 — 无外部依赖

此 avbtool 为独立自包含构建，所有加密运算和 FEC 生成均在进程内完成，无需 OpenSSL 二进制文件，无需外部 fec 工具。

• 进程内 RSA — RSA 密钥解析和签名通过打包的 cryptography 库完成，不再调用 openssl。冻结单文件可执行程序无外部二进制依赖。
• --pass-file 支持 — 通过 --pass-file 在进程内加载加密 RSA 私钥，口令仅在单次调用期间驻留内存，解密密钥永远不会写入磁盘。
• 进程内 FEC — Reed-Solomon 前向纠错码直接在 avbtool 内生成，输出与 AOSP 相同的 4 KiB libfec footer（magic 0xfecfecfe），calc_fec_data_size() 与磁盘布局保持一致。

--dynamic_partition_size

add_hash_footer 和 add_hashtree_footer 新增选项：

• 自动从镜像大小推导 partition_size（在截断已有 footer 后计算），重复执行仍保持幂等。
• AVB footer 和元数据紧随镜像载荷之后放置，无需手动计算分区大小。
• --dynamic_partition_size 与 --partition_size 或 --calc_max_image_size 同时使用时会报错。

Python 3 兼容性与 CLI 加固

• 默认写入 sys.stdout 的二进制命令（generate_test_image、extract_vbmeta_image、ATX 证书/元数据/凭据命令）现在使用 stdout.buffer，消除 Python 3 下的 TypeError。
• 缺少必需参数（--image、--partition_size）时给出明确错误信息，不再抛出 NoneType traceback。
• set_ab_metadata 在 misc 镜像不存在时自动创建，与文档描述一致。
• 无子命令时打印 usage error，其他属性错误不再被静默吞掉。

安全默认值

• add_hashtree_footer 默认哈希算法从 sha1 改为 sha256，满足 Android 10+ dm-verity 要求，开箱即用。--hash_algorithm sha1 仍可用于旧设备。

冻结二进制打包

• CI 使用 PyInstaller 和 Nuitka 为 amd64 和 arm64 构建单文件可执行程序，统一在一个 workflow 中完成。
• 每个二进制文件以版本、架构、构建器和上海日期命名。

兼容性注意事项

• --dynamic_partition_size 将 AVB footer 紧接镜像末尾放置，若目标分区小于 image_size + metadata + footer 则会失败并给出明确错误。
• 进程内 RSA 加载器支持 PEM 和 DER 格式的私钥和公钥，不支持指数非 65537 的密钥。
• 运行要求：Python 3.8+，或直接使用预构建的冻结二进制（无需 Python 环境）。