itms-services / OTA installs broken by raw ZIP copy in #506 (regression of #469 / #433)

[iosconstantine]

Summary

PR #506 (v2.5.0) switched replicateZip to OpenRaw/CreateRaw. Apps installed
over the air (itms-services / MDM / OTA) then fail to install — a regression of
#469, which fixed exactly this for #433.

Symptom

The produced .ipa installs fine over USB (ideviceinstaller) but an OTA
install reaches ~100% and then iOS demotes the app to a placeholder ("cloud")
icon. Device syslog:

SZExtractorErrorDomain Code=3 "Finishing an incomplete stream! This stream will not be resumable"
  -[StreamingUnzipState finishStream]

USB works because ideviceinstaller unpacks only after the whole file has
landed; itms-services streams and unzips as it downloads.

What differs (measured)

Running the same App Store base through both replicateZip implementations and
parsing the local file headers:

local file header	Apple original	CreateRaw (#506)	CreateHeader (#469)
general-purpose bit 3	clear	clear	set
CRC / sizes inline	yes	yes	zero (deferred)
trailing data descriptor	none	none	present

CreateRaw output is byte-identical to Apple's original (a verbatim copy).
CreateHeader recompresses each entry and re-frames it with a trailing
data-descriptor record (PK\x07\x08, general-purpose bit 3 set).

Observed behaviour

Only the CreateHeader (recompressed / data-descriptor) form installs over the
air; the verbatim CreateRaw form fails with the syslog above. App Store IPAs are
never OTA-installed as-is — the tool injects the sinf and repackages first, and
that repackage historically used CreateHeader, so the inline form first reached
the OTA path with #506, which is when OTA installs broke.

Most likely cause

iOS's OTA streaming installer (SZExtractor / StreamingUnzip) reads
front-to-back without the central directory and appears to finalize each entry on
the trailing data descriptor that only the recompress path emits; the
descriptor-less verbatim form is left unterminated (Finishing an incomplete stream). I can't fully isolate this from the recompression itself —
CreateHeader changes both the framing and the compressed bytes — so this is the
observed-effect explanation, not a proven mechanism.

Fix

Restore the Open/CreateHeader path (as #469). The cost is CPU to recompress;
if that regression matters, the raw fast path could be kept behind an opt-in flag
with CreateHeader as the OTA-safe default.

A branch with the fix + a unit test (asserts a Deflate entry is re-framed with a
data descriptor and round-trips; go test ./... passes) is here, since PRs are
limited to collaborators:
https://github.com/iosconstantine/ipatool/tree/fix/itms-ota-raw-zip

Refs: #433, #469, #506.

[jpstotz]

It is funny that Apple uses a ZIP streaming implementation that can't handle their own ZIP files.

Anyway it seems that there are good reasons to support both variants:

1. with disabled recompression after download if you want to get the original ZIP file (which is a fully valid ZIP file)
2. with enabled recompression for OTA-installation

Therefore my suggestion would be adding a command-line option to enable recompression if you need it (not sure how much users use OTA installation).

[DeadSkullzJr]

I feel like the byte-identical output brought by #506 should remain the default option, whilst recompression / ota compatible options should be secondary via a specified command flag (something like --ota-compat for example). The more accurate outputs should always take priority above other options. Considering issue #510 exists, it's clear that regardless of the options, there are still some formatting issues that need to be tackled.

[iosconstantine]

Thanks both, agreed on the shape. Raw copy stays the default, recompression is opt-in.

I've built exactly that on a branch: https://github.com/iosconstantine/ipatool/tree/feat/ota-compat-flag

It's additive to main and doesn't revert #506. Default output is unchanged (still OpenRaw/CreateRaw, byte-for-byte with what Apple serves). New download --ota-compat flag re-encodes each entry through Open/CreateHeader so entries get general-purpose bit 3 set plus the trailing data descriptor, which is what the OTA streaming installer needs. Went with the name DeadSkullzJr suggested. The flag is only on download since that's the only command that writes the IPA (--purchase goes through the same command anyway).

There's a test that checks both directions on the same App Store-shaped archive: default keeps bit 3 clear and copies verbatim, --ota-compat sets bit 3, writes the descriptor, and the content round-trips. go test ./... passes. #510 isn't affected either since the raw default is untouched.

Happy to rename the flag or move it if you'd rather have it somewhere else.

[DeadSkullzJr]

Thanks both, agreed on the shape. Raw copy stays the default, recompression is opt-in.

I've built exactly that on a branch: https://github.com/iosconstantine/ipatool/tree/feat/ota-compat-flag

It's additive to main and doesn't revert #506. Default output is unchanged (still OpenRaw/CreateRaw, byte-for-byte with what Apple serves). New download --ota-compat flag re-encodes each entry through Open/CreateHeader so entries get general-purpose bit 3 set plus the trailing data descriptor, which is what the OTA streaming installer needs. Went with the name DeadSkullzJr suggested. The flag is only on download since that's the only command that writes the IPA (--purchase goes through the same command anyway).

There's a test that checks both directions on the same App Store-shaped archive: default keeps bit 3 clear and copies verbatim, --ota-compat sets bit 3, writes the descriptor, and the content round-trips. go test ./... passes. #510 isn't affected either since the raw default is untouched.

Happy to rename the flag or move it if you'd rather have it somewhere else.

Nice work! One thing to mention, since you brought back the changes prior to #506 as a flag option, you now have issue #516 to tackle. I left a comment there that confirms that extracting and re-compressing the contents fixes the issue. Whether or not the end result works via OTA, I don't know. I've never installed applications that way before, and honestly have no idea how that's done lol. How ipatool re-compressed contents before needs to change.

[iosconstantine]

@DeadSkullzJr you were right, thanks for catching that. I went back and checked: the old CreateHeader path really does write a data descriptor for stored (method 0) entries too, and a stored entry with bit 3 set is exactly what strict streaming readers choke on in #516 ("only DEFLATED entries can have EXT descriptor"). So my first cut of --ota-compat would have brought that back.

I've fixed the branch. The recompression path now only touches DEFLATE entries. Those get re-encoded with bit 3 set and the trailing data descriptor the OTA installer wants. STORED entries and directories are copied through verbatim with their sizes inline and no descriptor. So no stored entry ever carries a descriptor, which is what #516 needs, while deflated entries still carry the one the OTA installer needs. The default (non-flag) output is untouched and still raw/byte-identical, so nothing here reverts #506.

Verified two ways:

• Parsed the local file headers of a real App Store IPA produced with the flag. Every deflated entry has bit 3 set and a trailing PK0708 descriptor. Every stored entry (directories plus files like com.apple.FixedZipMetadata.bin and PkgInfo) has bit 3 clear, inline sizes, and no descriptor. A full CRC read of every member passes. There's also a unit test on the branch that asserts both directions and fails if a stored entry ever gets a descriptor.
• Installed the flag-built IPA over the air (itms-services) on a real device. It installed cleanly and the icon came up normally, no stuck cloud/placeholder icon and no SZExtractorErrorDomain stall. So the mixed framing works in practice, not just on paper.

Branch: https://github.com/iosconstantine/ipatool/tree/feat/ota-compat-flag

I can keep all of this folded into the same branch and flag if that's easier to review, or split the #516 part out, whichever you prefer.

[DeadSkullzJr]

@DeadSkullzJr you were right, thanks for catching that. I went back and checked: the old CreateHeader path really does write a data descriptor for stored (method 0) entries too, and a stored entry with bit 3 set is exactly what strict streaming readers choke on in #516 ("only DEFLATED entries can have EXT descriptor"). So my first cut of --ota-compat would have brought that back.

I've fixed the branch. The recompression path now only touches DEFLATE entries. Those get re-encoded with bit 3 set and the trailing data descriptor the OTA installer wants. STORED entries and directories are copied through verbatim with their sizes inline and no descriptor. So no stored entry ever carries a descriptor, which is what #516 needs, while deflated entries still carry the one the OTA installer needs. The default (non-flag) output is untouched and still raw/byte-identical, so nothing here reverts #506.

Verified two ways:

* Parsed the local file headers of a real App Store IPA produced with the flag. Every deflated entry has bit 3 set and a trailing PK0708 descriptor. Every stored entry (directories plus files like com.apple.FixedZipMetadata.bin and PkgInfo) has bit 3 clear, inline sizes, and no descriptor. A full CRC read of every member passes. There's also a unit test on the branch that asserts both directions and fails if a stored entry ever gets a descriptor.

* Installed the flag-built IPA over the air (itms-services) on a real device. It installed cleanly and the icon came up normally, no stuck cloud/placeholder icon and no SZExtractorErrorDomain stall. So the mixed framing works in practice, not just on paper.

Branch: https://github.com/iosconstantine/ipatool/tree/feat/ota-compat-flag

I can keep all of this folded into the same branch and flag if that's easier to review, or split the #516 part out, whichever you prefer.

If you wouldn't mind. Would you use your build of ipatool sporting the recent changes, and download the United Airlines application (based on issue #510)? I know that's oddly specific, but I want to see if your recent changes actually impacts the date and timestamps of certain assets (CodeResources in this case). ipatool despite being byte-identical by default now, still has date and timestamp quirks, and I don't know if that will still carry over with the new OTA flag compression changes you made. IPA files provided through iTunes 12.6.5 do not seem to have this issue as indicated in #510. I left a comment there mentioning the application identifier and the version identifier.

[iosconstantine]

@DeadSkullzJr done. I pulled United (449945214, external version id 888081233) with this build twice, once default and once with --ota-compat, and also grabbed the untouched original off Apple's CDN so there's a known-good reference to diff against. Everything below comes from parsing the zip headers directly, not from what an extractor happens to display.

Short answer: yes, it changes the timestamp on _CodeSignature/CodeResources, and here it changes it for the better. That entry is deflate, so --ota-compat re-encodes it through CreateHeader instead of copying it verbatim. The default output has no timestamp extra on that entry at all, so extractors show the bare DOS digits, 16:58:26, as local time regardless of timezone. That's the "wrong" value from #510. The ota-compat output gets a 0x5455 extended timestamp of 2026-07-10 16:58:26 UTC appended, so anything that honors UT shows 09:58:26 under UTC-7, which lines up with the iTunes 12.6.5 copy. zipinfo confirms it: raw shows 16:58:26 in any timezone, ota-compat shows 09:58:26 with TZ=America/Los_Angeles.

I wouldn't call that a #510 fix though. It's a side effect of the recompress path, and it only lands on the right answer because Apple happened to write the DOS digits as UTC wall clock (every sibling entry's Unix mtime matches its DOS digits within the 2 second rounding). It also touches other entries: anything that does carry a Unix extra gets its DOS field rewritten from that value and truncated to even seconds (the main binary went from 16:58:26 to 16:58:24 in the DOS field while its new UT says 16:58:25), and the ipatool-written iTunesMetadata.plist, which has a zeroed DOS date, comes out of the recompress path with a nonsense date. So it shifts timestamps across the archive rather than fixing the one that was wrong.

The actual root cause is the useful bit. Apple's original stores the real Unix mtime (2026-07-10 16:58:25 UTC) in an Info-ZIP UX extra (0x5855) on the local file header only. The central directory record for CodeResources has no extra field at all. Go's archive/zip reader fills FileHeader.Modified and FileHeader.Extra from the central directory and never reads the local header extra (it only uses the local header to find the data offset). So by the time the default path calls CreateRaw, the extra it copies through is already empty and the Unix time is gone. ipatool never sees it, so nothing it does today can preserve it.

A proper fix for the default path would be one of:

• read the local header by hand at the data offset and carry its extra through, since the high-level zip API doesn't expose it, or
• synthesize a 0x5455 from the DOS digits for entries whose central record has no extra. That's only correct if Apple always writes those digits as UTC, which held for everything I checked here, but it's still an assumption.

Happy to post more of the raw header dumps if that helps.