v1.12.0

What's new

This release fixes a long list (but certainly not all) of cases where chisel could silently break a file or lose part of it, and it leaves digitally signed files alone. It also adds Photoshop documents, base64 images inside HTML/CSS/Markdown, and a batch of ZIP-based formats. And containers are now opened and rebuilt in parallel, which makes runs on many PDFs or archives much faster.

Behavior changes

• Digitally signed files are now left untouched and reported as skipped:

  • documents: signed PDFs, Office/ODF documents, legacy Office and MSI files;
  • packages: APK/JAR/XPI/VSIX/NuGet/IPA/EPUB, and APPX/MSIX even when unsigned, since their block map would break too;
  • Authenticode executables, S/MIME emails, XML signatures (e.g. XAdES e-invoices);
  • images, videos and audio with a C2PA manifest (JPEG, PNG, WebP, GIF, TIFF, JPEG XL, SVG, MP4/MOV/M4A, AVI, WAV, MP3).

  Pass --break-signatures to optimize them anyway, invalidating the signature. DKIM signatures of emails are not considered.

• -o mirrors the inputs' folder structure, relative to the deepest folder holding all of them. Files that can't be improved are copied unchanged. PATH is a file only when the input is stdin or a single file.

• XML is minified lexically: only whitespace inside tags and outside the root element changes, everything else stays byte-identical. Some formats (EPUB, GPX) gain less than before, because those gains came from altering the content (see the bug fixes below).

• JSON numbers are kept as written, and invalid JSON (e.g. with comments) is left as is.

• --no-meta keeps what changes how images look:

  • JPEG: the ICC color profile. Multi-image JPEGs also keep what their secondary images need: MPF, a reduced XMP and, for Apple HDR photos, the two EXIF tags that drive HDR rendering;
  • PNG: the color profile, gamma and HDR chunks (iCCP, sRGB, gAMA, cHRM, cICP, mDCV, cLLI), plus stereo and Android 9-patch chunks;
  • JPEG 2000: color specifications, channel definitions (alpha) and palettes. XML, XMP, GeoJP2 and GMLJP2 boxes, the resolution box and codestream comments go;
  • Ogg FLAC: its cover pictures, as FLAC files already did.

• PNG:

  • files with a chunk that can't be carried over safely (an unknown chunk not marked safe to copy) are left as is;
  • files with chunks tied to the color type (bKGD, sBIT, hIST, pCAL) keep their color type.

• Damaged JPEGs are left as they are: a JPEG whose image data libjpeg reports as damaged (e.g. cut short) was rewritten with libjpeg's own filling for the missing part, which other decoders may show differently. This also covers JPEGs inside PDFs and audio covers.

• Matroska/WebM: mkclean no longer rewrites MuxingApp, WritingApp and DateUTC.

• Ogg:

  • chained Vorbis and FLAC files (several streams one after another, like a saved web radio stream) are left as they are, except for their covers. Recompressed chained Vorbis lost audio in players built on FFmpeg, and chained FLAC lost every stream after the first;
  • Vorbis files keep their vendor string (OptiVorbis no longer appends its own tag) and their stream serial number, so the same file always gives the same output.

• Detection:

  • empty files are skipped;
  • .dwf, .cdr, .csl, .puz, .wal and .easm are no longer treated as zips by extension (real zips among them are still recognized by content);
  • .xsn goes to the CAB processor.

New processors

• PsdProcessor for Photoshop documents (.psd, .psb):
  • RLE channel data (layers and composite image, 8/16/32-bit) is re-encoded with an optimal PackBits encoder;
  • the JPEG thumbnail and the files embedded in smart objects (PNG, JPEG, nested PSDs, SVG...) are optimized and put back;
  • -0.9% to -1.2% on real Photoshop files; smart-object PNGs lose about a third of their size.
• DataUriProcessor for HTML, CSS and Markdown:
  • base64 data: images and fonts are optimized and put back in place;
  • every byte outside the payloads stays untouched;
  • -4% on average on 30 real HTML exports, up to -33%;
  • XML and SVG get the same treatment.

New formats

• Office Open XML packages: XPS/OXPS, DWFx, 3MF, Visio (.vsdx, .vsdm, .vssx, .vssm, .vstx, .vstm), Excel binary workbooks and add-ins (.xlsb, .xlam, .ppam), PowerPoint slides (.sldx, .sldm) and Office themes (.thmx). Measured -18% to -28% on Visio, 3MF and theme files.
• OpenRaster (.ora) and KMZ.
• Apple Mail .emlx, with its length line updated.
• The .vcard extension.

Improvements

• Containers are opened and rebuilt in parallel. Extracting containers and putting them back together used to run on one core: on 136 PDFs the rebuild took almost 6 hours of a 6h13m run. Now each container is rebuilt as soon as the ones inside it are done, on all threads, and the same run takes 70 minutes. Peak memory grows with it (1.8 GB to 2.6 GB with 10 threads).
• The console lists the containers found in each input together, as a tree, once that input has been analyzed.
• PDF images stored with Flate go through the PNG optimizer, keeping their color type and bit depth. They come back with the PNG row filters that make them smallest, or with none. -5% on 312 real PDFs, up to -37%, on top of what chisel already saved.
• PDF Flate streams that use a predictor (/DecodeParms) are recompressed too; before they were left as they were.
• Small JPEGs are also encoded as baseline, which beats progressive below about 20 KB: -10% on those, e.g. the images inside PDFs and Office documents.
• JPEG secondary images (Ultra HDR and Apple HDR gain maps, MPO views) are recompressed too: gain maps shrink by about 43%.
• --dry-run now reports the size containers would really reach.
• MIME files (.eml, .mht...) keep their own line breaks when attachments are re-encoded.
• Ogg FLAC covers stored as picture blocks are optimized too, like normal FLAC files.
• JPEG 2000 images with a palette are recompressed too, with their palette and indices untouched.
• Large PDFs need much less memory, thanks to qpdf 12.4.2: on a 1,240-page book chisel peaked at 78 MB instead of 187 MB.

Bug fixes

Data loss

• JPEG: everything after the first image was dropped.
  • HDR photos lost their gain map (Ultra HDR, Apple HDR), MPO files their second view and motion photos their video.
  • Secondary images are now kept and MPF offsets updated.
  • --verify-checksums checks secondary images too.
• PNG:
  • 16-bit images were saved as 8-bit, losing the low byte of every sample; --verify-checksums compared them at 8 bits and missed it. They now stay 16-bit, unless every sample fits in 8 bits exactly;
  • data after IEND was dropped, e.g. a PNG with a zip appended lost the zip;
  • chunks were dropped even with metadata kept: EXIF (eXIf), HDR metadata (cICP, mDCV, cLLI), Android 9-patch data, and any chunk chisel didn't know about.
• Full disk: many processors didn't check that their output had been written in full. On a full disk, a truncated output could look smaller and replace the original. Every output and every extracted file is now checked.
• Replacing files on macOS: chisel keeps its temporary files in the system temp folder there.
  • For files on another drive, or when the rename was refused (sandboxed folders), chisel truncated the original and then copied the new data into it. A copy failing midway left the original truncated.
  • The new file is now copied next to the original and renamed over it.
  • Where that isn't allowed, the original is overwritten without truncating it first, and restored from a backup if that fails.
• Archives: entries were lost when rebuilding.
  • Entries whose names differ only by case, on macOS and Windows: a real APK lost 42 of its 866 entries. This is typical of obfuscated APKs and JARs.
  • On Windows, entries whose extraction path went past 260 characters (4 of the 990 in a Kotlin JAR).
  • Archives are now rebuilt from a manifest of their entries. Entries that can't be written under their own name are extracted under a short one.
  • Kept as they were: order, dates, permissions, owners, links, FIFOs, stored entries, and the tar/cpio/ar variant.
  • Zips made on Windows still change a little: DOS times are written in the local time zone (the UTC time, when the zip has one, is kept), permissions become the default 664, and the NT security descriptor extra field is dropped. On macOS, __MACOSX/ entries are folded into the files' metadata by libarchive, so rebuilt zips lose them.
  • BSD static libraries are no longer broken, and .deb packages stay GNU ar.
• -o:
  • files extracted from containers leaked into the output folder;
  • those loose copies were optimized, instead of the ones put back in the container;
  • with a single container and -o FILE, FILE became one of its children (e.g. the cover art of a FLAC).
• XML: the old minifier changed the content.
  • Words were glued together in DOCX, EPUB and SVG text.
  • DOCTYPE entities were escaped, so legacy Illustrator SVGs stopped rendering.
  • UTF-16 files were re-encoded as UTF-8.
  • Line-wrapped base64 images (Inkscape) were truncated.
• JSON: numbers were rounded (19.90 became 19.9, long decimals lost digits).
• MNG/JNG:
  • MNG files were cut after their first image, losing every later frame (since 1.10.0); --verify-checksums stopped at the same point and missed it;
  • a corrupt JPEG inside a JNG made chsl exit in the middle of a run;
  • the JPEG inside a JNG lost its markers and became progressive while the JNG header still said sequential.
• JPEG 2000: the whole file was rebuilt from the decoded image, so everything the decoder doesn't carry over was lost. Now only the codestream changes and every other box is kept byte for byte.
  • Color spaces other than sRGB, gray, sYCC and CMYK (Lab, e-sRGB, "any ICC" profiles) were dropped: colors changed, and macOS Preview couldn't open the result.
  • Premultiplied alpha became straight alpha, and the alpha channel of CMYK images lost its role.
  • Resolution, XML, XMP, GeoJP2 and GMLJP2 boxes, and any codestream after the first, were dropped.
• FLAC and other audio:
  • ID3v1 and APEv2 tags at the end of FLAC files were dropped;
  • ID3v2 tags in front of FLAC, APE, WavPack and Musepack files were dropped;
  • FLAC files whose metadata couldn't be read in full silently lost it; they're now left as is.
• Ogg FLAC: recompression dropped every metadata block (tags, pictures, cue sheets, application data). They're now kept byte for byte; only padding and the seek table, stale after re-encoding, go.
• MKV/WebM, fixed in our libmkclean fork:
  • a WebM processed after an MKV came out as an unplayable Matroska file;
  • WebM lost its color and HDR metadata;
  • negative integers were corrupted, so Opus padding got played back;
  • on ARM64, every rewritten floating-point value had its last byte altered.
• PDF:
  • JPEG 2000 images with a palette were re-encoded with the palette applied, so pages that use them through an indexed color space rendered with the wrong colors;
  • attachments kept their old /Size and /CheckSum after being optimized.
• Zip mimetype entry: EPUB, ODF and OpenRaster files got an extra field on it, which these formats forbid (epubcheck PKG-005).
• Rainmeter: .rmskin packages lost the footer their installer needs.
• Cover art: FLAC and Ogg covers in GIF or BMP, or with the wrong MIME type, got zeroed dimensions.

Metadata changed

• JPEG: libjpeg added a JFIF segment to JPEGs that had none (about a quarter of real files) and an Adobe one to RGB and CMYK JPEGs, and APPn/COM segments were reordered. The original segments are now kept, in their order.
• Audio covers, when put back after being optimized:
  • MP3, WAV, AIFF, DSF and DSDIFF: the ID3v2 tag was rewritten as v2.4, and MP3s got an ID3v1 tag they didn't have. The version is now kept (v2.2 becomes v2.3, the oldest TagLib can write), and TTA files with an older tag are left as they are;
  • non-ASCII picture descriptions were garbled (MP3, FLAC, Ogg, ASF);
  • the tags of FLAC, Ogg Vorbis, Opus, Ogg FLAC and Speex files were sorted by name and upper-cased;
  • WAV and AIFF: the ID3 chunk, and in WAV files the bext and iXML chunks, were moved to the end of the file.
• MP3: when the audio was repacked:
  • the two checksums in the LAME tag weren't updated, so tools that check them saw a damaged tag (the one that carries the gapless playback info);
  • the frame CRCs were dropped, and the Xing/Info frame was rebuilt, losing any extension chisel didn't know. Both are now kept; the CRCs cost about 0.5% on the rare files that have them.
• JPEG 2000: OpenJPEG signed every recompressed codestream with a comment of its own.

Errors and reporting

• Audio:
  • MPEG Layer I and II files (.mp2...) were taken for MP3 and logged [ERROR]; they're now left as is, and their covers are still optimized;
  • FLAC, APE, WavPack, TTA and Musepack files starting with an ID3v2 tag were taken for MP3 and logged [ERROR] (since 1.7.0), also when zero padding followed the tag;
  • Speex files, and Ogg FLAC files in the pre-1.1.1 layout, logged [ERROR]; they're now left as is, except for their covers.
• JPEG: libjpeg's warnings (e.g. Corrupt JPEG data: premature end of data segment) were printed straight to the console, even at the default log level and without the file name. They now go to the log, and a damaged JPEG is reported as a warning that names it.
• Warnings: every unsupported file (e.g. .lrc lyrics) and every audio file without cover art logged a warning; these are now debug messages.
• JPEG 2000: --verify-checksums read past the end of subsampled components, failing with a false INTEGRITY CHECK FAILED (and crashing debug builds).
• PDF:
  • raw image streams were extracted and misdetected as MP3, logging errors; only whole files are extracted now (JPEG, JPEG 2000, XMP, attachments);
  • qpdf warnings are no longer logged as errors.
• --verify-checksums:
  • it also checked outputs that were going to be discarded, so files left as is could fail it (e.g. invalid XML, JSON or PSD);
  • XML with control characters, which XML 1.0 forbids (e.g. the XMP metadata written by some scanners), failed it. That XML is now left as is.
• LibreOffice: every document logged an [ERROR], because of an empty file inside it.
• Encrypted Office documents logged errors.
• Zip extensions: files with a zip extension that aren't zips logged [ERROR] (old CorelDRAW, crosswords, Quake textures, eDrawings), and so did the riffData.cdr inside every CorelDRAW X4/X5 file.
• Reported sizes: for files handled both as a file and as a container (e.g. XML, SVG), the console and the CSV report showed the original size even after they had been shrunk.
• Temporary files:
  • a failed in-place replacement wasn't reported, and left its temporary file next to the original (outside macOS);
  • MIME and PE processing left temporary folders behind;
  • stopping a run (Ctrl-C) left every extracted container in the temp folder, hundreds of MB on a large run.
• Library (Chisel):
  • containers' final results never reached ChiselObserver;
  • files skipped at analysis (signed, empty, unsupported...) weren't reported. They now go to the new onFileSkipped(path, reason);
  • each recompress() call on the same Chisel subscribed the callbacks again, so after N calls every event reached the observer N times.

Dependencies

• Updated libmkclean fork: the Matroska fixes above, plus --keep-info.
• Updated qadmimes:
  • MP3 detection no longer matches raw data starting with FF Fx;
  • vCard and Photoshop files are detected by their content;
  • FLAC, APE, WavPack, TTA and Musepack files are recognized behind ID3v2 tags, with or without zero padding after them.
• Updated flacoutcpp: what follows the audio (ID3v1, APEv2 tags) is kept, and metadata that can't be copied stops the file instead of being dropped.
• Updated qpdf fork to qpdf 12.4.2: sturdier handling of damaged and malicious PDFs, and linearization with far less memory on large documents.
• Updated xz to 5.8.4, with a security fix in the .xz, .lzma and .lz decoders (GHSA-5qpq-xqfv-j9pg).
• Updated libpng with the fixes of 1.6.59.
• Updated optigif (faster) and mp3packercpp (the MP3 fixes above).

Development

• -DCHISEL_TSAN=ON (with -DCMAKE_BUILD_TYPE=RelWithDebInfo) builds chisel with ThreadSanitizer, into bin/RelWithDebInfo.

---

macOS: removing the Gatekeeper quarantine

When downloading binaries from the internet, macOS automatically marks them as “quarantined”.
If you try to run chsl and macOS shows a warning like:

“chsl cannot be opened because it is from an unidentified developer”

…you need to remove the quarantine attribute manually.

How to remove the quarantine flag
Open Terminal in the folder where you downloaded chsl and run:

xattr -d com.apple.quarantine ./chsl

To verify that the quarantine flag is gone:

xattr ./chsl

If com.apple.quarantine no longer appears, you're good to go.
Now you can run:

./chsl ...

Note
This issue does not occur if you compile chsl yourself.