TIKA-4801: Extract cover art from MP3, MP4, Vorbis and FLAC audio - #2983
Open
dschmidt wants to merge 5 commits into
Open
TIKA-4801: Extract cover art from MP3, MP4, Vorbis and FLAC audio#2983dschmidt wants to merge 5 commits into
dschmidt wants to merge 5 commits into
Conversation
Pictures embedded in ID3v2 tags (APIC in v2.3/v2.4, PIC in v2.2) are handed to the standard EmbeddedDocumentExtractor as inline embedded documents. The declared mime type becomes the embedded Content-Type, the picture description becomes dc:title and the APIC picture type is mapped to dc:description via the spec's label list. No new metadata properties are recorded on the audio document itself: consumers that only want to know whether cover art exists can use the embedded document count. Malformed picture frames (missing encoding, unterminated mime type or description, empty picture data) are skipped silently, matching the existing handling of malformed COMM frames. Linked v2.2 pictures (image format "-->") are skipped as they carry a URL, not image data. The fixture is a copy of testMP3id3v24.mp3 with a self-crafted APIC frame (minimal 1x1 PNG) inserted before the existing frames and the synchsafe tag size rewritten accordingly.
The covr ilst entry, previously skipped explicitly, is handed to the standard EmbeddedDocumentExtractor as an inline embedded document, one per data atom (the atom may repeat for multiple images). The data atom's well-known value type declares the image format (13 = jpeg, 14 = png, 27 = bmp); any other type is left for auto-detection. As in the ID3v2 commit, no new metadata properties are recorded on the audio document itself. The ParseContext needed for the embedded document extractor is plumbed from MP4Parser through TikaMp4BoxHandler into TikaUserDataBox. The fixture is a copy of testMP4.m4a with a self-crafted covr entry (minimal 1x1 PNG, type 14) appended to the ilst box and the ancestor box sizes patched.
METADATA_BLOCK_PICTURE comments, base64 encoded FLAC picture blocks as used by Vorbis and Opus (and shared by all Ogg family parsers through OggAudioParser.extractComments), are handed to the standard EmbeddedDocumentExtractor as inline embedded documents. The declared mime type becomes the embedded Content-Type, the description becomes dc:title and the picture type is mapped to dc:description through the ID3v2 APIC label list, which the FLAC picture spec reuses. The raw comment is excluded from the generic vorbis:* passthrough: a giant base64 metadata value helps nobody once the picture is an embedded document. Malformed or truncated blocks and linked pictures (mime "-->") are skipped silently. Native FLAC PICTURE blocks remain out of scope: vorbis-java 0.8 keeps them in FlacFile.otherMetadata without a public accessor. The fixture is a copy of testVORBIS.ogg with a self-crafted metadata_block_picture comment (minimal 1x1 PNG) inserted into the comment header packet, with the Ogg page lacing values and CRC rewritten.
vorbis-java 0.8 parses the native FLAC metadata blocks but offers no accessor for them (FlacFile.getOtherMetadata() exists on their master, unreleased since 0.8 in 2016), so FlacParser walks the metadata region itself through a second stream over the spooled file: fLaC magic, then per block 1 header byte (last-block flag plus type) and a 24 bit BE length, stopping at the last-block flag, at end of stream or at a block that declares more data than is left. The PICTURE block payload (type 6) is byte-identical to the metadata_block_picture comment, so the parsing and inline embedded-document emission are shared with OggAudioParser through the extracted extractPictureBlock helper, keeping the semantics of the previous commit unchanged. Scope is native FLAC only: FLAC-in-Ogg carries its pictures in metadata_block_picture comments, which the previous commit already handles, and its PICTURE packets stay out of scope. The fixture is a copy of testFLAC.flac from the standard integration tests with a self-crafted PICTURE block (minimal 1x1 PNG) appended after the previous last metadata block, whose last-block flag is cleared.
dschmidt
changed the title
dschmidt
marked this pull request as ready for review
July 30, 2026 13:57
An ID3v2.4 tag with several APIC frames only yielded the first picture. ID3v2.4 encodes frame sizes as synchsafe integers, but RawTag read them as plain integers, so any frame above 127 bytes, which every real cover picture is, made the frame walk overshoot into the middle of the following frames: with two covers the walk consumed the second APIC as trailing bytes of the first and stopped in the padding. The pictures were also emitted with that trailing garbage attached. RawTag now reads v2.4 frame sizes as synchsafe. Because widespread taggers (e.g. older iTunes) wrote plain integers into v2.4 tags, the two readings are disambiguated when they disagree: a size byte with the high bit set forces plain, otherwise the reading wins that lands the walk on a plausible next frame (frame id of capitals and digits, padding, or the tag end). Sizes below 128 bytes, like all frames in the existing v2.4 fixtures, are identical in both encodings. The ID3v2.3 (plain sizes) path, the MP4 covr repeats, the FLAC PICTURE walk and the Vorbis metadata_block_picture loop were checked against the same loss pattern and were already correct; multi-cover fixtures and tests for all of them are added as regression coverage. The MP3 fixtures were generated with ffmpeg -id3v2_version 4 / 3 (front 64x40 png, back 30x30 png, both above 127 bytes); the MP4, FLAC and Ogg fixtures extend the single-cover fixtures with a second image (MP4: one covr record with a png and a jpeg data atom).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes TIKA-4801: ID3v2 APIC frames (v2.2 PIC included), the MP4/M4A
covratom (previously explicitly skipped), Vorbis/Opusmetadata_block_picturecomments and native FLAC PICTURE blocks.Each picture is emitted through the standard EmbeddedDocumentExtractor mechanism as an inline embedded document, the same way PDF and Office parsers emit embedded images: /rmeta returns one metadata entry per cover with the declared content type (or auto-detection where MP4 only has a type code), the declared description as dc:title and the picture type as dc:description; /unpack delivers the bytes. Deliberately no new metadata properties on the audio document and no mapping of the width/height fields that Vorbis picture blocks declare: ID3 and MP4 declare no dimensions at all, and the embedded entry carries the real ones once image parsers are on the classpath.
The raw metadata_block_picture comment is no longer passed through as a giant base64 vorbis: value, it becomes the embedded document instead.
Native FLAC PICTURE blocks are read by a small metadata block walk in FlacParser itself: vorbis-java 0.8 offers no public accessor for the blocks (their master already has FlacFile.getOtherMetadata(), but nothing has been released since 0.8), so the parser walks the native metadata region through a second stream and reuses the shared picture handler. A TODO in the code links Gagravarr/VorbisJava#46 so the walk can be dropped once a release ships. FLAC-in-Ogg needs no extra path, pictures arrive there as metadata_block_picture.
One commit per format, each with its own fixture and end-to-end test; the fixtures are minimally modified copies of existing test files carrying a 1x1 PNG cover (for the Ogg fixture the comment packet was rewritten with recomputed page CRC). All four tests fail without their parser change.