fix(filetype): emit each identity tag once, under the File group - #650
Merged
Conversation
oxidex emitted FileType/MIMEType twice with contradictory values on any file handled by the plain-text fallback: Geotag.log reported File:FileType "Unknown" and File:MIMEType "application/octet-stream" beside a bare FileType "TXT" and MIMEType "text/plain", and nothing downstream could say which was meant. Roughly forty parsers insert those tags ungrouped while the File:-grouped ones come from extract_file_metadata and add_identity_tags, and no step reconciled them. normalize_identity_tags folds the parser's answer into the File group and drops the ungrouped keys. A real File: value is kept -- it came from ExifTool's own tables, which outrank a parser's private spelling (WebP where ExifTool says WEBP) -- so this never invents a value, it decides which of two the output already contained survives. Two supporting corrections the corpus forced: FileTypeExtension follows the file type, not the filename. extract_file_metadata echoes the on-disk extension, so a .log file reported "log" where ExifTool reports "txt". It now comes from ExifTool's table via filetype::canonical_extension, which declines on names the tables do not carry rather than lowercasing a guess. The text fallback could not tell XML from plain text, so .gpx/.kml/.xml were named TXT. xml_file_type implements XMP.pm:4344-4426 -- <?xml is XML unless <rdf:RDF or x:xmpmeta makes it XMP. A DOCTYPE or <?aid returns None and the file stays TXT rather than being guessed at. Also drops "gz" => "TAR" from the hand-written extension table. A gzip stream is not a tar archive; the error was invisible while the GZIP parser published a second, ungrouped FileType, because the output carried both answers. Identity sweep over ExifTool's t/images corpus (194 files, pinned 13.59): File:-group matches 165 -> 176, per-tag matches 533 -> 562, ungrouped duplicates 50 -> 0, contradictory pairs 28 -> 0, regressions 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n-25f44a # Conflicts: # src/core/file_metadata.rs # src/filetype/mod.rs
swackhamer
added a commit
that referenced
this pull request
Aug 10, 2026
* fix(tags): report a file's size under one key, not three
`oxidex -json` reported the size of a plain-text file three times over:
"File:FileSize": "785 bytes", <- what ExifTool 13.59 reports
"FileSize": "785", <- raw byte count, ungrouped
"TEXT:FileSize": "785",
`extract_file_metadata` owns `File:FileSize` and formats it as ExifTool
prints it. 41 parsers additionally recorded `reader.size()` under a bare
`FileSize`, and the text parser mirrored that into `TEXT:FileSize`, so one
fact arrived under three keys with two spellings, two of which disagree
with the oracle. Under a group-aware sweep of ExifTool's `t/images` (194
files) 51 files carried the duplicate, 20 of them all three keys.
The dedup goes in `operations::drop_redundant_file_size`, after the one
merge where parser tags enter, because it is 41 parsers and not one. It
removes only the exactly-ungrouped key and only once `File:FileSize`
exists -- a badly-formatted answer still beats no answer.
Grouping alone does not make a distinct fact, so every grouped lookalike
was checked against the pinned oracle rather than inferred from its name.
`XML:FileSize`, `File:DPXFileSize` (12812288 against a 2.1 kB file),
`Prefetch:FileSize` and `LNK:TargetFileSize` are all real and untouched.
`EXE:FileSize` was the opposite -- `reader.size()` again, wearing a group
prefix that hid it from the `insert("FileSize"` search, and a tag ExifTool
emits for no Mach-O -- so it is removed at its source.
Also drops the `TEXT:` alias scheme. ExifTool 13.59 has no `TEXT` group:
the family-0 group of `Image::ExifTool::Text::Main` is `File`, so the
oracle reports `File:MIMEEncoding`, `File:Newlines`, `File:LineCount`,
`File:WordCount` and `File:ByteOrderMark`. The aliases emitted 101 keys
the oracle never produces under any group, each a second copy of a fact
already present, under names it does not even use (`LineEnding` for
`Newlines`, `HasBOM` for `ByteOrderMark`).
`FileType`/`MIMEType` are deliberately left duplicated: for a `.log` the
parser's ungrouped `FileType: TXT` is *correct* while `File:FileType`
still reads `Unknown`, so folding it away would delete the only right
copy. That is a detection gap, not this one.
Verified with a group-aware sweep (`scripts/compare_file.py` compares by
bare name, so it cannot see a group-placement defect and reported
`MISSING 0` throughout). Over all 194 files, per file, not just in total:
File:MATCH held at 2223, and no file's File:WRONG (40) or File:MISSING
(352) set moved. Every file now reports exactly one size key, down from
{1: 143, 2: 31, 3: 20}. The only keys that disappeared anywhere were 49
bare `FileSize` and 101 `TEXT:*`; none were added.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(tags): point the size dedup at its now-merged siblings
#648 and #650 landed while this was in review and moved the ground under
two comments here.
`drop_redundant_file_size` claimed `FileType`/`MIMEType` were "not folded
away" and called it a detection gap. #650 folds them, in
`normalize_identity_tags`, and #648 fixed the detection -- `Geotag.log`
now reports `File:FileType "TXT"`, not `Unknown`. Says instead why the two
stay separate functions: identity has rival *answers* to arbitrate and a
parser can name a type the tables left `Unknown`, whereas both size keys
report one byte count and the parser's is merely the unformatted spelling,
so it can never fill a gap in `File:FileSize`.
The text-parser note said relocating those facts into `File:` "needs the
FileType gate first". The gate now exists: `File:FileType` is the oracle's
answer on 12 of the 13 text-family files (XMP.xml still reads TXT). Records
that, and ExifTool's actual rule from `Text.pm` `ProcessTXT`, for whoever
picks it up. The nine-file figure still holds -- format dispatch continues
to route AFM, XML, JSON, RTF, URL and INX files through this parser.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
swackhamer
added a commit
that referenced
this pull request
Aug 10, 2026
…iscard #650 and #654 left one answer per identity fact in the output, both by discarding what the parsers wrote: `normalize_identity_tags` removes an ungrouped `FileType`/`FileTypeExtension`/`MIMEType`, and `drop_redundant_file_size` removes an ungrouped `FileSize`. Neither touched the source. 45 inserts across 40 parsers therefore still computed values that could not reach a caller, and the parsers still read as though `FileSize` and `MIMEType` were theirs to report -- the trap that produced the duplicates in the first place. Removes them. `FileType` deliberately stays: `normalize_identity_tags` promotes it into an absent or `Unknown` `File:FileType`, and that fallback is the only reason SQLite, EVTX, Registry Hive, HDF5, X.509 and ELF are named at all -- ExifTool's tables have no row for any of them, and it refuses to open four outright. Both normalizers stay too, as the backstop for anything that reintroduces one. fits.rs is the one behavioural loose end. It wrote `File:FileType`, `File:FileTypeExtension` and `File:MIMEType` directly -- hardcoded literals, not table lookups -- which put it out of both normalizers' reach, since they only ever removed the ungrouped spellings. `merge` gives format metadata precedence over the `File:` group, so it was the last place a parser could still outrank ExifTool's own tables, silently, had the two ever drifted. The generated tables carry FITS in full (the `SIMPLE = {20}T` magic number, the `fits` extension row, `("FITS", "image/fits")`) and answer exactly as those literals did. Verified (instruments named): - Full-JSON diff of both binaries over 207 combined-samples, this branch against main @9270f94a: no key lost, none gained, no value changed. The cleanup is provably invisible, fits.rs included -- which is the evidence that removing its literals cost nothing. - `scripts/compare_file.py` over the same corpus: WRONG 134 -> 134, MISSING 1726 -> 1726, graded output byte-identical. - `cargo test --workspace`: 4840 passed, 0 failed. tests/integration.rs 574 -> 575, the new guard below; no test removed. - clippy and fmt clean; no new warning in any changed file. Guarded by `parsers_do_not_answer_identity_tags_the_file_group_owns`, which runs eight formats through the real `read_metadata` path and asserts a parser answers none of the three, while `File:FileType` still reads the expected value and `File:FileSize` keeps ExifTool's formatting rather than the raw byte count the parsers used to publish beside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
swackhamer
added a commit
that referenced
this pull request
Aug 10, 2026
…iscard (#651) #650 and #654 left one answer per identity fact in the output, both by discarding what the parsers wrote: `normalize_identity_tags` removes an ungrouped `FileType`/`FileTypeExtension`/`MIMEType`, and `drop_redundant_file_size` removes an ungrouped `FileSize`. Neither touched the source. 45 inserts across 40 parsers therefore still computed values that could not reach a caller, and the parsers still read as though `FileSize` and `MIMEType` were theirs to report -- the trap that produced the duplicates in the first place. Removes them. `FileType` deliberately stays: `normalize_identity_tags` promotes it into an absent or `Unknown` `File:FileType`, and that fallback is the only reason SQLite, EVTX, Registry Hive, HDF5, X.509 and ELF are named at all -- ExifTool's tables have no row for any of them, and it refuses to open four outright. Both normalizers stay too, as the backstop for anything that reintroduces one. fits.rs is the one behavioural loose end. It wrote `File:FileType`, `File:FileTypeExtension` and `File:MIMEType` directly -- hardcoded literals, not table lookups -- which put it out of both normalizers' reach, since they only ever removed the ungrouped spellings. `merge` gives format metadata precedence over the `File:` group, so it was the last place a parser could still outrank ExifTool's own tables, silently, had the two ever drifted. The generated tables carry FITS in full (the `SIMPLE = {20}T` magic number, the `fits` extension row, `("FITS", "image/fits")`) and answer exactly as those literals did. Verified (instruments named): - Full-JSON diff of both binaries over 207 combined-samples, this branch against main @9270f94a: no key lost, none gained, no value changed. The cleanup is provably invisible, fits.rs included -- which is the evidence that removing its literals cost nothing. - `scripts/compare_file.py` over the same corpus: WRONG 134 -> 134, MISSING 1726 -> 1726, graded output byte-identical. - `cargo test --workspace`: 4840 passed, 0 failed. tests/integration.rs 574 -> 575, the new guard below; no test removed. - clippy and fmt clean; no new warning in any changed file. Guarded by `parsers_do_not_answer_identity_tags_the_file_group_owns`, which runs eight formats through the real `read_metadata` path and asserts a parser answers none of the three, while `File:FileType` still reads the expected value and `File:FileSize` keeps ExifTool's formatting rather than the raw byte count the parsers used to publish beside it. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
oxidex -jsonemittedFileTypeandMIMETypetwice on any file that reached a parser — once underFile:, once ungrouped — and on 21 of the 194 files in ExifTool's ownt/imagesthe two disagreed.Font.dfontwas the sharp version:File:FileType "DFONT"beside a bareFileType "ICO", left there by the ICO parser, which the file reaches because ExifTool'sFontmagic number matches anything starting\0\x01. Nothing downstream could say which of the two was meant.Roughly forty parsers insert these tags ungrouped while the
File:-grouped ones come fromextract_file_metadataandadd_identity_tags. No step reconciled them, andMetadataMap::mergedoes no group normalization.The fix
normalize_identity_tagsruns afteradd_identity_tagsand leaves one answer per tag, underFile:.The
File:value is the one kept. Both resolvers take it fromcrate::filetype, generated from ExifTool's%fileTypeLookup,%fileTypeExtand%mimeType; a parser's own string is its private spelling of the same fact at best (WebPwhere ExifTool saysWEBP,Plistwhere it saysPLIST) and a loose magic match at worst. Across the 21 disagreements theFile:value is ExifTool's answer 20 times.The one thing a parser can still contribute is a name where the tables produced none, so a bare
FileTypefills an absent orUnknownone before being dropped —EXE.elfkeepsELFrather than losing it toUnknown.MIMETypeis deliberately not treated the same way.application/octet-streamreads like a placeholder but is ExifTool's real answer for DR4, VRD, LNK, MOI and the Mach-O family, so overwriting it with a parser's value would replace a correct answer rather than fill a gap.Verification
Identity sweep over ExifTool's
t/images(194 files) against the pinned 13.59 oracle, before and after:File:-group matchesThe match counts are deliberately unchanged: #644, #648 and #641 already fixed the resolution side, so this is purely the de-duplication that was left over. Zero per-tag regressions.
cargo test --workspacegreen across 47 targets; clippy and fmt clean.Test changes
19 assertions in
production_wiring_tests.rsread the now-removed ungrouped key and were pointed atFile:FileType. Three changed value, each toward ExifTool:Plist→PLIST.emlfixtures:EML→TXT. ExifTool 13.59 has no EML file type —.emlis absent from%fileTypeLookupand an email reportsTXT/txt/text/plain.EMLwas only OxiDex's name for the parser; theEML:tags those tests already assert are what show the routing worked.Not in scope
FileSizehas the same duplicate shape on the same files ("785 bytes"vs a bare"785"vsTEXT:FileSize), as doLineCount/WordCount/Newlinesagainst theirTEXT:copies.XMP.xmlreportsTXTwhere ExifTool saysXMP. fix(filetype): identify unlisted text files as XML or TXT by content #648'sidentify_textdeclines XMP by design; closing that gap means routing<rdf:RDF>content to the XMP parser, which is a coverage change rather than an identity one.Win32 EXE,JXL Codestream,Extended WEBP, and several MIME-value mismatches.🤖 Generated with Claude Code