DDL-Comparison

We are testing the dfdl and kaitai parsers on four different filetypes: bmp, gif, jpg, and png. To verify the validity of files being tested, we're also utilizing the Pillow python library and the snowy python library. Our interest lies in images that dfdl and kaitai disagree on (i.e. one marks it as valid while another doesn't), and investigate why that is the case.

Gathering Images

We created a python script (download_images.py) that would scrape images of different file extensions from common crawler.

Usage is python3 download_images.py -l <limit> -f <formats>

The program also expects a subdirectory to be in the same directory as download_images.py, within this directory should be a file called wet.paths, this is downloaded from https://commoncrawl.org/the-data/get-started/. Images that are scraped from this wet.paths file will then be placed in this subdirectory also.

On line 40 of the python script, you can manually edit the number of threads you want to use. (The line NUM_OF_THREADS = 1 can be edited)

Comparing DDLs

We created a python script (original-test.py) that would go through given directories and compare how kaitai and dfdl parsed the file, more specifically whether or not any of them failed. The script then writes this information to a database.

Usage is python3 original-test -f <filetype> -p <path> path refers to the directory storing our desired images

Files

Local Libraries/Schema (Dependencies)

Kaitai and DFDL schemas are already in this file. We include Kaitai formats in the ksy-formats directory, which are used to produce python-based schemas in ksy-schemas. For DFDL, we store schemas in DFDL-schemas. Other DFDL schemas can be found online.

Another dependency is Apache's daffodil, download the binary from their releases page.

Scripts

original-test.py
download_images.py

Testing

To be able to recreate the testing done, depencies will have to be downloaded (python dependencies are in requirements.txt, but apache's daffodil will have to be downloaded and added to your path).

When running this, you should also pipe stdout to an output file.

DFDL

The daffodil command line interface is utilized, it requires a parser or a schema.
We're using DFDL schemas for our 5 image types from GitHub
- png - schema at: PNG-master/src/main/resources/com/mitre/png/xsd/png.dfdl.xsd
- jpeg - schema at: JPEG-master/src/main/resources/com/mitre/jpeg/xsd/jpeg.dfdl.xsd
- gif - schema at: GIF-master/src/main/resources/com/mitre/gif/xsd/gif.dfdl.xsd
- bmp - schema at: BMP-master/src/main/resources/com/mitre/bmp/xsd/bmp.dfdl.xsd
- nitf - schema at: https://github.com/DFDLSchemas/NITF/blob/master/src/main/resources/com/tresys/nitf/xsd/nitf.dfdl.xsd
testing on command line:

# tested parsing a valid jpeg file with a jpeg schema

$ daffodil parse -s dfdl-schemas/jpeg.dfdl.xsd valid.jpg > valid.out

$ head valid.out

<?xml version="1.0" encoding="UTF-8"?>
<JFIF>
  <Segment>
    <SOI></SOI>
  </Segment>
  <Segment>
    <APP0>9+


# tested parsing an invalid jpeg file with a jpeg schema
# empty file outputted instead

$ daffodil parse -s dfdl-schemas/jpeg.dfdl.xsd invalid.jpg > invalid1.out

$ head invalid1.out

# same applies for any other file type, empty file also outputted

$ daffodil parse -s dfdl-schemas/jpeg.dfdl.xsd valid.png > invalid2.out

$ head invalid2.out

We automate testing in python by using the subprocess library and looking at the return code to see if it's successful or not

Kaitai

We found Kaitai struct formats for each image type here
using these we can generate python files for each corresponding image type. For example, for a bmp filetype, a python file is generated using: ksc bmp.ksy -t python which produced bmp.py
note: for gif.ksy, I removed the last 8 lines because they weren't needed, and caused errors.
Then, with these files, we can refer to it like a library for images.
Testing on python:

import jpeg # jpeg.py was generated by ksc (KaitaiStruct Compiler)
from kaitaistruct import *

object = jpeg.Jpeg.from_file("file")

if the file passed into the python function call doesn't adhere to its specification, an exception is raised, specifically a kaitaistruct.ValidationNotEqualError!

Pillow

Pillow's open() function is used
Taking in a filepath, it generates an object, we then check the object's format as a way of verifying its filetype.

from PIL import Image

im1 = Image.open("validbmp.bmp")
print(im1.format) # prints "BMP"

im2 = Image.open("validjpeg.jpg")
print(im2.format) # prints "JPEG"

Results (Winter 2022)

We're dividing the images that we're going to individually look at into 4 cases. But here's a basic overview of interesting things found:

100/166 of images that KaitaiStruct found was invalid was also found to be invalid by DFDL
3212 images were flagged invalid by DFDL but valid for KaitaiStruct
snowy failed only 16 images, and with the exception of one image, either DFDL or KaitaiStruct also found the image to be invalid.
Pillow failed only 1 image, every other parser also flagged it as invalid.

So we start by looking at two images for each of these cases,

⌐Snowy ∧ DFDL ∧ Kaitai ∧ Pillow
- this may help identify if Snowy is wrong or not
Snowy ∧ Pillow ∧ ⌐Kaitai ∧ DFDL
- this should help identify bugs in kaitai
Snowy ∧ Pillow ∧ ⌐Kaitai ∧ ⌐DFDL
Snowy ∧ Pillow ∧ Kaitai ∧ ⌐DFDL
- this should help identify bugs in DFDL

⌐Snowy ∧ DFDL ∧ Kaitai ∧ Pillow

658423.jpg

Snowy_string: ValueError

Snowy error message:

  File "/home/kris/.local/lib/python3.9/site-packages/imageio/plugins/pillow_legacy.py", line 654, in pil_try_read
    im.getdata()[0]
  File "/home/kris/.local/lib/python3.9/site-packages/PIL/Image.py", line 1294, in getdata
    self.load()
  File "/home/kris/.local/lib/python3.9/site-packages/PIL/ImageFile.py", line 272, in load
    raise_oserror(err_code)
  File "/home/kris/.local/lib/python3.9/site-packages/PIL/ImageFile.py", line 67, in raise_oserror
    raise OSError(message + " when reading image file")
OSError: broken data stream when reading image file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kris/.local/lib/python3.9/site-packages/snowy/io.py", line 99, in load
    return reshape(np.float64(_load(filename, ext, not linearize)))
  File "/home/kris/.local/lib/python3.9/site-packages/snowy/io.py", line 80, in _load
    img = imageio.imread(filename)
  File "/home/kris/.local/lib/python3.9/site-packages/imageio/core/functions.py", line 160, in imread
    return file.read(index=0, **kwargs)
  File "/home/kris/.local/lib/python3.9/site-packages/imageio/core/legacy_plugin_wrapper.py", line 129, in read
    reader = self.legacy_get_reader(**kwargs)
  File "/home/kris/.local/lib/python3.9/site-packages/imageio/core/legacy_plugin_wrapper.py", line 99, in legacy_get_reader
    return self._format.get_reader(self._request)
  File "/home/kris/.local/lib/python3.9/site-packages/imageio/core/format.py", line 167, in get_reader
    return self.Reader(self, request)
  File "/home/kris/.local/lib/python3.9/site-packages/imageio/core/format.py", line 217, in __init__
    self._open(**self.request.kwargs.copy())
  File "/home/kris/.local/lib/python3.9/site-packages/imageio/plugins/pillow_legacy.py", line 468, in _open
    return PillowFormat.Reader._open(self, pilmode=pilmode, as_gray=as_gray)
  File "/home/kris/.local/lib/python3.9/site-packages/imageio/plugins/pillow_legacy.py", line 300, in _open
    pil_try_read(self._im)
  File "/home/kris/.local/lib/python3.9/site-packages/imageio/plugins/pillow_legacy.py", line 665, in pil_try_read
    raise ValueError(error_message)
ValueError: Could not load "" 
Reason: "broken data stream when reading image file"
Please see documentation at: http://pillow.readthedocs.io/en/latest/installation.html#external-libraries

Snowy ∧ Pillow ∧ ⌐Kaitai ∧ DFDL

325385.jpg

Kaitai_string = EOFError

Kaitai error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/kaitaistruct.py", line 35, in from_file
    return cls(KaitaiStream(f))
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 40, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 46, in _read
    self.segments.append(Jpeg.Segment(self._io, self, self._root))
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 90, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 109, in _read
    self.data = Jpeg.SegmentApp0(_io__raw_data, self, self._root)
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 243, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 254, in _read
    self.thumbnail = self._io.read_bytes(((self.thumbnail_x * self.thumbnail_y) * 3))
  File "/usr/lib/python3/dist-packages/kaitaistruct.py", line 290, in read_bytes
    raise EOFError(
EOFError: requested 21840 bytes, but got only 3853 bytes

325394.jpg

Kaitai_string = EOFError

Kaitai error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/kaitaistruct.py", line 35, in from_file
    return cls(KaitaiStream(f))
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 40, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 46, in _read
    self.segments.append(Jpeg.Segment(self._io, self, self._root))
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 90, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 109, in _read
    self.data = Jpeg.SegmentApp0(_io__raw_data, self, self._root)
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 243, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 254, in _read
    self.thumbnail = self._io.read_bytes(((self.thumbnail_x * self.thumbnail_y) * 3))
  File "/usr/lib/python3/dist-packages/kaitaistruct.py", line 290, in read_bytes
    raise EOFError(
EOFError: requested 21840 bytes, but got only 2817 bytes

Snowy ∧ Pillow ∧ ⌐Kaitai ∧ ⌐DFDL

420325.jpg

DFDL_string = Schema Definition Warning

DFDL error message:

[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 632 column 58 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 624 column 58 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 715 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 731 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 360 column 46 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[3] Location line 762 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 345 column 46 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: DFDL property was ignored: representation="text"
Schema context: Text Location line 459 column 18 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[error] Parse Error: Assertion failed: Does not end with an EOI segment.
Schema context: sequence[2] Location line 63 column 18 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
Data location was preceding byte 82491

Kaitai_string = EOFError

Kaitai error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/kaitaistruct.py", line 35, in from_file
    return cls(KaitaiStream(f))
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 40, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 46, in _read
    self.segments.append(Jpeg.Segment(self._io, self, self._root))
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 90, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 109, in _read
    self.data = Jpeg.SegmentApp0(_io__raw_data, self, self._root)
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 243, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 254, in _read
    self.thumbnail = self._io.read_bytes(((self.thumbnail_x * self.thumbnail_y) * 3))
  File "/usr/lib/python3/dist-packages/kaitaistruct.py", line 290, in read_bytes
    raise EOFError(
EOFError: requested 21840 bytes, but got only 2501 bytes

705149.jpg

DFDL_string = Schema Definition Warning

DFDL error message:

[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 632 column 58 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 624 column 58 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 715 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 731 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 360 column 46 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[3] Location line 762 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 345 column 46 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: DFDL property was ignored: representation="text"
Schema context: Text Location line 459 column 18 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[error] Parse Error: Assertion failed: Does not end with an EOI segment.
Schema context: sequence[2] Location line 63 column 18 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
Data location was preceding byte 33292

Kaitai_string = EOFError

Kaitai error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/kaitaistruct.py", line 35, in from_file
    return cls(KaitaiStream(f))
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 40, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 46, in _read
    self.segments.append(Jpeg.Segment(self._io, self, self._root))
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 90, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 109, in _read
    self.data = Jpeg.SegmentApp0(_io__raw_data, self, self._root)
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 243, in __init__
    self._read()
  File "/home/kris/Projects/DDL-Comparison/jpeg.py", line 254, in _read
    self.thumbnail = self._io.read_bytes(((self.thumbnail_x * self.thumbnail_y) * 3))
  File "/usr/lib/python3/dist-packages/kaitaistruct.py", line 290, in read_bytes
    raise EOFError(
EOFError: requested 189720 bytes, but got only 32634 bytes

Snowy ∧ Pillow ∧ Kaitai ∧ ⌐DFDL

823804.jpg

DFDL_string = Schema Definition Warning

DFDL error message:

[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 632 column 58 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 624 column 58 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 715 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 731 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 360 column 46 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[3] Location line 762 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 345 column 46 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: DFDL property was ignored: representation="text"
Schema context: Text Location line 459 column 18 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[error] Parse Error: Assertion failed: Does not end with an EOI segment.
Schema context: sequence[2] Location line 63 column 18 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
Data location was preceding byte 57279

823662.jpg

DFDL_string = Schema Definition Warning

DFDL error message:

[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 632 column 58 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 624 column 58 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 715 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 731 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[2] Location line 360 column 46 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[3] Location line 762 column 30 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents.
Schema context: sequence[1] Location line 345 column 46 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Schema Definition Warning: DFDL property was ignored: representation="text"
Schema context: Text Location line 459 column 18 in file:/home/kris/files/dfdl-schemas/jpeg.dfdl.xsd
[warning] Left over data. Consumed 836328 bit(s) with at least 616 bit(s) remaining.
Left over data (Hex) starting at byte 104542 is: (0x3c42523e0d0a3c48...)
Left over data (UTF-8) starting at byte 104542 is: (<BR>␍␊<H...)

Side note on Pillow

We've looked into images that were accepted by Pillow but failed for KaitaiStruct and noticed that the APP0 marker is identified by a JFXX string (introduced in JPEG file interchange format v1.02). We noticed that the Pillow library does not check for the JFXX string at all, in their elif branches checking for other identifiers, they don't end with an else branch -- hence an image like 325385.jpg was accepted by Pillow but not KaitaiStruct.

We wanted to then find out if Pillow should've supported functionality for JFXX. I started by looking at the sustainability of digital formats library and what they had to say about the jpeg image encoding family and JPEG v1.02. According to the latter link, "in 2013, the same version of JFIF was approved as ISO standard ISO/IEC 10918-5:2013. ISO/IEC 10918-5:2013 also seems to be the latest format specification, as it's mentioned that "this standard was last reviewed and confirmed in 2018. Therefore this version remains current". Since the standard is reviewed every 5 years, this is the latest standard.

After writing an issue on GitHub we found that Pillow handled it right, it may not be explicit in their code but they handle the extension app0 segment.

Flaw in Kaitai Struct

KaitaiStruct is not checking for app0 JFIF extension segments, so when there is a JFXX app0 segment following a JFIF app0 segment, the parser is not handling it correctly and calculates the wrong value, leading to an error.

The parser is treating it as an individual app0 segment, when its behaviour is meant to be different because it is an extension app0 marker segment.

Right after the JFIF app0 marker segment, there might be a JFIF extension app0 marker (Kaitai doesn't check for this). Here's the spec details that Kaitai currently doesn't cover: X'ff', APP0, length, identifier, extension code, extension data

length is 2bytes
identifier is 5 bytes, should say JFXX
extension code/ thumbnail format 1 byte
- 0x10 - jpeg
- 0x11 - 1 byte per pixel
- 0x13 - 3 byte per pixel rgb format thumbnail data depends on thumbnail format. The info from this link specifies the jpeg format that's missing from kaitaistruct.

Currently Kaitai struct does what the JFIF app0 marker segment does to find thumbnail data: 3 * thumbnailx * thumbnaily, but that's incorrect for 2 out of 3 app0 extension segments.

Now when trying to fix this problem by adjusting KaitaiStruct's spec for jpeg

the jfif app0 extension always comes following the first app0 extension
there was no way to add an optional section directly following another, so I couldn't add the details at the end of the app0 segment.

Kaitai's online web IDE doesn't seem to be working perfectly, so I'm testing the edited kaitai jpeg spec by running and examining by hand.

On Kaitai's switch statement I wanted to be able to check for hex JFXX00 and hex JFIF00, but switch statements don't allow me to express values as ["JFXX",null], these also don't work: JFXX\x00, JFXX\0. So instead, for the app0 extension, I'm checking 4 bytes, checking if they're JFXX or JFIF in the switch

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
__pycache__		__pycache__
dfdl-schemas		dfdl-schemas
ksy-formats		ksy-formats
ksy_schemas		ksy_schemas
LICENSE		LICENSE
README.md		README.md
download_images.py		download_images.py
exif.py		exif.py
original-test.py		original-test.py
requirements.txt		requirements.txt

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

DDL-Comparison

Gathering Images

Comparing DDLs

Files

Local Libraries/Schema (Dependencies)

Scripts

Testing

DFDL

Kaitai

Pillow

Results (Winter 2022)

⌐Snowy ∧ DFDL ∧ Kaitai ∧ Pillow

Snowy ∧ Pillow ∧ ⌐Kaitai ∧ DFDL

Snowy ∧ Pillow ∧ ⌐Kaitai ∧ ⌐DFDL

Snowy ∧ Pillow ∧ Kaitai ∧ ⌐DFDL

Side note on Pillow

Flaw in Kaitai Struct

About

Releases

Packages

Contributors 2

Languages

License

Dartmouth-Trustlab/commoncrawl-images

Folders and files

Latest commit

History

Repository files navigation

DDL-Comparison

Gathering Images

Comparing DDLs

Files

Local Libraries/Schema (Dependencies)

Scripts

Testing

DFDL

Kaitai

Pillow

Results (Winter 2022)

⌐Snowy ∧ DFDL ∧ Kaitai ∧ Pillow

Snowy ∧ Pillow ∧ ⌐Kaitai ∧ DFDL

Snowy ∧ Pillow ∧ ⌐Kaitai ∧ ⌐DFDL

Snowy ∧ Pillow ∧ Kaitai ∧ ⌐DFDL

Side note on Pillow

Flaw in Kaitai Struct

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Languages

Packages