test_file_jpeg2k fails with IOError: broken data stream when reading image file on Windows

[cgohlke]

Using Pillow af4424e linked to openjpeg 2.0svn on Windows, the following tests fail on all Python versions.

Tests\test_file_jpeg2k.py:35: im.load() failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 35, in test_sanity
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
Tests\test_file_jpeg2k.py:47: im.load() failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 47, in test_lossless
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
Tests\test_file_jpeg2k.py:53: im.load() failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 53, in test_lossy_tiled
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
Tests\test_file_jpeg2k.py:59: im = roundtrip(test_card) failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 59, in test_lossless_rt
    im = roundtrip(test_card)
  File "Tests\test_file_jpeg2k.py", line 25, in roundtrip
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
Tests\test_file_jpeg2k.py:63: im = roundtrip(test_card, quality_layers=[20]) failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 63, in test_lossy_rt
    im = roundtrip(test_card, quality_layers=[20])
  File "Tests\test_file_jpeg2k.py", line 25, in roundtrip
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
Tests\test_file_jpeg2k.py:67: im = roundtrip(test_card, tile_size=(128, 128)) failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 67, in test_tiled_rt
    im = roundtrip(test_card, tile_size=(128, 128))
  File "Tests\test_file_jpeg2k.py", line 25, in roundtrip
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file






Tests\test_file_jpeg2k.py:72: offset=(32, 32)) failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 72, in test_tiled_offset_rt
    offset=(32, 32))
  File "Tests\test_file_jpeg2k.py", line 25, in roundtrip
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
Tests\test_file_jpeg2k.py:76: im = roundtrip(test_card, irreversible=True, quality_layers=[20]) failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 76, in test_irreversible_rt
    im = roundtrip(test_card, irreversible=True, quality_layers=[20])
  File "Tests\test_file_jpeg2k.py", line 25, in roundtrip
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
Tests\test_file_jpeg2k.py:80: im = roundtrip(test_card, quality_layers=[60, 40, 20], progression='LRCP') failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 80, in test_prog_qual_rt
    im = roundtrip(test_card, quality_layers=[60, 40, 20], progression='LRCP')
  File "Tests\test_file_jpeg2k.py", line 25, in roundtrip
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
Tests\test_file_jpeg2k.py:84: im = roundtrip(test_card, num_resolutions=8, progression='RLCP') failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 84, in test_prog_res_rt
    im = roundtrip(test_card, num_resolutions=8, progression='RLCP')
  File "Tests\test_file_jpeg2k.py", line 25, in roundtrip
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
Tests\test_file_jpeg2k.py:92: im.load() failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 92, in test_reduce
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
Tests\test_file_jpeg2k.py:103: im.load() failed:
Traceback (most recent call last):
  File "Tests\test_file_jpeg2k.py", line 103, in test_layers
    im.load()
  File "D:\Build\Pillow\Pillow-git\PIL\Jpeg2KImagePlugin.py", line 181, in load
    ImageFile.ImageFile.load(self)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 234, in load
    raise_ioerror(e)
  File "D:\Build\Pillow\Pillow-git\PIL\ImageFile.py", line 56, in raise_ioerror
    raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file

[wiredfool]

@al45tair? Ideas?

[aclark4life]

FYI: release is Tuesday

[al45tair]

I'll take a look first thing tomorrow (Monday); I suspect it's something silly and easily fixed.

[al45tair]

OK, so I've taken a look at this, and after fixing a problem introduced by e7e103b (I'm using the OpenJPEG binaries installed using the installer from the OpenJPEG website, not building my own, so forcing it to statically link does not work), and after removing a line from the test script that tried to save a file in /tmp (my fault, shouldn't have been left in, sorry), I'm unable to reproduce this problem.

The most likely reason for this to happen is that Pillow was built against a different version of the Microsoft C runtime than the one that Python was linked with. As a result, when Python passes an fd to the code in Pillow, it won't work properly (for those who don't already know, fds on Windows are simulated by the C runtime — Windows’ native handle type is not int; the problem is that there are multiple versions of the C runtime, and if you link together code that expects different C runtimes, it will load all of the C runtimes separately into the process, which means each of them has a separate handle table).

Of course, I can't be certain that's the problem because I don't have the @cgohlke's build environment to look at, but it seems likely.

To be clear: I’ve built Pillow with Visual Studio 2008 (VC9) using Python 2.7 to test this. I haven't tried Python 3 on Windows (IIRC 3.4 needs a newer version of Visual Studio, but not the latest version).

[cgohlke]

Closing.

Turns out that using openjpeg sources from svn\tags\version.2.0 instead of svn\branches\openjpeg-2.0 fixes these errors. This might come up again with the next 2.0.1 or 2.1 release.

I assume this is the line containing the hard coded /tmp/ that should be fixed before the release?

[al45tair]

@cgohlke Yes, that's the line in question.