How to tell at run time whether libjpeg-turbo version of libjpeg is used?

[stas00]

tl;dr:

Is there some way to accomplish: PIL.Image.libjpeg_turbo_is_enabled()?

The full story:

Is there a way to tell from a pre-built Pillow whether it was built against libjpeg-turbo or not?

This is assuming that all I have is libjpeg.so.X.X and no way to tell where it came from.

I see there is a symbol in the library:

nm _imaging.cpython-36m-x86_64-linux-gnu.so | grep -I turbo
000000000007e5a0 D libjpeg_turbo_version

but I don't know how to access its value from python.

If there is a way to tell the same from from shell using ldd/nm or other linker tools, it'd do too.

The intention is to be able to tell a user at run-time to re-build Pillow after installing libjpeg-turbo to gain speed. The problem is that It's not enough to build Pillow against libjpeg-turbo. Given how conda/pip dependencies work, a new prebuilt package of Pillow could get swapped in as a dependency for some other package and the user won't know that they now run a less efficient Pillow unless they watch closely any install/update logs.

Currently the only solution I can think of (in conda env) is to take the output of:

cd ~/anaconda3/envs/pytorch-dev/lib/python3.6/site-packages/PIL
ldd  _imaging.cpython-36m-x86_64-linux-gnu.so | grep libjpeg

which wold give me something like:

libjpeg.so.8 => ~/anaconda3/envs/pytorch-dev/lib/libjpeg.so.8

And then to try to match it to:

grep libjpeg  ~/anaconda3/envs/pytorch-dev/conda-meta/libjpeg-turbo-2.0.1-h470a237_0.json

which may work. There is a problem with this approach

It's very likely that conda is going to reinstall jpeg since many packages depend on it, and when it does, there is going to be 2 libjpeg libs.

ldd  _imaging.cpython-36m-x86_64-linux-gnu.so | grep libjpeg
    libjpeg.so.8 => /home/stas/anaconda3/envs/pytorch-dev/lib/libjpeg.so.8 (0x00007f92628c8000)
    libjpeg.so.9 => /home/stas/anaconda3/envs/pytorch-dev/lib/./libjpeg.so.9 (0x00007f9261c4e000)

And now I can no longer tell which is which, since I can no longer tell which of the two Pillow will load at run time. Well, I can go one step further and check /proc//maps to get the library, but it's getting more and more convoluted. And I won't even know how to do the same on non-linux platform. And this is just for the conda setup, for pip setup it'd be something else.

Also what happens if libjpeg-turbo and libjpeg are the same version?

Perhaps there is an easier way? Any chance to have PIL.Image.libjpeg_turbo_is_enabled()?

Thank you.

[radarhere]

Would you be able to test PR #3493, and see if that works for you? That PR would let you use the following code to test for libjpegturbo -

from PIL import features
features.check_feature('libjpegturbo')

[stas00]

Thank you for the PR, @radarhere! It needs more work with the right includes.

It works (returns True) only with this patch on top of your PR:

--- a/src/_imaging.c
+++ b/src/_imaging.c
@@ -79,6 +79,10 @@

 #include "Imaging.h"

+#ifdef HAVE_LIBJPEG
+#include "jconfig.h"
+#endif
+

Otherwise LIBJPEG_TURBO_VERSION is not visible inside src/_imaging.c.

In src/libImaging/JpegDecode.c it came through #include "Jpeg.h", but if I try to include the same in _imaging.c it won't compile.

I don't know enough about this codebase to tell whether this might break something else.

Also may I ask to rename the new "feature" to libjpeg-turbo to match the package name (an extra -)? and perhaps (less important) HAVE_LIBJPEG_TURBO in the C code.

Thank you.

[hugovk]

Variable name bikeshedding time!

The existing features use underscores, let's be consistent with our existing API. And the others are all HAVE_SOMEFEATURE (eg. not HAVE_WEBP_ANIM):

 features = {
    "webp_anim": ("PIL._webp", 'HAVE_WEBPANIM'),
    "webp_mux": ("PIL._webp", 'HAVE_WEBPMUX'),
    "transp_webp": ("PIL._webp", "HAVE_TRANSPARENCY"),
    "raqm": ("PIL._imagingft", "HAVE_RAQM"),
}

So I'd probably go for:

    "libjpeg_turbo": ("PIL._imaging", "HAVE_LIBJPEGTURBO"),

[radarhere]

Okay, the PR now has the header from @stas00, and the strings from @hugovk

[stas00]

Looking good, @radarhere.

My only concern was of what would happen if after compiling Pillow, someone swaps in an identical version of libjpeg.so.X.Y.Z (i.e. conda uninstall libjpeg-turbo; conda install jpeg), but I wasn't able to find or build an identical version pair, so I can't validate what would happen then (Pillow thinks it's linked against libjpeg-turbo version of libjpeg.so, but that might not be the case).

[stas00]

And I wrote a little code snippet to use this feature (perhaps it's doc worthy) and I assume it'll be available in 5.4.0.

from PIL import features, Image
from distutils.version import LooseVersion

if LooseVersion(Image.PILLOW_VERSION) >= LooseVersion("5.4.0"): 
    if features.check_feature('libjpeg_turbo'):
        print("libjpeg-turbo is on")
    else:
        print("libjpeg-turbo is not on")
else:
    print(f"libjpeg-turbo' status can't be derived - need Pillow(-SIMD)? >= 5.4.0 to tell, current version is {Image.PILLOW_VERSION}")

packaging's version is probably a better choice as it correctly deals with .postX that pillow-simd uses.

from PIL import features, Image
from packaging import version

if version.parse(Image.PILLOW_VERSION) >= version.parse("5.4.0"): 
    if features.check_feature('libjpeg_turbo'):
        print("libjpeg-turbo is on")
    else:
        print("libjpeg-turbo is not on")
else:
    print(f"libjpeg-turbo' status can't be derived - need Pillow(-SIMD)? >= 5.4.0 to tell, current version {Image.PILLOW_VERSION}")