How to tell if vips is using libjpeg or libjepg-turbo? #229
Comments
keeganwitt
changed the title
|
I saw libvips/libvips#1098, but in my case, it doesn't look like the binary got installed as part of the vips package. |
|
On debian, just do |
I've done that, but libjpeg8 is also an installed package. I was concerned it might not be using what I thought it was using. I have no particular reason to suspect so, other than |
|
You could try: Your slow tiffsave is probably caused by something else in your pipeline :( I could check your code if you upload a benchmark. |
|
Thanks for your kind offer. Do you have a doc describing how to capture a benchmark for analysis (sorry for my ignorance)? I installed the tools package so I have access to the vips command, and it looks like it is indeed linking to the wrong library This is running a bit of a tweaked vips Debian package, so I can run a reasonably recent vips on an older Ubuntu, which I describe here. I'm compiling it in whatever way Ubuntu was, unless I'm missing something. Support for turbo isn't directly compiled in, right (so it's OK I'm not directly invoking |
|
That could be the right binary --- try the I usually benchmark stuff by writing a few lines of python that does what I need to do. pyvips is high-level and very quick to develop. |
|
Ah, you're right. I thought the object files would have a different name. |
|
For API mode you can also use: |
|
It's because libjpeg-turbo is completely binary-compatible. You can swap the |
|
I shared a file with you on Google Drive. # docker build -t vips-test .
# docker run --rm -v $PWD/images:/images vips-test time vips tiffsave 13.tiff 13-pyramid.tiff --compression=jpeg --tile --tile-width=256 --tile-height=256 --pyramid --bigtiff
FROM ubuntu:20.10
VOLUME /images
WORKDIR /images
RUN set -o errexit -o nounset \
&& apt-get -y update \
&& apt-get install --no-install-recommends --no-install-suggests -y \
libjpeg-turbo8-dev \
libpng-dev \
libtiff5-dev \
libvips-dev \
libvips-tools \
time \
&& rm -rf /var/lib/apt/lists/*It could be we just have to live with this due to file size (there are some files we deal with that are even bigger), but let me know if you have any suggestions. |
|
Hi again, I tried this dockerfile: I see: So I think you're probably using |
|
I tried on my PC fwiw and I see: I think that's probably as fast as libvips can easily go at this task. There are some things that could be improved internally (moving jpg-encode outside the lock, buffering other pyr layers in ram), but they would be a fair amount of work. |
|
It looked like when this was running in Kubernetes only 1 vcpu was being used. Is there some arg needed to make better use of the cores? |
|
libvips checks the number of CPUs available on startup and will use all of them by default. Perhaps your k8 setup needs something to allocate another CPU to this task? |
|
I know this is a potentially complicated question, but what's the mechanism Vips uses for core detection? I'm wondering if there's a mismatch between how K8s configures this and how Vips detects it. |
|
It's this horrible thing: https://github.com/libvips/libvips/blob/master/libvips/iofuncs/threadpool.c#L315-L392 Which on linux boils down to just: int n_proc = sysconf( _SC_NPROCESSORS_ONLN ); |
|
I've seen the vcpu usage go up to like 1500 or 1700 at one point, but then dropped down to like 1010, so still not using much more than 1 core. Maybe the "reduceh: 13 point mask" (the last thing I saw logged) just isn't something very parallelizable... |
|
I think we've answered the original question associated with this issue though, so closing. Thanks for your help! |
Similar to the question asked about Pillow (where they added a way in Python to check), I'm wondering how I can know whether libjpeg or libjepg-turbo is being used by pyvips/libvips.
The text was updated successfully, but these errors were encountered: