cjxl - temp JPEG files remain

[eddiezato]

If the input files are JPEGs, cjxl creates jpe????.tmp files in the system %temp% folder and doesn't delete them after the encoding is finished.

Windows 10 20H2 x64, cjxl v0.3.7-12-g04267a8

[jonsneyers]

I wonder what is creating these temp files. I don't think jxl itself does that; perhaps libjpeg does it for some reason?

Scope on discord pointed to https://github.com/libjpeg-turbo/libjpeg-turbo/blob/main/example.txt#L208 which might be an explanation.

[JamesPous]

I recognized the same problem with jpe*.tmp files with the cjxl.exe (JPEG XL encoder 0.3.7) under Windows 10 64bit (lastest updates).

[eustas]

"On some systems you may need to set up a signal handler to ensure that temporary files are deleted if the program is interrupted. See libjpeg.txt."

[eustas]

But: "NOTE: Unless you develop your own memory manager back end, then temporary files
will never be used. The back end provided in libjpeg-turbo (jmemnobs.c) simply
malloc()s and free()s virtual arrays, and an error occurs if the required
memory exceeds the limit specified in cinfo->mem->max_memory_to_use."

[eddiezato]

It's best to use the memory manager jmemnobs.c if you can (ie, if you have enough real or virtual main memory). If not, at least make sure that max_memory_to_use is set as high as possible. If the JPEG memory manager has to use a temporary file, you will probably see a lot of disk traffic and poor performance.

Probably max_memory_to_use isn't used and defaults to 0. This forces the encoder to create temp files every time.
I even tried 1-pixel jpeg. Same thing.

[eustas]

So, it is easily reproducible? Good! Does it create file even if processing was successful?
(going to try to reproduce myself, but will bother you further if won't succeed).

[eddiezato]

Even a simple cjxl 0.jpg will create a temp file that's just a copy of the input jpeg. On Windows at least.

Does it create file even if processing was successful?

It always creates temp files, regardless of whether the process was successful or failed.

[eustas]

Spent some time trying to build project on windows, with limited success (got binary that does not support JPEG).
(Hmm, we have to improve that story - vcpkg and chocolatey is a must).

Next, built binaries via cross-compilation in docker container. Those, however work fine and does not create temporay files.

@eddiezato could you share your experience, how you build / obtain the cjxl / djxl binaries, please.

[eddiezato]

I build with mingw64/msys2 myself.
Share them here https://eddiezato.github.io/bin/

#!/bin/bash
export CC=clang CXX=clang++
git clone https://github.com/libjxl/libjxl.git --recursive
cd libjxl
./deps.sh
cmake -B build -G Ninja -S ./ \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_TESTING=OFF \
    -DJPEGXL_ENABLE_DEVTOOLS=ON \
    -DJPEGXL_ENABLE_OPENEXR=OFF \
    -DJPEGXL_ENABLE_SKCMS=OFF \
    -DJPEGXL_ENABLE_BENCHMARK=OFF \
    -DJPEGXL_WARNINGS_AS_ERRORS=OFF \
    -DJPEGXL_STATIC=ON \
    -DCMAKE_CXX_FLAGS='-ffunction-sections -fdata-sections' \
    -DCMAKE_EXE_LINKER_FLAGS='-Wl,--gc-sections -Wl,--no-export-dynamic'
ninja -C build
strip -s build/tools/{cjxl.exe,djxl.exe}

[eddiezato]

I haven't thought about it. But it could be a problem with the mingw package for libjpeg-turbo. If it turns out that the problem occurs only on mingw builds.

[eustas]

At last, succeeded with MSYS2 / mingw-w64-x86_64-clang.
Can't reproduce =(

[eustas]

(oh, but perhaps I have no libjpeg-turbo installed)

[eustas]

Was able to reproduce: https://github.com/eustas/libjxl-msys2/runs/3001986608?check_suite_focus=true#step:4:26

[eustas]

Tried building without libjpeg-turbo... same result.

[eddiezato]

This also happens if the input jpeg is not valid, not even jpg. 😀

PS > ls -Path $Env:temp\jpe*.tmp
PS > Rename-Item cjxl.exe cjxl.jpg
PS > cjxl cjxl.jpg
JPEG XL encoder v0.3.7 [SSE4]
Failed to read image cjxl.jpg.
PS > ls -Path $Env:temp\jpe*.tmp

    Directory: Z:\Temp

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          2021.07.07     8:55        6641152 jpeD8F3.tmp

[eustas]

Pinpointed the problem. It is: 🤦 🙀
Will fix soon.

[eustas]

(and it have nothing to do with JPEG)

[eddiezato]

@eustas I checked #300 , no temp files. Thanks 👍