3.0.0: test suite is failing in example-12bit-shared-decompress-cmp unit
#705
Comments
|
I am seeing two test failures as well as in CI relates to Homebrew/homebrew-core#135676 |
|
Hmm .. why those units are downloading dist tar ball? 🤔 |
|
We have seen no such failures in our own CI system, which has been continuously testing the code for the entire five months that libjpeg-turbo 3.0 has been in beta. In other to reproduce the failures, I would need to know more about your build environment. The unit tests do not download a distribution tarball. |
|
Also reproducable on FreeBSD 13.2-RELEASE (amd64) It fails when (in my case) |
|
I see 295 - example-12bit-shared-decompress-cmp (Failed) here as well Maybe it's a problem with running the tests in parallel? Or at least they reliably pass if I pass -j1 to ctest. |
|
Fails with |
|
The floating point test failures are probably compiler-specific. Our build system sets a default value for As far as the
No one has yet provided that information. |
|
What information specifically do you want? #705 (comment) seemed to give enough so I didn't bother looking myself yet. I can do if it'd be useful if you can't reproduce with that |
It would be a lot more useful for the OP to answer my question about their test environment than for everyone else to pile on. I cannot talk to five people at once. #705 (comment) describes testing with FreeBSD 13, and I don't think I have a test environment for that, so I would like to know how to repro this with Linux. |
This test failure occurs on macOS 13 on arm64 (Apple Silicon) with Xcode 14.3. We invoke We also build with the exact same configuration on Linux (Ubuntu 22.04, x86_64), macOS 13-x86_64 (also with Xcode 14.3), macOS 12 (both x86_64 and arm64, Xcode 14.2) and macOS 11 (both x86_64 and arm64, Xcode 13.2), but did not encounter any failures on these setups. Complete build logs are available at https://github.com/Homebrew/homebrew-core/actions/runs/5447007381/jobs/9916243774?pr=135676. This means, unfortunately, that this particular failure will likely not reproduce on Linux. Please let us know what else we can do to resolve this. How important are the failing test cases? |
The floating point tests are not important. They test the floating point DCT/IDCT algorithms, which are a legacy feature. (Back in 1991, when Tom Lane first released libjpeg, some CPUs had floating point units and some didn't. It could take minutes to compress or decompress a 1-megapixel JPEG image using the "slow integer" DCT/IDCT algorithms, and the floating point algorithms were significantly faster on systems that had an FPU. On systems without FPUs, floating point math was emulated and really slow, so Tom also developed "fast integer" DCT/IDCT algorithms to speed up JPEG performance, at the expense of accuracy, on those systems. In 2023, the floating point algorithms are significantly slower than the "slow integer" algorithms without being significantly more accurate, and the "fast integer" algorithms fail the ISO/ITU-T conformance tests without being any faster than the "slow integer" algorithms on x86 systems. Thus, the floating point and "fast integer" algorithms are considered legacy features.) The problem with the floating point algorithms vis-a-vis regression testing is that their output can vary based on how the compiler implements floating point math (which can vary based on the compiler settings.) The results for 8-bit data precision on x86 systems are deterministic, because there is a SIMD (SSE) implementation of the 8-bit-per-sample floating point algorithms for x86-64 and i386. For other architectures and for 12-bit data precision, however, the build system has to make an intelligent guess as to the expected floating point results, if it can. (It doesn't even try for i386, because there are too many possibilities.) You can override the build system's guess by setting the Lines 868 to 889 in 6c87537 For arm64 builds, the build system should set Now, as far as the |
|
Also, this is what I mean when I say that I can't talk to five people at once. It isn't always clear from the comments above who is experiencing which of the two separate issues and which of the issues were resolved by disabling parallel testing. If the |
Yes, I posted here as it seems the same issue or at least affects the same test as the OP. Sorry, if I added confusion. |
|
If you want a quick test setup, fire up a VM (1-2G of RAM will do if you don't use ZFS as filesystem).
|
|
The issue originally reported by @kloczek was indeed a test concurrency issue, which has been fixed (along with several other test concurrency issues.) The floating point test failures are unrelated, and I think those failures are related to: |
|
Looks like I can still reproduce it on homebrew CI for 13-arm https://github.com/Homebrew/homebrew-core/actions/runs/5549879272/jobs/10160053957?pr=135676 |
Please read my comments above. The Homebrew issue was not automatically fixed. You have to change the value of the CMake |
|
noted, thanks, doing the cmake change now. |
|
this is my tweaked the cmake build (I am on a ventura arm machine.) still run into the same error |
|
@chenrui333 Then you'll need to send me the verbose output of the failing tests, which will show the expected vs. actual MD5 sum of the generated test images. I asked for that information already (#705 (comment)), but because people chose to pile on to this issue with reports of an unrelated issue, my comment was probably buried in the noise. |
|
@dcommander, apologies for muddying up this issue with a different one. I'll try your suggestions from #705 (comment) and open a new issue to follow it up. Edit: I opened #709 for this. |
Because libjpeg-turbo 3.0.x now supports multiple data precisions in the same build, the regression test system can test the 8-bit and 12-bit floating point DCT/IDCT algorithms separately. The expected MD5 sums for those tests are communicated to the test system using the FLOATTEST8 and FLOATTEST12 CMake variables. Whereas it is possible to intelligently set a default value for FLOATTEST8 when building for x86[-64] and a default value for FLOATTEST12 when building for x86-64, it is not possible with other architectures. (Refer to #705, #709, and #710.) Clang 14, for example, now enables FMA (fused multiply-add) instructions by default on architectures that support them, but with AArch64 builds, the results are not the same as when using GCC/AArch64 with FMA instructions enabled. Thus, setting FLOATTEST12=fp-contract doesn't make the tests pass. It was already impossible to intelligently set a default for FLOATTEST8 with i386 builds, but referring to #710, that appears to be the case with other non-x86-64 builds as well. Back in 1991, when Tom Lane first released libjpeg, some CPUs had floating point units and some didn't. It could take minutes to compress or decompress a 1-megapixel JPEG image using the "slow" integer DCT/IDCT algorithms, and the floating point algorithms were significantly faster on systems that had an FPU. On systems without FPUs, floating point math was emulated and really slow, so Tom also developed "fast" integer DCT/IDCT algorithms to speed up JPEG performance, at the expense of accuracy, on those systems. Because of libjpeg-turbo's SIMD extensions, the floating point algorithms are now significantly slower than the "slow" integer algorithms without being significantly more accurate, and the "fast" integer algorithms fail the ISO/ITU-T conformance tests without being any faster than the "slow" integer algorithms on x86 systems. Thus, the floating point and "fast" integer algorithms are considered legacy features. In order for the floating point regression tests to be useful, the results of the tests must be validated against an independent metric. (In other words, it wouldn't be useful to use the floating point DCT/IDCT algorithms to determine the expected results of the floating point DCT/IDCT algorithms.) In the past, I attempted without success to develop a low-level floating point test that would run at configure time and determine the appropriate default value of FLOATTEST*. Barring that approach, the only other possibilities would be: 1. Develop a test framework that compares the floating point results with a margin of error, as TJUnitTest does. However, that effort isn't justified unless it could also benefit non-legacy features. 2. Compare the floating point results against an expected MD5 sum, as we currently do. However, as previously described, it isn't possible in most cases to determine an appropriate default value for the expected MD5 sum. For the moment, it makes the most sense to disable the 8-bit floating point tests by default except with x86[-64] builds and to disable the 12-bit floating point tests by default except with x86-64 builds. That means that the floating point algorithms will still be regression tested when performing x86[-64] builds, but other types of builds will have to opt in to the same regression tests. Since the floating point DCT/IDCT algorithms are unlikely to change ever again (the only reason they still exist at all is to maintain backward compatibility with libjpeg), this seems like a reasonable tradeoff.
FWIW, there's active discussion at https://gcc.gnu.org/pipermail/gcc/2023-September/242466.html as to whether GCC should change its default behaviour. |
The MD5 sums associated with FLOATTEST8=fp-contract and FLOATTEST12=fp-contract are appropriate for GCC (tested v5 through v13) with -ffp-contract=fast, which is the default when compiling for an architecture that has fused multiply-add (FMA) instructions. However, different MD5 sums are needed for Clang (tested v5 through v14) with -ffp-contract=on, which is now the default in Clang 14 when compiling for an architecture that has FMA instructions. Refer to #705, #709, #710
|
I performed some experiments in an attempt to figure out appropriate default values of Clang 14No The MD5 sum of the
The MD5 sum of the
The MD5 sums of all tests match the predefined values for It seems that Clang 14 is at least consistent between AArch64 and x86-64 as long as FMA instructions are enabled on the latter. Clang 5 through 13Same as the behavior of Clang 14, except GCC 5 through 13No The MD5 sums of all tests match the predefined values for
The MD5 sums of all tests match the predefined values for ConclusionAs a result of this research, I have modified CMakeLists.txt so that, when Clang is used rather than GCC, the default values of |
The MD5 sums associated with FLOATTEST=fp-contract are appropriate for GCC (tested v5 through v13) with -ffp-contract=fast, which is the default when compiling for an architecture that has fused multiply-add (FMA) instructions. However, different MD5 sums are needed for Clang (tested v5 through v14) with -ffp-contract=on, which is now the default in Clang 14 when compiling for an architecture that has FMA instructions. Refer to #705, #709, #710
The MD5 sums associated with FLOATTEST=fp-contract are appropriate for GCC (tested v5 through v13) with -ffp-contract=fast, which is the default when compiling for an architecture that has fused multiply-add (FMA) instructions. However, different MD5 sums are needed for Clang (tested v5 through v14) with -ffp-contract=on, which is now the default in Clang 14 when compiling for an architecture that has FMA instructions. Refer to #705, #709, #710
|
If the default in GCC is changed from |
Looks like something is wrong with just released 3.0.0
The text was updated successfully, but these errors were encountered: