Can we rely on nanosoldier (from 1.9.0 run)?

[PallHaraldsson]

A. I noticed some packages that passed tests (e.g. Plots), on older version, that still had errors (yes, only for precompile, is it by design to not fail on that?).

B. And also packages that are ok, but failed on the previous version, and I'm not sure why, also what exact version is it referring to?

#49302 (comment)

I'm guessing we should be most concerned with:

packages failed [or crashed or whatever] tests only on the current version.

Package has test failures (9 packages):
Unitful v1.13.0: fail vs. ok
OnlineStats v1.6.0: fail vs. ok
...

A.
Plots had a timeout:
https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/9aa7669_vs_7b39515/Plots.primary.log

I expect it to work after Kristoffer reruns it (it had no precompile errors). What I find intriguing is the older version that did NOT fail, still with many precompile errors:

ERROR: The following 1 direct dependency failed to precompile:

Images [916415d5-f1e6-5110-898d-aaa5f9f070e0]

Failed to precompile Images [916415d5-f1e6-5110-898d-aaa5f9f070e0] to /home/pkgeval/.julia/compiled/v1.9/Images/jl_YMGuJy.
ERROR: LoadError: UndefVarError: @_foldable_meta not defined
Stacktrace:
[1] include(mod::Module, _path::String)
[..]
ERROR: LoadError: Failed to precompile JLD2 [033835bb-8acc-5ee8-8aae-3f567f8a3819] to /home/pkgeval/.julia/compiled/v1.9/JLD2/jl_2TSnNN.
Stacktrace:
[1] error(s::String)

B.
E.g. FastTransforms did test ok (though with Broken yellow tests, I assume intentionally not failures), while on the older version there were failures e.g.:

Special functions: Test Failed at /home/pkgeval/.julia/packages/FastTransforms/xvEgx/test/specialfunctionstests.jl:31
Expression: norm((Λ.(x, λ₁, λ₂) .- Λ.(big.(x), big(λ₁), big(λ₂))) ./ Λ.(big.(x), big(λ₁), big(λ₂)), Inf) < 4 * eps()
Evaluated: 9.111009455283952578004582205091293326667462052583491936114275192978728268858538e-16 < 8.881784197001252e-16

This would normally worry me, should I not, since ok on the current version? There's a reason it failed, and then not, I'm just confused why such would happen.

[KristofferC]

I'm guessing we should be most concerned with:

Unitful: PainterQubits/Unitful.jl#642
OnlineStats: joshday/OnlineStatsBase.jl#31

What I find intriguing is the older version that did NOT fail, still with many precompile errors:

If you are talking about "IterativeLearningControl" it says that it crashed on the previous version.

This would normally worry me, should I not, since ok on the current version?

What's the worry? The tests pass on the newer release-1.9. And an error like that is almost always that the package made their tolerances too tight.

[KristofferC]

This doesn't seem to warrant a full issue. If you have some issues with that particular Nanosoldier run, you can comment in the PR.