Polynomials.fit causing LinearAlgebra.LAPACKException error on one computer

[zhopan77]

This is a very strange problem that I can reproduce every time with the original code, but I am not able to create a minimal example to reproduce it. And so far I only noticed this problem on one of my Windows 10 64-bit computers.

When I call Polynomials.fit to do a polynomial fit, some call would lead to a LinearAlgebra.LAPACKException(3) or (4) error.

If I save the data in a file, and do the same fit with the same data loaded in an REPL, or put that in a simple function inside a short script, the fit will work fine.

Not all the call to the polynomial fit function causes this problem - some number of fits can work before the problem shows up. However, it's very consistent which call causes this problem.

In one script a 1st order fit already leads to this issue; in another 1st order fits works, but a 4th order fit leads to the error.

Again, only one machine shows this problem, but the CPU is just a common Intel i9-9900K, and another machine with the same CPU and same Julia and Polynomials versions doesn't have this problem.

I tried completely removing and reinstalling Julia with all packages, but the problem persisted.

The versions: Julia v1.4.2, Polynomials.jl v1.1.3.

The full error message:

ERROR: LoadError: LinearAlgebra.LAPACKException(4)
Stacktrace:
 [1] chklapackerror at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\lapack.jl:38 [inlined]
 [2] gesdd!(::Char, ::Array{Float64,2}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\lapack.jl:1667
 [3] _svd! at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\svd.jl:105 [inlined]
 [4] svd!(::Array{Float64,2}; full::Bool, alg::LinearAlgebra.DivideAndConquer) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\svd.jl:98
 [5] #svd#93 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\svd.jl:158 [inlined]
 [6] pinv(::Array{Float64,2}; atol::Float64, rtol::Float64) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\dense.jl:1345
 [7] pinv at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\dense.jl:1324 [inlined]
 [8] fit(::Type{Polynomial}, ::Array{Float64,1}, ::SentinelArrays.ChainedVector{Float64,Array{Float64,1}}, ::Int64; weights::Nothing, var::Symbol) at C:\Users\zpan\.julia\packages\Polynomials\ZmARV\src\common.jl:83
 [9] #fit#17 at C:\Users\zpan\.julia\packages\Polynomials\ZmARV\src\common.jl:107 [inlined]
 [10] fit(::Array{Float64,1}, ::SentinelArrays.ChainedVector{Float64,Array{Float64,1}}, ::Int64) at C:\Users\zpan\.julia\packages\Polynomials\ZmARV\src\common.jl:107
 [11] main() at C:\depot\Projects\NOON\Documents\Arch\OpticalSystem_Architecture\ILS_Gen4\Controls\SupportingDocuments\Raman_Tilt_Gain\fiberIL_spectrum_CalcLossAndTilt_test.jl:211
 [12] top-level scope at .\util.jl:175
 [13] include(::String) at .\client.jl:439
 [14] top-level scope at REPL[1]:1
in expression starting at C:\depot\Projects\NOON\Documents\Arch\OpticalSystem_Architecture\ILS_Gen4\Controls\SupportingDocuments\Raman_Tilt_Gain\fiberIL_spectrum_CalcLossAndTilt_test.jl:305  

[jverzani]

Can you share some data? I'm guessing it is causing issues with the pinv call that generally don't occur.

[zhopan77]

Thanks for replying. I attached the data as a CSV file in the ZIP archive below:
testdata.zip

When the issue occurs the script is fitting a 4th order polynomial for y ~ f(x).

However here's the catch:

• The LAPACK exception only happens on one computer, when I'm running my long scripts.
• On the same computer if I use REPL or a short script to do the same fit with the attached data, it works fine.
• On a different computer (with same CPU model, Julia version, and Polynomials.jl version), if I run the same long scripts, they work fine.

I tried but failed to reproduce the problem using a shorter script. Maybe I will try again by deleting part of the script bit by bit carefully, but that'll take some time. : -(

[jverzani]

I'm not sure where to go here. I can run this all ok:

julia> u = CSV.read("testdata.csv");

julia> p = fit(Polynomial, u.x, u.y, 4)
Polynomial(758.5276612085363 - 15.211584214535229*x + 0.11455827999539943*x^2 - 0.0003839433667042987*x^3 + 4.83279143303645e-7*x^4)

julia> xs = range(extrema(u.x)..., length=200)
183.9217533742331:0.1518377719887784:214.13747

julia> GR.plot(u.x, u.y, xs, p.(xs))

The small leading coefficient might suggest some possible numeric issue, but I wouldn't expect it. Is the computer older hardware or some exotic hardware?

[zhopan77]

I'm also puzzled. It only happens on this computer. It's a new computer with Intel core i9-9900k CPU, pretty standard stuff. Another PC with the same CPU model does not have this issue.

Just when I am trying to re-produce the error by trimming down the script, another odd thing happened: the raw data was loaded from a bigger CSV file separately. That CSV reading never had issues. But just now Julia started to complain that CSV file is "not a valid file". I tried restarting Julia with no avail.

Finally I restarted my editor AND terminal (which is simply Sublime Text with Terminus). And when I run exactly the same script, CSV file reading is normal again!

This new computer is actually a gaming machine purchased from Amazon. Link below:
https://www.amazon.com/gp/product/B07TS7WKVS/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1

Would that make any difference? E.g. would gaming machine builders do any tweaks (overclocking?) that make the system unstable? But I don't have such weird problems other than in Julia...

This may not be a bug for the Polynomials.jl package anymore. I also posted this question at discourse.julialang.org

[jverzani]

Let's see if someone has a good idea. If it is something to address in Polynomials, please ping me.

[zhopan77]

One more discovery: it seems to be related to PyPlot. If I comment out the plotting code (regardless if PyPlot is imported or not), the weird errors for both Polynomials.fit() and CSV.read() go away!

I tried setting ENV["PYTHON"] to an ActivePython 3.7 installation instead of the MiniConda Python coming with Julia, then rebuilding PyCall. The errors still appear when I am doing the plots but disappear when I am not.

I do use PyPlot a lot so the problem is not resolved yet, but at least there is a very clear clue now.

[jverzani]

Whoa, never would of guessed that one. Good luck tracking this down. Please reopen if it comes back this way.

[zhopan77]

Just to follow up. A workaround has been found for my computer: install MKL.jl

Please refer to this issue at PyPlot.jl for details.