CUDSS internal error (invalid argument: inertia correction)

[KSepetanc]

I am using ExaModels documentation AC OPF example model. I am running into error with CUDSS solver. Other GPU solvers work fine.

I am providing the code to reproduce:
examodel_acopf_example.zip

Overall the error report is too long so I am providing the part of it and part of execution log:

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  6.9584335e+06 2.84e+01 1.00e+02  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
EXIT: Internal Error.
ERROR: LoadError: CUDSSError: an invalid value was used as an argument (code 3, CUDSS_STATUS_INVALID_VALUE)
Stacktrace:
  [1] throw_api_error(res::CUDSS.cudssStatus_t)
    @ CUDSS C:\Users\Korisnik\.julia\packages\CUDSS\JUOfJ\src\error.jl:41
  [2] check
    @ C:\Users\Korisnik\.julia\packages\CUDSS\JUOfJ\src\error.jl:52 [inlined]
  [3] cudssDataGet(handle::Ptr{CUDSS.cudssContext}, data::CUDSS.CudssData, param::String, value::Base.RefValue{Tuple{Int32, Int32}}, sizeInBytes::Int64, sizeWritten::Base.RefValue{Int32})
    @ CUDSS C:\Users\Korisnik\.julia\packages\CUDA\02Uw6\lib\utils\call.jl:30
  [4] cudss_get(data::CUDSS.CudssData, parameter::String)
    @ CUDSS C:\Users\Korisnik\.julia\packages\CUDSS\JUOfJ\src\interfaces.jl:172
  [5] cudss_get(solver::CUDSS.CudssSolver{Float64}, parameter::String)
    @ CUDSS C:\Users\Korisnik\.julia\packages\CUDSS\JUOfJ\src\interfaces.jl:156
  [6] inertia(M::MadNLPGPU.CUDSSSolver{Float64})
    @ MadNLPGPU C:\Users\Korisnik\.julia\packages\MadNLPGPU\DvKUv\src\cudss.jl:91
  [7] inertia_correction!(inertia_corrector::MadNLP.InertiaBased, solver::MadNLPSolver{Float64, CuArray{Float64, 1, CUDA.Mem.DeviceBuffer} ...

Configuration:
Windows,
CPU: AMD R9 5950x
GPU GTX 1060 6GB

[sshin23]

Could you share the version info via ] st?

[KSepetanc]

  [c52e3926] Atom v0.12.38
  [052768ef] CUDA v5.3.1
  [a93c6f00] DataFrames v1.6.1
  [1037b233] ExaModels v0.7.0
  [b6b21f68] Ipopt v1.6.2
  [4076af6c] JuMP v1.21.1
  [e5e0dc1b] Juno v0.8.4
  [63c18a36] KernelAbstractions v0.9.18
  [2621e9c9] MadNLP v0.8.1
  [d72a61cc] MadNLPGPU v0.7.1
  [7fb6135f] MadNLPHSL v0.5.1
  [3b83494e] MadNLPMumps v0.4.0
  [f4238b75] NLPModelsIpopt v0.10.1
  [c36e90e8] PowerModels v0.21.1
  [fdbf4ff8] XLSX v0.10.1
  [017b0a0e] HSL_jll v2023.11.7+0

OS CUDA version is 12.4
Julia version is 1.10.2

[sshin23]

@amontoison Any idea what's going on here?

[amontoison]

We can't easily test on Windows so I don't know if the issue is in CUDSS.jl or cuDSS.
I have a similar issue of a Windows user here.

@KSepetanc Could you try with this branch for MadNLPGPU?

pkg> add https://github.com/amontoison/MadNLP.jl.git#cudss_update

It will use cuDSS 2.0.1 and help us to isolate the culprit.

[KSepetanc]

As I run dual boot, so I tested on Linux too.

There is no error code, but it fails with restoration failed message.

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  6.9584335e+06 2.84e+01 1.00e+02  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
   1r 6.9584335e+06 8.88e-16 9.99e+02   1.5 0.00e+00    -  0.00e+00 0.00e+00   0
EXIT: Restoration Failed
"Execution stats: Restoration Failed"

I will test with new package version next.

[sshin23]

Note that to instasll MadNLPGPU with cudss_update branch, you should import Pkg; Pkg.add(url ="https://github.com/amontoison/MadNLP.jl.git#cudss_update", subdir = "lib/MadNLPGPU"), as MadNLPGPU is on a subdirectory

[KSepetanc]

With updated package as @sshin23 suggested, on Windows we now have weird behavior.

I still use Atom as IDE for Julia. First run of the file results in the same reported error. Second run of the same file in the same Julia session it computes to full optimality (no errors) and good performance. Tried it several times just to be sure.

As for Linux, I still always get Restoration Failed.

Windows package report now:

  [c52e3926] Atom v0.12.38
  [052768ef] CUDA v5.3.1
  [a93c6f00] DataFrames v1.6.1
  [1037b233] ExaModels v0.7.0
  [b6b21f68] Ipopt v1.6.2
  [4076af6c] JuMP v1.21.1
  [e5e0dc1b] Juno v0.8.4
  [63c18a36] KernelAbstractions v0.9.18
  [2621e9c9] MadNLP v0.8.1 `https://github.com/amontoison/MadNLP.jl.git#cudss_update`
  [d72a61cc] MadNLPGPU v0.7.1 `https://github.com/amontoison/MadNLP.jl.git#cudss_update:lib/MadNLPGPU#master`
  [7fb6135f] MadNLPHSL v0.5.1
  [3b83494e] MadNLPMumps v0.4.0
  [f4238b75] NLPModelsIpopt v0.10.1
  [c36e90e8] PowerModels v0.21.1
  [fdbf4ff8] XLSX v0.10.1
  [017b0a0e] HSL_jll v2023.11.7+0

Linux packages (I use VS Code on Linux):

  [052768ef] CUDA v5.3.1
  [a93c6f00] DataFrames v1.6.1
  [1037b233] ExaModels v0.7.0
  [b6b21f68] Ipopt v1.6.2
  [4076af6c] JuMP v1.21.1
  [63c18a36] KernelAbstractions v0.9.18
  [2621e9c9] MadNLP v0.8.1 `https://github.com/amontoison/MadNLP.jl.git#cudss_update`
  [d72a61cc] MadNLPGPU v0.7.1 `https://github.com/amontoison/MadNLP.jl.git#cudss_update:lib/MadNLPGPU#master`
  [7fb6135f] MadNLPHSL v0.5.1
  [f4238b75] NLPModelsIpopt v0.10.1
  [c36e90e8] PowerModels v0.21.1
  [fdbf4ff8] XLSX v0.10.1
  [017b0a0e] HSL_jll v2023.11.7+0

[sshin23]

Hmm... not sure yet where is this coming from. Could you provide Maniefst.toml and MWE so that we can reproduce this behavior?

[sshin23]

We don't have Windows machine we can test on, but we can look into the issue in Linux

[KSepetanc]

Can you provide instructions how get these files? I can get them tomorrow.

Can you reproduce the error with provided files in the header post?
As far as I know the issue is not only for this machine. I tested also on other Windows machine with rtx 3090 with same result. I can assume it would be the same on Linux.

[sshin23]

You can

cd project/directory
julia --project=.

and then

pkg> add PowerModels, ExaModels, KernelAbstractions, MadNLP, MadNLPGPU, CUDA
julia> include("examodel_example_bug1.jl") # confirm that there's an error

Manifest.toml file will be on the project/directory

[sshin23]

Also, make sure that you can run CUDSS (e.g., test> CUDSS) to check whether it's MadNLP or CUDSS issue

[KSepetanc]

I've run pkg> test CUDSS in different configurations:

1. 5950x and gtx 1060 6GB, Windows, default MadNLP package
2. 5950x and gtx 1060 6GB, Windows, updated branch of MadNLP package
3. 5950x and gtx 1060 6GB, Linux, default MadNLP package
4. 5950x and gtx 1060 6GB, Linux, updated branch of MadNLP package
5. 3960x and rtx 3090, Windows, default MadNLP package

All of them fail CUDSS test.

Options 1 and 2 have errored tests but continue to take GPU resource endlessly without final test report.
Options 3 and 4 have failed tests (wrong evaluated values) and end with [11322] signal (6.-6): Aborted but without finishing ready for new command julia>
Option 5 is the only option that finishes the test run with report. There are 15 errored tests.

Manifest for option 3:
Manifest.zip

Any idea why would default CUDSS tests fail?