convert_data Float32 support and missing documantation

[KSepetanc]

Performant double precision GPU's (such as V100 or A100) are very expensive and not available to most scientific communities. In my opinion we should support Float32 GPU optimization.

Currently, there is convert_data function to prepare data for different backends but does not have option to prepare data for 32bit optimization (despite ExaCore() having support for it). Running AC OPF example from documentation with T=Float32 setting and MadNLP with CuCholeskySolver results in restoration failure in second iteration.

Additionally, convert_data is not documented function. If possible, it should be documented and explained its usage in provided example codes.

[sshin23]

Note that the interior point method on GPUs relies on a condensation strategy, which is more susceptible to ill-conditioning than the full-space approaches on CPUs. Due to this, on double precision, the convergence behavior becomes less reliable when tol is set to below 1e-4. On single precision, it is probably around 1e-2, which may not be practical for many instances.

Documentation will be updated in the future PR.

[KSepetanc]

You are right. I tested your hypothesis. I converted input data manually to required types CuArray of Int32 or Float32 for GPU optimization. Also, I made CPU version without CuArray. Both non-condensed CPU version and GPU version fail to compute AC OPF in single precision. CPU version fares better however, finding 1.09e5 instead of 1.07e5 solution on 89 bus system (with worsening results with increasing network size). GPU 32 Fp appears to almost completely unusable.

As this findings conclude that Fp32 won't find use in this kind of optimization I am closing this issue, but I have some questions to help me on future hardware decisions and software understanding. I am tagging @amontoison as some of them are probably his expertise.

Does cuDSS use in anyway 64-bit tensor cores (TF64) such as available in Ampere architecture (or any other GPU solver used by MadNLP)?

Is the cuSOLVER the same thing as CuCholeskySolver solver in MadNLP? I got the impression that cuSOLVER is the umbrella of several solvers such as Cholesky, RF or GLU.

While cuDSS currently supports only single GPU, I noticed there are papers using multiple GPU's with cuSOLVER. Does any solver here support multiple GPU's?

I am really looking forward to future upgrades and documentation expansion. I can not express enough how incredible work you have done. I will try to support you by submitting issues if I find any.

[amontoison]

We never got good results with single precision to solve nonlinear optimization problems.
One of my colleague, Nick Gould (@nimgould), worked on GALAHAD for more 20 years and concluded that we should only use double precision (or higher) for IPM.

The sparse Cholesky factorization (CuCholeskySolver) is in CUSOLVER but it's not anymore documented, I discovered it by looking into the header files but you can't find it online.
They added the routines with CUDA 7.0 (a long time ago).
They also have a sparse QR!
You can see the interfacet here: https://github.com/JuliaGPU/CUDA.jl/blob/master/lib/cusolver/sparse_factorizations.jl
We can still find the symbols in libcusolver.so with a nm -D but It's not anymore maintened.

For the question about multiple GPUs, the library cusolvermg in CUSOLVER can solve DENSE linear systems with multiple devices.
We have nothing equivalent for sparse linear systems.
It's very hard to saturate almost 7000 cores on an A100 when we work on sparse problems because is difficult to find such parallelism.

For TF64 support in cuDSS, I don't know what they do internally (more or less a black-box) but because it's require recent GPU architectures and CUDA 12.x, I highly expect that they want to use the last hardware accelerations available.

[KSepetanc]

Currently computation time for 13k bus system is for JuMP at 27s, 12.5s for AMPL, 8s for ExaModels cpu, 5.0s for gtx 1060 and 2.2s for rtx 3090. I wonder if even rtx 3090 is saturated which is not good double precision gpu. It could even occur that A100 takes longer due to slower single-core performance of Epyc CPU's as compared to desktop/workstation CPU's. I am basically trying to determine if there is any benefit from using compute X100 chips (e.g. A100, H100 or V100) as compared to good gaming chips (more than x10 lower double precision compute performance). Next week I will be connecting to our university supercomputer and try it on A100 (with Zen 3 Epyc). I will also try even bigger networks to try to saturate it. I will make sure to report this back to you here.