SCOPFLOW Functionality Tests failing with hiop@1.0.0

[jaelynlitz]

Issue type: Bug
Relates to:

• SCOPFLOW
• CMake build system
• Spack configuration

Summary:

Incline CI fails on the following tests:
Failed Pipeline:
https://gitlab.pnnl.gov/exasgd/frameworks/exago/-/jobs/125455
Raw Job output:
https://gitlab.pnnl.gov/exasgd/frameworks/exago/-/jobs/125455/raw
The objective value and expected values are not within tolerance. Meaning we could increase tolerance or change the expected value for tests that run on Incline.
Things to further do:
Confirm Hiop version
Run Crusher test and compare objective values across platforms.
FUNCTIONALITY_TEST_SCOPFLOW_HIOP_PBHIOP_TESTSUITE is failing for Newell at the moment as well - https://gitlab.pnnl.gov/exasgd/frameworks/exago/-/jobs/126069

Types of failures:

• divide by zero (various tests and platforms, incline, deception, newell) (FUNCTIONALITY_TEST_SCOPFLOW_HIOP_PBHIOP_TESTSUITE, FUNCTIONALITY_TEST_SCOPFLOW_HIOP_SERIAL_TESTSUITE, FUNCTIONALITY_TEST_SCOPFLOW_HIOP_RAJA_TESTSUITE)
• num iters (FUNCTIONALITY_TEST_SCOPFLOW_HIOP_MPI_TESTSUITE, incline)
• obj value out of tolerance (FUNCTIONALITY_TEST_SCOPFLOW_EMPAR_CONT_TESTSUITE, incline)
• timeout (FUNCTIONALITY_TEST_SOPFLOW_SCENARIO_RAJA_GPU_TOML, incline)

[cameronrutherford]

Does this close/reference a GitLab issue?

I think this is also just with hiop@1.0.0 since that is what is running in CI, and this is similar to #43

[jaelynlitz]

This was me copying over issue#459 https://gitlab.pnnl.gov/exasgd/frameworks/exago/-/issues/459

[jaelynlitz]

the only remaining failure on PNNL systems in SOPFLOW/SCOPFLOW with hiop@1.0.1 is the timeout on FUNCTIONALITY_TEST_SOPFLOW_SCENARIO_RAJA_GPU_TOML on incline https://gitlab.pnnl.gov/exasgd/frameworks/exago-github-mirror/-/jobs/138118#L88

[cameronrutherford]

Same with #43 - if this is just incline, might be worth making a new issue for Incline specific issues and moving on for now.