Failed to evaluate function: Modelica.Math.Matrices.LAPACK.dhseqr when upgrading to Buildings v10.0.0

[javiarrobas]

Description

At this repository we configure district models using component models from Buildings and IDEASv3.0.0 for a project we are working on. Our testing environment uses Buildingspy with OpenModelica to compile the models and check reference results. The environment is defined in the Dockerfile at this folder, and the InstallThirdPartyLibraries.mos is used to install a specific version of each library dependency: IDEAS and Buildings.
I'm trying to upgrade our testing environment from Buildings 9.1.1. to Buildings 10.0.0., but upon doing so I get the following error when compiling some of our models that I'm not able to circumvent.

[/home/developer/.openmodelica/libraries/Modelica 4.0.0+maint.om/Math/package.mo:7653:5-7842:15:writable] Error: External function ‘dhseqr‘ could not be found in any of the given shared libraries:
  /usr/bin/../lib/x86_64-linux-gnu/omc/lapack.so
  /home/developer/.openmodelica/libraries/Modelica 4.0.0+maint.om/Resources/Library/linux64/lapack.so
  /home/developer/.openmodelica/libraries/Modelica 4.0.0+maint.om/Resources/Library/lapack.so
  /usr/bin/../lib/x86_64-linux-gnu/omc/liblapack.so
  /home/developer/.openmodelica/libraries/Modelica 4.0.0+maint.om/Resources/Library/linux64/liblapack.so
  /home/developer/.openmodelica/libraries/Modelica 4.0.0+maint.om/Resources/Library/liblapack.so
  /usr/bin/../lib/x86_64-linux-gnu/omc/ffi/lapack.so
  /usr/bin/../lib/x86_64-linux-gnu/omc/ffi/liblapack.so
[/home/developer/.openmodelica/libraries/Modelica 4.0.0+maint.om/Math/package.mo:10492:9-10492:55:writable] Error: Failed to evaluate function: Modelica.Math.Matrices.LAPACK.dhseqr.

See e.g. here.

I've tried the following without success:

• Change to older Python versions, in case Lapack was installed with some Python package and the function dhseqr was deprecated in the newest versions.
• Switch between current release, stable, and nightly versions of OMC.
• Install Liblapack from source.

Notice that I get the same error when using an OMEdit installed in Windows connected to OpenModelica v1.21.0-dev-281-g001d260048 (only when switching to Buildings v9.1.1.):

Steps to reproduce

Steps to reproduce using the container image with omc v1.21.0-1

• Download the MoPED library available through the link above.
• cd MoPED/Resources/Scripts/tests
• Build the Docker image with make build
• Switch between Buildings 9.1.1 and Buildings 10.0.0 at MoPED/Resources/Scripts/tests/installThirdPartyLibraries.mos. The tests will pass when using Buildings 9.1.1, but not when using Buildings 10.0.0.
• Trigger the tests with, make test-districts
  or, alternatively, after building the image:
• make run
• docker exec -it moped bash
• Once inside the image, the library is mounted at /home/developer/moped. From there you can try to translate one of the following models (the behavior is the same, but you may want to use the latter one since it's the most lightweight):
  MoPED.Districts.Examples.Layout3
MoPED.Districts.Examples.Layout4
MoPED.Districts.Examples.Layout4RC

Steps to reproduce using OMEdit in Windows with OpenModelica v1.21.0-dev-281-g001d260048

• Open OMEdit and load IDEAS v3.0.0 and Buildings 10.0.0.
• Open the MoPED library.
• Try to simulate one of the following models:
  MoPED.Districts.Examples.Layout3
MoPED.Districts.Examples.Layout4
MoPED.Districts.Examples.Layout4RC

Version and OS

• OpenModelica Version 1.21.0
• Versions of used Modelica libraries: Buildings 9.1.1, Buildings 10.0.0 and IDEAS v3.0.0
• OS: Ubuntu 64 bit and Windows 10

Additional context

Just as complementary information:
I've opened an issue in lbl-srg/modelica-buildings#3548, but it does not sound familiar to the developers of the Buildings library.

I've also tried to isolate the problem from Modelica in the Docker testing container by using the following Python example which uses the lu decomposition from Lapack:

import numpy as np
from scipy.linalg import lu

# Create a random square matrix
n = 3
A = np.random.rand(n, n)

# Perform LU decomposition using LAPACK
P, L, U = lu(A)

# Check if the LU decomposition was successful
if np.allclose(np.dot(P, np.dot(L, U)), A):
    print("LAPACK LU decomposition successful:")
    print("P (permutation matrix):")
    print(P)
    print("L (lower triangular matrix):")
    print(L)
    print("U (upper triangular matrix):")
    print(U)
else:
    print("LAPACK LU decomposition failed!")

But when trying to use dhseqr in the following example, I get the error below:

import numpy as np
from scipy.linalg import lapack

# Create a random symmetric matrix
n = 3  # Size of the matrix
A = np.random.rand(n, n)
A = (A + A.T) / 2  # Make it symmetric

# Call dhseqr for eigenvalue decomposition
T, Z, info = lapack.dhseqr(A)

if info == 0:
    print("Eigenvalues (real part):")
    print(T)
else:
    print("LAPACK dhseqr failed with info code:", info)

root@49792943d574:/tmp/lapack-3.9.0/build# python3 testlapack.py
Traceback (most recent call last):
  File "/tmp/lapack-3.9.0/build/testlapack.py", line 10, in <module>
    T, Z, info = lapack.dhseqr(A)
AttributeError: module 'scipy.linalg.lapack' has no attribute 'dhseqr'. Did you mean: 'dpteqr'?

So it seems there is a general issue with that particular dhseqr function from Lapack because I can successfully run some methods from Lapack but not that particular one. However, I would expect that the installation of OpenModelica ensures that its dependencies are covered, which does not seem to be the case here. When installing OpenModelica upon building the Docker image I see the following prompt:

ISetting up liblapack-dev:amd64 (3.9.0-1build1) ...
update-alternatives: using /usr/lib/x86_64-linux-gnu/lapack/liblapack.so to provide /usr/lib/x86_64-linux-gnu/liblapack.so (liblapack.so-x86_64-linux-gnu) in auto mode

Which makes me guess there might be an issue with the dynamic linking of the existing Lapack library in the environment. Could we enforce reinstalling a complete Lapack version for OpenModelica?

[perost]

Fixed in #11315.

The Lapack functions are a bit special due to being Fortran functions, so we have some special handling for them in the compiler. Unfortunately this means we need to add the Lapack functions we need in some places in the compiler, and it seems we'd missed dhseqr. I will check if there are any other we've missed too.

[perost]

@mahge: Do you know why we can't find Lapack functions when trying to look them up as normal external functions? If we could find them we could probably implement support for calling external Fortran functions too and get rid of all the special handling for Lapack, but we need to at least be able to call the functions to get anywhere.

[javiarrobas]

Thanks a lot for the quick reply and the fix @perost, highly appreciated!!

[javiarrobas]

@mwetter FYI.

[mahge]

@mahge: Do you know why we can't find Lapack functions when trying to look them up as normal external functions? If we could find them we could probably implement support for calling external Fortran functions too and get rid of all the special handling for Lapack, but we need to at least be able to call the functions to get anywhere.

I was just wondering the same thing when I saw your changes. I did not know we handled them like that in the FrontEnds. We should be able to under normal circumstances. I am guessing maybe they are handled the way they are handled now because it was implemented a long time ago.

If it does not work for some Fortran related reason we also have the option of requiring lapacke instead of lapack depending on availability on the platforms we support. I will check properly.

[perost]

If it does not work for some Fortran related reason we also have the option of requiring clapack instead of lapack depending on availability on the platforms we support. I will check properly.

I don't think the fact that they're Fortran functions are the issue. We do need some special handling such as transposing array arguments since Fortran arrays are column-major instead of row-major, but that shouldn't affect the actual calling of the functions.

[perost]

The issue should be fixed, but I'll keep this open a while longer since we might want to change how we handle this.

[javiarrobas]

@perost do you know approximately how much should I wait for your edit to take effect in the nightly compiled version here: https://build.openmodelica.org/apt/dists/focal/nightly/binary-amd64/?

[perost]

@perost do you know approximately how much should I wait for your edit to take effect in the nightly compiled version here: https://build.openmodelica.org/apt/dists/focal/nightly/binary-amd64/?

It seems the Linux builds are currently broken due to some changes we made to the third-party libraries we use, but as soon as that's fixed (hopefully soon) it should be included in the next nightly builds.

[javiarrobas]

That's great, @perost! thanks for your quick answer once again!

[casella]

You can check the status of the Linux nightly builds here: https://test.openmodelica.org/jenkins/job/LINUX_BUILDS/

[javiarrobas]

Thanks @casella. That helps a lot!

[adrpo]

All green now.