Please add a cmake variable to use externally installed flint

[yurivict]

flint is packaged on most systems, see https://repology.org/projects/?search=flint

[mahrud]

@DanGrayson the CMake side of this is done. The only exception being when we build with mpir, which requires compiling Flint in order to avoid linking conflicts. So I believe this is a duplicate of #1242.

@yurivict if you'd like to use gmp rather than mpir, and use the system Flint if it has version at least 2.6, add -DWITH_MPIR=OFF to the arguments of CMake.

[DanGrayson]

There seems to be no autotools side of this, since the title refers to cmake, so closing.

[yurivict]

FYI: mpir is also available in ports.

[yurivict]

if you'd like to use gmp rather than mpir, and use the system Flint if it has version at least 2.6, add -DWITH_MPIR=OFF to the arguments of CMake.

This didn't work. It still tries to use the bundled flint project, despite flint2-2.6.0 being present.

-- Found LibXml2: /usr/local/lib/libxml2.so (found suitable version "2.9.10", minimum required is "2.9") 
--  Checking library compatibility
--  Checking library compatibility -  Libraries are compatible!
CMake Error at /usr/local/share/cmake/Modules/ExternalProject.cmake:2659 (message):
  No download info given for 'build-flint' and its source directory:

   /usr/ports/math/macaulay2/work/M2-version-1.15.1.0-460-g6da6a66ec/M2/submodules/flint2

  is not an existing non-empty directory.  Please specify one of:

[DanGrayson]

@DanGrayson the CMake side of this is done. The only exception being when we build with mpir, which requires compiling Flint in order to avoid linking conflicts.

I'd still like to understand how there can be linking conflicts.

[mahrud]

FYI: mpir is also available in ports.

If everything linked with gmp, we could use the system gmp and flint. But an issue occurs when flint is linked with gmp while factory is linked with flint and mpir and M2-binary is linked with flint, factory, and mpir. At best, this produces conflict errors during linking, but at worst it goes unnoticed until freaky bugs present. Here's my thought process on the way the CMake build handles this:

First, if we want to use mpir but the installed flint is linked with gmp, we need to compile flint to avoid link conflicts.

Now that we want to compile flint, we should be careful about the compile flags we give it. On my system (Fedora 32) the installed mpir is mixed with gmp in /usr/lib:

Because of this, we can't just tell flint to compile with -I/usr/include -L/usr/lib, because then it would link with the real gmp in /usr/lib (even if we use the gmp-to-mpir header trick, the library that flint looks for is libgmp).

Instead, CMake defaults to also compiling mpir in usr-host so we can tell flint to compile with -I usr-host/include -L usr-host/lib, so that it finds the "fake gmp" that mpir installs there.

Does this make sense? I don't know of a way to avoid a conflict in this case if we don't compile both flint and mpir. If you know a way, I'd be happy to simplify the build!

[DanGrayson]

If everything linked with gmp, we could use the system gmp and flint. But an issue occurs when flint is linked with gmp while factory is linked with flint and mpir and M2-binary is linked with flint, factory, and mpir. At best, this produces conflict errors during linking, but at worst it goes unnoticed until freaky bugs present.

This is the part I don't understand. I'll do some experiments ...

[yurivict]

There should be a workaround for this. I would like to at least be able to build it, and only using the external gmp and flint and not mpir would be fine.

Rebuilding all dependencies isn't an option due to the need to patch dependencies which is highly undesirable (copying patches from the port to dependent project bundling the same package). Another reason is everything can't be bundled into everything.

[mahrud]

There should be a workaround for this. I would like to at least be able to build it, and only using the external gmp and flint and not mpir would be fine.

Ah, that's my bad ... the correct option is -DUSING_MPIR=OFF (it is mentioned correctly in INSTALL-CMake.md)! Try that instead!

[yurivict]

With -DUSING_MPIR=OFF it fails the same way:

  No download info given for 'build-flint' and its source directory:

   /usr/ports/math/macaulay2/work/M2-version-1.15.1.0-460-g6da6a66ec/M2/submodules/flint2

[mahrud]

Could you send the full output of CMake, perhaps in a gist?

[DanGrayson]

The development branch has further changes:

einsteinium$ git diff --stat 6da6a66
 .github/workflows/test_build.yml     |  8 ++++----
 M2/INSTALL                           |  2 +-
 M2/Macaulay2/e/Makefile.in           |  3 ++-
 M2/Macaulay2/m2/html.m2              |  3 +--
 M2/Macaulay2/packages/CMakeLists.txt |  3 ++-
 M2/cmake/build-libraries.cmake       | 18 ++++++++----------
 M2/cmake/check-libraries.cmake       |  6 +++---
 M2/configure.ac                      | 49 ++++++++++++++++++++++++++++---------------------
 8 files changed, 49 insertions(+), 43 deletions(-)

[mahrud]

I'd still like to understand how there can be linking conflicts.

This is the part I don't understand. I'll do some experiments ...

@DanGrayson You've already seen these errors, but didn't realize they're caused by gmp/mpir conflicts. Here's an example from your own commit this morning: 5dfe1b2

This particular instance might not have to do with mpir, but the errors are exactly the same kind.

[yurivict]

https://people.freebsd.org/~yuri/macaulay2-configure.log

[mahrud]

@yurivict thanks! Could you also upload /usr/ports/math/macaulay2/work/.build/CMakeFiles/CMakeError.log and /usr/ports/math/macaulay2/work/.build/CMakeFiles/CMakeOutput.log?

[yurivict]

https://people.freebsd.org/~yuri/macaulay2-CMakeError.log
https://people.freebsd.org/~yuri/macaulay2-CMakeOutput.log

[mahrud]

Looks like the error is literally just that the submodules/flint2 directory doesn't have a submodule in it. I'm fairly certain that it would not actually build flint. Is using M2 as a git repository not feasible?

I'll try to figure out why it's behaving badly to not finding the submodule even when it isn't needed.

[yurivict]

I use tarballs from GitHub with some submodules added into the source tree. flint isn't added because it is a separate package.

Using Git repositories isn't feasible for packaging due to security issues: all build sources need to be fingerprinted and 100% reproducible.

[mahrud]

Using Git repositories isn't feasible for packaging due to security issues: all build sources need to be fingerprinted and 100% reproducible.

Even if you provide a git commit hash? That would provide the same security and reproduciblity.

[mahrud]

@yurivict here's a hack for now: just do touch M2/submodules/flint2/x and that'll do it!! I'll see if I can make this unnecessary, but I'm leaning towards not fixing it and just providing a FAQ item in INSTALL-CMake about it.

Does that work?

[yurivict]

But why have a local copy of flint if it isn't going to be used?

[yurivict]

Same with this hack:

[yuri@yv /usr/ports/math/macaulay2]$ find . -name x
./work/M2-version-1.15.1.0-460-g6da6a66ec/M2/submodules/x
[yuri@yv /usr/ports/math/macaulay2]$ ls -l ./work/M2-version-1.15.1.0-460-g6da6a66ec/M2/submodules/x
-rw-r--r--  1 yuri  users  0 Jun 15 10:27 ./work/M2-version-1.15.1.0-460-g6da6a66ec/M2/submodules/x
[yuri@yv /usr/ports/math/macaulay2]$ 

[mahrud]

Oh my bad again, I meant touch M2/submodules/flint2/x, in order to satisfy the error:

...  is not an existing non-empty directory.  Please specify one of:
   * SOURCE_DIR with an existing non-empty directory
...

[yurivict]

The problem with flint2 went away, but now this:

-- Downloading...
   dst='/usr/ports/math/macaulay2/work/M2-version-1.15.1.0-460-g6da6a66ec/M2/BUILD/tarfiles/glpk-4.59.tar.gz'
   timeout='none'
--- CMakeFiles/build-mpsolve-install.dir/all ---
-- Downloading...

You should never need to download glpk. It is a 20+ yo project available on all systems: https://repology.org/project/glpk/versions And if it isn't available on some system, maybe nobody cares about this system, and you shouldn't either.

All cmake should do is to search for the glpk package, and fail with an error message if not found. No need to download it.

[mahrud]

All cmake should do is to search for the glpk package, and fail with an error message if not found. No need to download it.

This is the error:

-- Could NOT find GLPK (missing: GLPK_INCLUDE_DIR GLPK_LIBRARIES) (Required is at least version "4.59.0")

@yurivict can you tell me where it is installed? I think the reason we offer compiling it is that it links with mpir, and so that whole story ... but in this case it's just not finding it!

[DanGrayson]

I'd still like to understand how there can be linking conflicts.

This is the part I don't understand. I'll do some experiments ...

@DanGrayson You've already seen these errors, but didn't realize they're caused by gmp/mpir conflicts. Here's an example from your own commit this morning: 5dfe1b2

Okay -- how would I come to realize it?

[mahrud]

Okay -- how would I come to realize it?

@DanGrayson I think my first clue came when I built shared libraries for everything and noticed that some of flint, ntl, and factory where linked with incorrect gmp libraries. Then I wrote a short code that used a function that is different between mpir and gmp and linked it with both static mpir and a static flint that is linked with the incorrect gmp, and the code executed the gmp function rather than mpir. This was a few months ago and a low priority for me at the moment, but if you're curious, try -DBUILD_SHARED_LIBS=ON with the CMake build and remove the three lines in cmake/configure.cmake that force compiling mpir, flint, ntl, etc. then use ldd/otools to see what version of gmp is each library linked with. This will work fine on some systems, but not on others, e.g. Fedora.

[mahrud]

@yurivict since the flint issue is fixed (and I added a FAQ entry to INSTALL-CMake.md) I'll close this. Feel free to message here or open a new issue if CMake can't find glpk. We may need to patch cmake/FindGLPK.cmake if it can't find it.

[DanGrayson]

a static flint that is linked with the incorrect gmp

I think the phrase "a static flint that is linked with the incorrect gmp" has no meaning, as a static library is not linked yet: it is only dynamic libraries that are pre-linked.

the code executed the gmp function rather than mpir.

That's not an error, as mpir is a fork of gmp, and the two libraries define hundreds of identical functions.

[mahrud]

I think the phrase "a static flint that is linked with the incorrect gmp" has no meaning, as a static library is not linked yet: it is only dynamic libraries that are pre-linked.

Unless I'm deeply confused about the terminology, I'm pretty sure it's exactly the opposite. For instance, the symbols in a static library libflint.a are already linked with gmp, hence you don't need gmp around anymore. Here is the definition of a static library:

In computer science, a static library or statically-linked library is a set of routines, external functions and variables which are resolved in a caller at compile-time and copied into a target application by a compiler, linker, or binder, producing an object file and a stand-alone executable

For a dynamic library libflint.so, you still need libgmp.so around, but a problem that arises is that different versions of gmp and mpir have different compatibility versions, and if libflint.so depends on libgmp.so.8 (which is the real gmp) while M2-binary depends on libgmp.so.10 (which is actually mpir), then you have a link conflict.

the code executed the gmp function rather than mpir.

That's not an error, as mpir is a fork of gmp, and the two libraries define hundreds of identical functions.

Dan, there's no need to explain to me what mpir is! When I tell the build system to use mpir, it's because I want the routines as implemented in mpir to be used (e.g. for speed), so I don't want an important library like flint to instead use the routines as implemented in gmp.

Anyway, I think I've explained the issue as best as I understand it, and since the cmake build system works out of the box with both -DBUILD_SHARED_LIBS=ON and OFF on every system I could get my hands on, I'd like to switch to other issues for now. Let me know if you figure out a way to link libraries correctly or get autotools to build both static and shared libraries without any issues when using mpir,

[DanGrayson]

I think the phrase "a static flint that is linked with the incorrect gmp" has no meaning, as a static library is not linked yet: it is only dynamic libraries that are pre-linked.

Unless I'm deeply confused about the terminology, I'm pretty sure it's exactly the opposite. For instance, the symbols in a static library libflint.a are already linked with gmp, hence you don't need gmp around anymore. Here is the definition of a static library:

In computer science, a static library or statically-linked library is a set of routines, external functions and variables which are resolved in a caller at compile-time and copied into a target application by a compiler, linker, or binder, producing an object file and a stand-alone executable

Just look at the symbol table in libflint.a:

$ nm -o /usr/local/lib/libflint.a | grep ___gmpz_sub | head -3
/usr/local/lib/libflint.a:long_extras_inlines.o: no symbols
/usr/local/lib/libflint.a:fq_zech_poly_factor_inlines.o: no symbols
/usr/local/lib/libflint.a:fmpz_add.o:                  U ___gmpz_sub_ui
/usr/local/lib/libflint.a:fmpz_fdiv_qr_preinvn.o:                  U ___gmpz_sub_ui
/usr/local/lib/libflint.a:fmpz_fdiv_r_2exp.o:                  U ___gmpz_sub_ui

The symbols are marked with "U", which means undefined. The job of "ld" when linking later is to locate those symbols in libgmp or libmpir and link them in.

For a dynamic library libflint.so, you still need libgmp.so around, but a problem that arises is that different versions of gmp and mpir have different compatibility versions, and if libflint.so depends on libgmp.so.8 (which is the real gmp) while M2-binary depends on libgmp.so.10 (which is actually mpir), then you have a link conflict.

Why would that be a link conflict?

the code executed the gmp function rather than mpir.

That's not an error, as mpir is a fork of gmp, and the two libraries define hundreds of identical functions.

Dan, there's no need to explain to me what mpir is! When I tell the build system to use mpir, it's because I want the routines as implemented in mpir to be used (e.g. for speed), so I don't want an important library like flint to instead use the routines as implemented in gmp.

If you use the flint dynamic library, you can't control it, as the dynamic library is pre-linked. But if you use the libflint.a, then you can easily link with libgmp or libmpir, as you wish, provided the versions and API's are compatible with what the compiler saw when creating libflint.a.

Anyway, I think I've explained the issue as best as I understand it, and since the cmake build system works out of the box with both -DBUILD_SHARED_LIBS=ON and OFF on every system I could get my hands on, I'd like to switch to other issues for now. Let me know if you figure out a way to link libraries correctly or get autotools to build both static and shared libraries without any issues when using mpir,