`build/pkgs/gap`: Update to 4.13.1, require >= 4.13 #38169

mkoeppe · 2024-06-07T18:42:18Z

📝 Checklist

The title is concise and informative.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation and checked the documentation preview.

⌛ Dependencies

Depends on CI: Fix handling of deletions/renames; Build&Test: Uninstall/build changed packages; Linux Incremental: Update platforms #38144 (merged here for testing)

github-actions · 2024-06-07T19:23:22Z

Documentation preview for this PR (built with commit ebee948; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

mkoeppe · 2024-06-08T01:52:36Z

The automatically run incremental builds do not work well because of the soname change.

Building from scratch: https://github.com/mkoeppe/sage/actions/runs/9425219803, https://github.com/mkoeppe/sage/actions/runs/9425219809

mkoeppe · 2024-06-08T18:40:54Z

The Linux tests are OK, but on the macOS platforms, we see
(e.g. https://github.com/mkoeppe/sage/actions/runs/9425219809/job/25969750545#step:10:3687)

  [sagemath_doc_html-none]   [spkg-install]     from sage.libs.gap.libgap import libgap
  [sagemath_doc_html-none]   [spkg-install] ImportError: dlopen(/Users/runner/work/sage/sage/src/sage/libs/gap/libgap.cpython-39-darwin.so, 0x0002): Library not loaded: 'libgap.9.dylib'
  [sagemath_doc_html-none]   [spkg-install]   Referenced from: '/Users/runner/work/sage/sage/src/sage/libs/gap/libgap.cpython-39-darwin.so'
  [sagemath_doc_html-none]   [spkg-install]   Reason: tried: 'libgap.9.dylib' (no such file), '/usr/local/lib/libgap.9.dylib' (no such file), '/usr/lib/libgap.9.dylib' (no such file), '/Users/runner/work/sage/sage/src/doc/libgap.9.dylib' (no such file), '/usr/local/lib/libgap.9.dylib' (no such file), '/usr/lib/libgap.9.dylib' (no such file)

mkoeppe · 2024-06-08T19:10:16Z

I also see this in a local build on macOS.
@culler Could this be an rpath problem?

culler · 2024-06-08T20:18:51Z

@mkoeppe Yes, it looks like a missing or incorrect rpath. Since you have a local build, you could do this:

Download and install macher
run macher info sage/src/sage/libs/gap/libgap.cpython-39-darwin.so
That will show you which libraries libgap needs, and where it is looking for them. The load paths that it tries are obtained by substituting each of its rpaths for the @rpath variable in the load command.

mkoeppe · 2024-06-08T20:24:08Z

Thanks! I get

$ macher info src/sage/libs/gap/libgap.cpython-312-darwin.so
Slice: 0
Filetype: MH_BUNDLE
Architecture: x86_64
Load Commands:
    LC_SEGMENT_64 __TEXT [0 : 110592]
    LC_SEGMENT_64 __DATA_CONST [110592 : 114688]
    LC_SEGMENT_64 __DATA [114688 : 118784]
    LC_SEGMENT_64 __LINKEDIT [118784 : 165680]
    LC_DYLD_INFO_ONLY
    LC_SYMTAB: offset is 122544, 1039 symbols, strings in [140160 : 165680]
    LC_DYSYMTAB
    LC_UUID: 402330AF-4268-39AD-BC64-FD0A50F99343
    LC_BUILD_VERSION: min = 14.0.0, sdk = 14.4.0
    LC_SOURCE_VERSION: 0.0.0.0.0
    LC_LOAD_DYLIB: libgap.9.dylib
    LC_LOAD_DYLIB: /usr/local/opt/gmp/lib/libgmp.10.dylib
    LC_LOAD_DYLIB: /usr/lib/libSystem.B.dylib
    LC_RPATH: /Users/mkoeppe/s/sage/sage-rebasing/worktree-pristine/local/lib
    LC_FUNCTION_STARTS
    LC_DATA_IN_CODE

mkoeppe · 2024-06-08T20:26:58Z

Looks to me like the libgap.9.dylib has been installed without properly setting the install_name

$ otool -L local/lib/libgap.9.dylib
local/lib/libgap.9.dylib:
	libgap.9.dylib (compatibility version 10.0.0, current version 10.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.120.2)
	/usr/local/opt/gmp/lib/libgmp.10.dylib (compatibility version 16.0.0, current version 16.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.12)
	/usr/local/opt/readline/lib/libreadline.8.dylib (compatibility version 8.2.0, current version 8.2.0)

culler · 2024-06-08T20:35:48Z

The error seems to be that
LC_LOAD_DYLIB: libgap.9.dylib
should be
LC_LOAD_DYLIB: @rpath/libgap.9.dylib

So the problem is not that the rpath is missing, the load command itself is wrong. (This is in the python extension module, libgap.so, not in libgap.dylib). Adding an -rpath option won't help.

You could repair the load path with macher or install_name_tool in a post-build script. But it would be better if you could get the linker to use the correct load command for libgap when building the extension module.

mkoeppe · 2024-06-08T20:37:59Z

Thanks! I agree

culler · 2024-06-08T20:44:30Z

I am not sure what you mean by the install name. I think you might mean the LC_ID_DYLIB load command. Running macher info on libgap.dylib in SageMath 10.3, I see:
LC_ID_DYLIB: /private/var/tmp/sage-10.3-current/local/lib/libgap.8.dylib
As far as I know, that is set by the linker to whatever filename is being written by the linker. And also as far as I know it is not used for anything and would not cause a library not to load if it differs from the actual path or does not exist. You could check, nonetheless, and see what you get on your local build.

mkoeppe · 2024-06-08T20:47:05Z

Yes, same thing, as per man ld:

   Options when creating a dynamic library (dylib)
     -install_name name
             Sets an internal "install path" (LC_ID_DYLIB) in a dynamic library. Any clients linked against the library will record that path as the way dyld should
             locate this library.  If this option is not specified, then the -o path will be used.  This option is also called -dylib_install_name for compatibility.

culler · 2024-06-08T20:50:10Z

I think the rpath may be added when the linker sees a -L command and when it actually links with a library in that directory. Can you find the link command used for libgap.cpython-312-darwin.so.

Also I don't understand the path src/sage/libs/gap/libgap.cpython-312-darwin.so -- why isn't that library in the venv?
That is probably just one of those mysteries of the sage build process, though.

mkoeppe · 2024-06-08T20:52:01Z

I don't understand the path src/sage/libs/gap/libgap.cpython-312-darwin.so -- why isn't that library in the venv?

We are using an editable build

mkoeppe · 2024-06-08T20:52:30Z

https://github.com/gap-system/gap/blob/stable-4.13/Makefile.rules#L451
;)

culler · 2024-06-08T20:54:25Z

Hmmm. The ld man page does suggest that if there were an install_name in libgap.so then it would have added the @rpath to the load command.

You can use macher set_id to add an LC_ID_DYLIB load command. You could try that on libgap.so and rebuild the extension module to see if it fixes the problem.

mkoeppe · 2024-06-08T20:59:46Z

Yes, it's a fix that I've had to make several times in past, basically for every package that tries to build a shared library without using libtool...

mkoeppe · 2024-06-08T21:25:02Z

It works now. Thanks a lot for the discussion @culler

antonio-rojas · 2024-06-09T19:39:31Z

Works for me with system GAP, modulo the already known segfaults in sage/libs/gap/element.pyx with GCC 14 (which I assume are independent of the GAP version)

mkoeppe · 2024-06-09T19:40:55Z

Thank you!

@tornaria

- Based on sagemath#37884 by @tornaria - Fixes sagemath#37616 ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    - Depends on sagemath#38144 (merged here for testing) URL: sagemath#38169 Reported by: Matthias Köppe Reviewer(s):

vbraun · 2024-06-10T22:34:48Z

spkg-check.in doesn't work any more

[spkg-check] config.status: creating Makefile
[spkg-check] config.status: creating gen/pkgconfig.h
[spkg-check] ./config.status gen/pkgconfig.h
[spkg-check] config.status: creating gen/pkgconfig.h
[spkg-check] config.status: gen/pkgconfig.h is unchanged
[spkg-check] echo > gen/pkgconfig.h.stamp
[spkg-check] "/Users/buildbot-sage/worker/sage_git/build/local/var/tmp/sage/build/gap-4.13.0.p0/src/gac" -d -p "-MQ "gen/src/io.o" -MMD -MP -MF gen/src/io.d" -p "-Igen  -g -O2" -c src/io.c -o gen/src/io.o
[spkg-check] gcc -pthread -g -O2 -fno-common -MQ gen/src/io.o -MMD -MP -MF gen/src/io.d -Igen -g -O2 -o gen/src/io.o -I/Users/buildbot-sage/worker/sage_git/build/local/var/tmp/sage/build/gap-4.13.0.p0/src/build -I/Users/buildbot-sage/worker/sage_git/build/local/var/tmp/sage/build/gap-4.13.0.p0/src/src -I/Users/buildbot-sage/worker/sage_git/build/local/var/tmp/sage/build/gap-4.13.0.p0/src -DUSE_GASMAN=1 -c src/io.c
[spkg-check] "/Users/buildbot-sage/worker/sage_git/build/local/var/tmp/sage/build/gap-4.13.0.p0/src/gac" -d -p "-MQ "bin/x86_64-apple-darwin21-default64-kv9/io.so" -MMD -MP -MF bin/x86_64-apple-darwin21-default64-kv9/.d" -P "-L/Users/buildbot-sage/worker/sage_git/build/local/lib -L/Users/buildbot-sage/worker/sage_git/build/local/lib " gen/src/io.o -o bin/x86_64-apple-darwin21-default64-kv9/io.so
[spkg-check] gcc -o bin/x86_64-apple-darwin21-default64-kv9/io.so gen/src/io.o -L/Users/buildbot-sage/worker/sage_git/build/local/lib -L/Users/buildbot-sage/worker/sage_git/build/local/lib -bundle -flat_namespace -bundle_loader /Users/buildbot-sage/worker/sage_git/build/local/var/tmp/sage/build/gap-4.13.0.p0/src/gap -L/Users/buildbot-sage/worker/sage_git/build/local/lib -L/Users/buildbot-sage/worker/sage_git/build/local/lib
[spkg-check] rm -f
[spkg-check] make[5]: *** No rule to make target `testinstall'.
************************************************************************
Error testing package gap-4.13.0.p0
************************************************************************

mkoeppe · 2024-06-10T23:35:02Z

With this change, I'm getting

[gap-4.13.0.p0] [spkg-check] rm -f
[gap-4.13.0.p0] [spkg-check]  *********   GAP 4.13.0 of 2024-03-15
[gap-4.13.0.p0] [spkg-check]  *  GAP  *   https://www.gap-system.org
[gap-4.13.0.p0] [spkg-check]  *********   Architecture: x86_64-apple-darwin23-default64-kv9
[gap-4.13.0.p0] [spkg-check]  Configuration:  gmp 6.3.0, GASMAN, readline
[gap-4.13.0.p0] [spkg-check]  Loading the library and packages ...
[gap-4.13.0.p0] [spkg-check]  Packages:   Alnuth 3.2.1, AutPGrp 1.11, CRISP 1.4.6, FactInt 1.6.3, FGA 1.5.0, GAPDoc 1.6.7, IRREDSOL 1.4.4, LAGUNA 3.9.6, Polycyclic 2.16, PrimGrp 3.4.4, 
[gap-4.13.0.p0] [spkg-check]              SmallGrp 1.5.3, Sophus 1.27, TransGrp 3.6.5
[gap-4.13.0.p0] [spkg-check]  Try '??help' for help. See also '?copyright', '?cite' and '?authors'
[gap-4.13.0.p0] [spkg-check] Error, IsDirectoryPathString: <filename> must be a string (not the value 'fail') in
[gap-4.13.0.p0] [spkg-check]   IsDirectoryPathString( dirname ) at /Users/mkoeppe/s/sage/sage-rebasing/worktree-pristine/local/share/gap/lib/files.gd:327 called from 
[gap-4.13.0.p0] [spkg-check] IsDirectoryPath( f ) at /Users/mkoeppe/s/sage/sage-rebasing/worktree-pristine/local/share/gap/lib/test.gi:939 called from
[gap-4.13.0.p0] [spkg-check] <function "TestDirectory">( <arguments> )
[gap-4.13.0.p0] [spkg-check]  called from read-eval loop at tst/testinstall.g:30
[gap-4.13.0.p0] [spkg-check] type 'quit;' to quit to outer loop

@fingolfin Any hints?

build/pkgs/gap/spkg-check.in

fingolfin · 2024-06-17T10:21:01Z

build/pkgs/gap/spkg-configure.m4

@@ -2,7 +2,7 @@ SAGE_SPKG_CONFIGURE([gap], [
  # Default to installing the SPKG, if the check is run at all.
  sage_spkg_install_gap=yes

-  m4_pushdef([GAP_MINVER],["4.12.2"])
+  m4_pushdef([GAP_MINVER],["4.13.0"])


Just to mention it, I released GAP 4.13.1 last week. There are no build system changes and also overall I'd hope there are no changes problematic for SageMath. But of course you may wish to focus on getting 4.13.0 done first.

Thanks, I've updated it

In gap 4.13 there are some improvements, e.g. converting fp groups to permutation groups, computing abelianization of fp groups, which lead to different generators. This commit fixes doctests so they pass using gap 4.13.

…ests Co-authored-by: Max Horn <max@quendi.de>

…ose test suite is disabled

…ced to dummy from list of packages whose test suite is disabled

fingolfin · 2024-06-18T04:51:47Z

There are some CI errors because something is looking for libgap.so.8 -- but GAP 4.13.x provides libgap.so.9

mkoeppe · 2024-06-18T04:56:07Z

Yes, as noted in #38169 (comment); the incremental build that we use in the CI does not handle this well

mkoeppe added the c: packages: standard label Jun 7, 2024

github-actions bot added the v: small label Jun 7, 2024

mkoeppe mentioned this pull request Jun 7, 2024

gap: support gap 4.13 in sagelib (does NOT touch sage-the-distro) #37884

Open

2 tasks

mkoeppe added the c: group theory label Jun 7, 2024

github-actions bot added v: moderate and removed v: small labels Jun 7, 2024

mkoeppe self-assigned this Jun 8, 2024

mkoeppe mentioned this pull request Jun 8, 2024

Fix shared library build on macOS gap-system/gap#5736

Open

mkoeppe added the s: needs review label Jun 8, 2024

mkoeppe added this to the sage-10.4 milestone Jun 8, 2024

antonio-rojas added s: positive review and removed s: needs review labels Jun 9, 2024

vbraun added the s: needs work label Jun 10, 2024

mkoeppe removed the s: positive review label Jun 15, 2024

mkoeppe modified the milestones: sage-10.4, sage-10.5 Jun 15, 2024

fingolfin reviewed Jun 17, 2024

View reviewed changes

build/pkgs/gap/spkg-check.in Outdated Show resolved Hide resolved

fingolfin reviewed Jun 17, 2024

View reviewed changes

mkoeppe and others added 9 commits June 17, 2024 15:18

build/pkgs/gap: Update to 4.13.0

5412dbd

build/pkgs/gap/spkg-configure.m4: Require >= 4.13

f723ab7

gap: fix doctests for gap 4.13

fc2fde2

In gap 4.13 there are some improvements, e.g. converting fp groups to permutation groups, computing abelianization of fp groups, which lead to different generators. This commit fixes doctests so they pass using gap 4.13.

build/pkgs/gap/SPKG.rst: Remove outdated info

b760ffc

build/pkgs/gap: Add gap-system/gap#5736 as a patch

7258033

build/pkgs/gap/spkg-check.in: Replace use of 'make testinstall'

f01c900

build/pkgs/gap/spkg-check.in: use the GAP just built to perform the t…

984d6c6

…ests Co-authored-by: Max Horn <max@quendi.de>

build/pkgs/gap: Update to 4.13.1

c5e60cb

tox.ini, .ci/write-dockerfile.sh: Remove gap from list of packages wh…

9b4e794

…ose test suite is disabled

mkoeppe force-pushed the gap-4.13.0 branch from fea0de4 to 9b4e794 Compare June 17, 2024 22:21

mkoeppe changed the title ~~build/pkgs/gap: Update to 4.13.0, require >= 4.13~~ build/pkgs/gap: Update to 4.13.1, require >= 4.13 Jun 17, 2024

tox.ini, .ci/write-dockerfile.sh: Remove packages that have been redu…

fe5c6cb

…ced to dummy from list of packages whose test suite is disabled

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`build/pkgs/gap`: Update to 4.13.1, require >= 4.13 #38169

`build/pkgs/gap`: Update to 4.13.1, require >= 4.13 #38169

mkoeppe commented Jun 7, 2024 •

edited

Loading

github-actions bot commented Jun 7, 2024

mkoeppe commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

culler commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

culler commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

culler commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

culler commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

culler commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

antonio-rojas commented Jun 9, 2024

mkoeppe commented Jun 9, 2024

vbraun commented Jun 10, 2024

mkoeppe commented Jun 10, 2024

fingolfin Jun 17, 2024

mkoeppe Jun 17, 2024

fingolfin commented Jun 18, 2024

mkoeppe commented Jun 18, 2024

build/pkgs/gap: Update to 4.13.1, require >= 4.13 #38169

Are you sure you want to change the base?

build/pkgs/gap: Update to 4.13.1, require >= 4.13 #38169

Conversation

mkoeppe commented Jun 7, 2024 • edited Loading

📝 Checklist

⌛ Dependencies

github-actions bot commented Jun 7, 2024

mkoeppe commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

culler commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

culler commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

culler commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

culler commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

culler commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

mkoeppe commented Jun 8, 2024

antonio-rojas commented Jun 9, 2024

mkoeppe commented Jun 9, 2024

vbraun commented Jun 10, 2024

mkoeppe commented Jun 10, 2024

fingolfin Jun 17, 2024

Choose a reason for hiding this comment

mkoeppe Jun 17, 2024

Choose a reason for hiding this comment

fingolfin commented Jun 18, 2024

mkoeppe commented Jun 18, 2024

`build/pkgs/gap`: Update to 4.13.1, require >= 4.13 #38169

`build/pkgs/gap`: Update to 4.13.1, require >= 4.13 #38169

mkoeppe commented Jun 7, 2024 •

edited

Loading