Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.12.0: build fails with psm3 provider #6614

Closed
kloczek opened this issue Mar 9, 2021 · 23 comments
Closed

1.12.0: build fails with psm3 provider #6614

kloczek opened this issue Mar 9, 2021 · 23 comments

Comments

@kloczek
Copy link

kloczek commented Mar 9, 2021

Source tree configured with:

%configure \
        --disable-bgq \
        --disable-debug \
        --disable-gni \
        --disable-picky \
        --disable-static \
        --enable-atomics \
        --enable-efa \
        --enable-hook_debug \
        --enable-mrail \
        --enable-perf \
        --enable-psm \
        --enable-psm2 \
        --enable-rstream \
        --enable-rxd \
        --enable-rxm \
        --enable-shm \
        --enable-sockets \
        --enable-tcp \
        --enable-udp \
        --enable-usnic \
        --enable-verbs \
        --enable-xpmem \
        --with-numa \
        %{nil}

and it fails with:

[tkloczko@barrel libfabric-1.12.0]$ make
make  all-am
make[1]: Entering directory '/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0'
/bin/sh ./libtool  --tag=CC   --mode=link /usr/bin/gcc  -mavx -mavx2 -O2 -DNDEBUG -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -Os  -Wl,-z,relro -Wl,--as-needed -Wl,--gc-sections -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld     -o .libs/libpsm3_full.lo -r  prov/psm3/src/libpsm3_la-psmx3_am.lo prov/psm3/src/libpsm3_la-psmx3_atomic.lo prov/psm3/src/libpsm3_la-psmx3_attr.lo prov/psm3/src/libpsm3_la-psmx3_av.lo prov/psm3/src/libpsm3_la-psmx3_cm.lo prov/psm3/src/libpsm3_la-psmx3_cntr.lo prov/psm3/src/libpsm3_la-psmx3_cq.lo prov/psm3/src/libpsm3_la-psmx3_domain.lo prov/psm3/src/libpsm3_la-psmx3_ep.lo prov/psm3/src/libpsm3_la-psmx3_fabric.lo prov/psm3/src/libpsm3_la-psmx3_init.lo prov/psm3/src/libpsm3_la-psmx3_mr.lo prov/psm3/src/libpsm3_la-psmx3_msg.lo prov/psm3/src/libpsm3_la-psmx3_rma.lo prov/psm3/src/libpsm3_la-psmx3_tagged.lo prov/psm3/src/libpsm3_la-psmx3_trx_ctxt.lo prov/psm3/src/libpsm3_la-psmx3_util.lo prov/psm3/src/libpsm3_la-psmx3_wait.lo prov/psm3/src/libpsm3_la-psm3_revision.lo libpsm3i.la
libtool: link: /usr/bin/ld -m elf_x86_64 -r -o .libs/libpsm3_full.o  prov/psm3/src/.libs/libpsm3_la-psmx3_am.o prov/psm3/src/.libs/libpsm3_la-psmx3_atomic.o prov/psm3/src/.libs/libpsm3_la-psmx3_attr.o prov/psm3/src/.libs/libpsm3_la-psmx3_av.o prov/psm3/src/.libs/libpsm3_la-psmx3_cm.o prov/psm3/src/.libs/libpsm3_la-psmx3_cntr.o prov/psm3/src/.libs/libpsm3_la-psmx3_cq.o prov/psm3/src/.libs/libpsm3_la-psmx3_domain.o prov/psm3/src/.libs/libpsm3_la-psmx3_ep.o prov/psm3/src/.libs/libpsm3_la-psmx3_fabric.o prov/psm3/src/.libs/libpsm3_la-psmx3_init.o prov/psm3/src/.libs/libpsm3_la-psmx3_mr.o prov/psm3/src/.libs/libpsm3_la-psmx3_msg.o prov/psm3/src/.libs/libpsm3_la-psmx3_rma.o prov/psm3/src/.libs/libpsm3_la-psmx3_tagged.o prov/psm3/src/.libs/libpsm3_la-psmx3_trx_ctxt.o prov/psm3/src/.libs/libpsm3_la-psmx3_util.o prov/psm3/src/.libs/libpsm3_la-psmx3_wait.o prov/psm3/src/.libs/libpsm3_la-psm3_revision.o   --whole-archive ./.libs/libpsm3i.a --no-whole-archive
/usr/bin/ld: ./.libs/libpsm3i.a(libopa_la-opa_syslog.o):/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0/./prov/psm3/psm3/include/opa_user.h:167: multiple definition of `PACK_SUFFIX'; ./.libs/libpsm3i.a(libopa_la-opa_debug.o):/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0/./prov/psm3/psm3/include/opa_user.h:167: first defined here
/usr/bin/ld: ./.libs/libpsm3i.a(libopa_la-opa_time.o):/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0/./prov/psm3/psm3/include/opa_user.h:167: multiple definition of `PACK_SUFFIX'; ./.libs/libpsm3i.a(libopa_la-opa_debug.o):/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0/./prov/psm3/psm3/include/opa_user.h:167: first defined here
/usr/bin/ld: ./.libs/libpsm3i.a(libopa_la-opa_utils.o):/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0/./prov/psm3/psm3/include/opa_user.h:167: multiple definition of `PACK_SUFFIX'; ./.libs/libpsm3i.a(libopa_la-opa_debug.o):/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0/./prov/psm3/psm3/include/opa_user.h:167: first defined here
/usr/bin/ld: ./.libs/libpsm3i.a(libpsm_hal_gen1_la-opa_proto_gen1.o):/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0/./prov/psm3/psm3/include/opa_user.h:167: multiple definition of `PACK_SUFFIX'; ./.libs/libpsm3i.a(libopa_la-opa_debug.o):/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0/./prov/psm3/psm3/include/opa_user.h:167: first defined here
/usr/bin/ld: ./.libs/libpsm3i.a(libpsm_hal_gen1_la-opa_utils_gen1.o):/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0/./prov/psm3/psm3/include/opa_user.h:167: multiple definition of `PACK_SUFFIX'; ./.libs/libpsm3i.a(libopa_la-opa_debug.o):/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0/./prov/psm3/psm3/include/opa_user.h:167: first defined here
make[1]: *** [Makefile:23212: .libs/libpsm3_full.lo] Error 1
make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0'
make: *** [Makefile:6169: all] Error 2
[tkloczko@barrel libfabric-1.12.0]$
@kloczek
Copy link
Author

kloczek commented Mar 9, 2021

Also looks like when LTO is used it is even worse

[tkloczko@barrel libfabric-1.12.0]$ make
make  all-am
make[1]: Entering directory '/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0'
/bin/sh ./libtool  --tag=CC   --mode=link /usr/bin/gcc -Wall -O2 -DNDEBUG -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -Os  -Wl,-z,relro -Wl,--as-needed -Wl,--gc-sections -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto -flto-partition=none -fuse-linker-plugin     -o util/fi_info util/info.o src/libfabric.la -latomic -lrt -lpthread -ldl
libtool: link: /usr/bin/gcc -Wall -O2 -DNDEBUG -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -Os -Wl,-z -Wl,relro -Wl,--as-needed -Wl,--gc-sections -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto -flto-partition=none -fuse-linker-plugin -o util/.libs/fi_info util/info.o  src/.libs/libfabric.so -lrdmacm -lefa -lnl-3 -lnl-route-3 -lpsm_infinipath -lpsm2 -lnuma -libverbs -latomic -lrt -lpthread -ldl
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_recv.c.231a0c9f'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_utils_gen1.c.27be1719'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_error.c.05d8ee0c'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_connect.c.f8583e1d'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ptl_rcvthread.c.0cb19d5a'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_av.c.8229f959'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_am.c.2787bee6'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_tagged.c.943fcf64'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `unparse.c.855ea578'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ptl.c.7ae7f884'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_cm.c.782cecc1'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_memcpy.c.a847e9e0'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_attr.c.bf7615df'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_rma.c.508366e3'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_domain.c.d3ceabe5'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_proto_gen1.c.54a9c0e3'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_verbs_mr.c.d2bb17f6'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_debug.c.3b630c3c'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_fabric.c.7d7a7039'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_sysbuf.c.19d4996a'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ptl.c.66d530ad'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_atomic.c.07b7dfec'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_cq.c.a4507095'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_recvhdrq.c.72e04c08'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_syslog.c.89d6eb09'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_msg.c.252f72b6'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm2_hal.c.68b4ec0d'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_trx_ctxt.c.7481c73d'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `cmarwu.c.e065cc37'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_ep.c.96bd95c6'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_am.c.91fd4121'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_mq_utils.c.19d3ad4e'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_diags.c.8ee35ec6'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_mr.c.c144fbe3'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_timer.c.1b606306'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_uuid.c.feb483d4'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_sysfs.c.4cc15b4b'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_utils.c.3f0fa3d0'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_init.c.8ca60d56'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_hal_gen1.c.b49c3867'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_expected.c.d20625a6'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm.c.e32f2438'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_cntr.c.f02e2815'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `unpack.c.5af14dc4'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_am.c.89f1d233'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_mq.c.5ae7c573'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto.c.3951ea62'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_tidflow.c.fca9d221'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_mq.c.b13dea40'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_mq_recv.c.81e03a45'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_epstate.c.a63b7bb5'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_stats.c.971fa64e'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `am_reqrep.c.862ec2b0'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_scb.c.aae32194'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm3_revision.c.869793f5'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_service_gen1.c.301f0fae'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_path_rec.c.8687658e'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_ep_connect.c.e29d2035'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_wait.c.31531468'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_util.c.6849a3c2'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_crc32.c.aaf9931c'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_context.c.e48d38ae'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_utils.c.0fac555d'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_mpool.c.3282e25c'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_ep.c.047de635'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `am_reqrep_shmem.c.8fdc8a45'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_dump.c.1a9cfe7a'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ptl.c.b69a94c9'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_verbs_ep.c.fb8daad1'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_time.c.3628cf86'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_opp_path_rec.c.00b5bccf'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_dwordcpy_x86_64.c.2d1e7ef0'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:10123: util/fi_info] Error 1
make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0'
make: *** [Makefile:6169: all] Error 2

@shefty
Copy link
Member

shefty commented Mar 9, 2021

I obviously don't see this. This is based simply on untarring the package and running configure && make, correct?

@shefty
Copy link
Member

shefty commented Mar 9, 2021

Looking closer at your output, this is an rpmbuild. We do an rpmbuild test as part of our CI, though the configure params will be different. I will try to reproduce. (The build works on my system, but I have to disable efa and psm2 from the build.)

@kloczek
Copy link
Author

kloczek commented Mar 9, 2021

Using rpm means only that I'm 100% reusing build and testing procedure from prev version of the package.
Ypu dpn't need tp use rpm to be able to reproduce anything.
Did you try to use LTO?

@kloczek
Copy link
Author

kloczek commented Mar 9, 2021

Just tested build 1.11.2 in exactly the same build env and everything is OK.

@acgoldma
Copy link
Contributor

acgoldma commented Mar 9, 2021

The first Issue appears to be a missing define opa_user.h, not sure why this was not seen earlier. @kloczek, can you describe your build environment (OS, compiler, and their versions)?

acgoldma pushed a commit to acgoldma/libfabric that referenced this issue Mar 9, 2021
ofiwg#6614

Reviewed-by: Goldman, Adam <adam.goldman@intel.com>
Signed-off-by: Todd Rimmer <todd.rimmer@intel.com>
@shefty shefty changed the title 1.12.0: build fails 1.12.0: build fails with psm3 provider Mar 9, 2021
@shefty
Copy link
Member

shefty commented Mar 9, 2021

The psm3 provider, which is failing here, is new to v1.12. Adding --disable-psm3 will likely allow your build to succeed. but we need to understand why psm3 fails for you, provide a fix, and generate a new package.

@acgoldma
Copy link
Contributor

acgoldma commented Mar 9, 2021

This appears to be related to #6580.
simple workaround: replacing the objcopy ... command in the prov/psm3/Makefile.include file with a simple cp ... seems to allow LTO to work.

However, this will not work if you build with psm2 src. (--with-psm2-src=DIR)
If you are not building with psm2 src, then you can workaround this while we continue to investigate the root issue:

diff --git a/prov/psm3/Makefile.include b/prov/psm3/Makefile.include
index 6a8a3b841..f72f932ef 100644
--- a/prov/psm3/Makefile.include
+++ b/prov/psm3/Makefile.include
@@ -315,6 +315,7 @@ libpsm3_la_LIBADD = $(psm3_LIBS) $(_psm3_LIBS)
 src_libfabric_la_LIBADD += libpsm3.la
 src_libfabric_la_DEPENDENCIES += libpsm3.la

+if HAVE_PSM2_SRC
 .libs/libpsm3_full.lo: $(libpsm3_la_OBJECTS) $(libpsm3_la_DEPENDENCIES) $(EXTRA_libpsm3_la_DEPENDENCIES)
        @sed -i.bak "/dependency_libs/s/='.*'/=''/" libpsm3i.la
        $(AM_V_CCLD)$(libpsm3_la_LINK) -r $(am_libpsm3_la_rpath) $(libpsm3_la_OBJECTS) libpsm3i.la
@@ -328,6 +329,7 @@ libpsm3.la: .libs/libpsm3_exp.o
        rm -f .libs/libpsm3.a libpsm3.a; \
        $(AR) cru .libs/libpsm3.a .libs/libpsm3_exp.o; \
        $(RANLIB) .libs/libpsm3.a
+endif HAVE_PSM2_SRC

 endif !HAVE_PSM3_DL

@kloczek
Copy link
Author

kloczek commented Mar 10, 2021

I'm using my own distribution which is fedora rawhide based (with tons of bugs fixed).
gcc 11.0.1.
Kernel 5.11.3/x86_64

@kloczek
Copy link
Author

kloczek commented Mar 10, 2021

That part in prov/psm3/Makefile.include looks looks like dirty hack.
automake provides fully declarative way of description of building libraries and that part with objcopy and $AR is nothing more than just causing reported issue.
Whole part in main Makefile.am:

include prov/sockets/Makefile.include
include prov/udp/Makefile.include
include prov/verbs/Makefile.include
include prov/efa/Makefile.include
include prov/usnic/Makefile.include
include prov/psm/Makefile.include
include prov/psm2/Makefile.include
include prov/psm3/Makefile.include
include prov/gni/Makefile.include
include prov/rxm/Makefile.include
include prov/mrail/Makefile.include
include prov/rxd/Makefile.include
include prov/bgq/Makefile.include
include prov/shm/Makefile.include
include prov/tcp/Makefile.include
include prov/rstream/Makefile.include
include prov/hook/Makefile.include
include prov/hook/perf/Makefile.include
include prov/hook/hook_debug/Makefile.include

looks like someone have been trying to use automake MyWay(tm) instead like it is used almost everywhere else. IMO those hacks are nothing more than asking for troubles. None of the automake code should use anything from any .libs/ directories.
Really automake provides pretty straight forward way to build any library and no one needs to (re)invent how to do that yet another time.
I have no idea what was the intention of those hacks (and I'm not going to try to understand that) but rewrite all those hacks to use automake NormalWay(tm) seems should fix all found build issues.

@shefty
Copy link
Member

shefty commented Mar 10, 2021

I don't see any issue with using the include directive to include other makefiles. (https://www.gnu.org/software/make/manual/html_node/Include.html) That part in particular is allowing separation of developer responsibilities. @jsquyres may be able to help with specific automake issues. (Despite his attempt to create memes stating otherwise, I'm still not an automake expert.)

If there are specific issues in the psm3 makefile that you can point to directly, it sounds like that would be helpful.

@kloczek
Copy link
Author

kloczek commented Mar 10, 2021

I don't see any issue with using the include directive to include other makefiles.

In that approach issue is that you don't have per directory generated Makefile.
Processing each time single Makefile with all dependencies takes more time.
Going to the project tree root and back to do someting in subdiretory takes more time.

Only BecauaseWeCan(tn) should not mean to do sometbing :)
Neverteless it is not core of the issue of that ticket.

If there are specific issues in the psm3 makefile that you can point to directly, it sounds like that would be helpful.

If someone using automake and libtool is think that usimg in Makefile.am $AR is necrssary to solve something that maybe truth because that person may be only thinking that it is RightWay(tm) :P
Here is the problem :)
Again: automake provides declarative description of building any type of DSOs (library of loadable module). Uning funtional descriptiom with sequence of commands is wrong way (by definition).

To solve that issue question "why here objcopy and repackaging ar archive is used?" needs to be answered first ..

@acgoldma
Copy link
Contributor

I have been able to reproduce LTO issue on stock RHEL 8.1 (kernel/compiler) with the following command:

./autogen.sh && \
./configure CFLAGS=" -O2 -g -grecord-gcc-switches -pipe -Wformat -Werror=format-security \
 -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS \
 -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong \
 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic \
 -fasynchronous-unwind-tables -fstack-clash-protection  -fcf-protection  \
 -fdata-sections -ffunction-sections  -flto=auto  -flto-partition=none -Os \
 -Wl,-z,relro  -Wl,--as-needed  -Wl,--gc-sections  \
 -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto \
 -flto-partition=none -fuse-linker-plugin" \
 AR="gcc-ar" NM="gcc-nm" RANLIB="gcc-ranlib" && \
make clean && make -j

I extracted the flags from your make output, but not sure if there is a better way to generate these. I had to add "-Wformat " to support "-Werror=format-security"

I am also not an automake expert.
PSM3's custom code was to isolate the psm3 source code in a process similar to kernel modules.
Unfortunately, Automake (to my knowledge) does not allow this normally .

@jsquyres
Copy link
Member

It does look like prov/psm3/Makefile.include is different than the intent of the Makefile.include-based design. The intent was just to have separation of content by authors, not to subvert Automake / Libtool's native mechanisms.

For example, I see:

Indeed, I get my own compile errors with PSM3:

...
  CC       prov/psm3/src/libpsm3_la-psmx3_trx_ctxt.lo
  CC       prov/psm3/src/libpsm3_la-psmx3_util.lo
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_am.lo] Error 1
make[1]: *** Waiting for unfinished jobs....
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_atomic.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_attr.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_cm.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_av.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_cq.lo] Error 1
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_cntr.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_domain.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_fabric.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_init.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_ep.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_mr.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_msg.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_rma.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_trx_ctxt.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_tagged.lo] Error 1
cc1: error: unrecognized command line option "-mavx2"
make[1]: *** [prov/psm3/src/libpsm3_la-psmx3_util.lo] Error 1
make[1]: Leaving directory `/home/jsquyres/git/libfabric'
make: *** [all] Error 2

If you're going to require the mavx2 compiler flag, you should test for it. Either outright fail configure or just disqualify the provider if the compiler does not support that flag.

@acgoldma
Copy link
Contributor

@jsquyres, I am new to automake, can you provide a better target to generate the checksum.

Yes, the second bullet, specifically the objcopy line is what seems to cause the issue. Making this a simple copy allows LTO to work, breaks the isolation we were going for.

As for the mavx2 option, I missed that in the other configure checks I added. I will introduce a fix for that.

@jsquyres
Copy link
Member

@jsquyres, I am new to automake, can you provide a better target to generate the checksum.

My $0.02 is that it's fairly meaningless to checksum the source every time you build. That seems more like a configure-time test to me (and a corresponding AC_DEFINE[_UNQUOTED]). Probably same for the timestamp. There's not already a timestamp in libfabric already that you could use?

Yes, the second bullet, specifically the objcopy line is what seems to cause the issue. Making this a simple copy allows LTO to work, breaks the isolation we were going for.

@kloczek was a little snarky about it, but he's not wrong: you probably shouldn't be trying to out-Automake Automake. Meaning: you're doing un-Automake-ish things in the Automake framework. This LTO issue could be the first of multiple that you'll run into. I don't fully grok what you're trying to do, but the typical way such things are done inside Automake projects is via source code manipulation and multiple compilations (e.g., via sym links, or multiple Automake targets, ...).

As for the mavx2 option, I missed that in the other configure checks I added. I will introduce a fix for that.

👍

acgoldma pushed a commit to acgoldma/libfabric that referenced this issue Mar 24, 2021
ofiwg#6614

Reviewed-by: Goldman, Adam <adam.goldman@intel.com>
Signed-off-by: Todd Rimmer <todd.rimmer@intel.com>
(cherry picked from commit 8967633)
acgoldma pushed a commit to acgoldma/libfabric that referenced this issue Mar 26, 2021
ofiwg#6614

Reviewed-by: Goldman, Adam <adam.goldman@intel.com>
Signed-off-by: Todd Rimmer <todd.rimmer@intel.com>
(cherry picked from commit 8967633)
@kloczek
Copy link
Author

kloczek commented Mar 26, 2021

OK I've tested 74c9186. Looks like it fixes build when LTO is not used
Thank you.

When LTO is used build still fails.

[tkloczko@barrel libfabric-1.12.0]$ make
make  all-am
make[1]: Entering directory '/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0'
/bin/sh ./libtool  --tag=CC   --mode=link /usr/bin/gcc -Wall -O2 -DNDEBUG -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -Os  -Wl,-z,relro -Wl,--as-needed -Wl,--gc-sections -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto -flto-partition=none -fuse-linker-plugin     -o util/fi_info util/info.o src/libfabric.la -latomic -lrt -lpthread -ldl
libtool: link: /usr/bin/gcc -Wall -O2 -DNDEBUG -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -Os -Wl,-z -Wl,relro -Wl,--as-needed -Wl,--gc-sections -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto -flto-partition=none -fuse-linker-plugin -o util/.libs/fi_info util/info.o  src/.libs/libfabric.so -lrdmacm -lefa -lnl-3 -lnl-route-3 -lpsm_infinipath -lpsm2 -lnuma -libverbs -latomic -lrt -lpthread -ldl
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_recvhdrq.c.8a579708'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_utils.c.b634f6d1'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_error.c.05d8ee0c'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_scb.c.b7697977'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_expected.c.693b2b45'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_am.c.5ebd020a'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_av.c.8229f959'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_ep.c.487469bf'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_am.c.2787bee6'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_tagged.c.943fcf64'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `unparse.c.855ea578'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_cm.c.782cecc1'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_memcpy.c.a847e9e0'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_attr.c.bf7615df'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_rma.c.508366e3'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ptl.c.2ff288af'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_domain.c.d3ceabe5'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_opp_path_rec.c.bd02facc'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_verbs_mr.c.d2bb17f6'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_context.c.f54be187'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_verbs_ep.c.bb2226c3'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto.c.eb941372'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_fabric.c.7d7a7039'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_sysbuf.c.19d4996a'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_time.c.fd0d21f4'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_dump.c.77a35e9b'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ptl.c.66d530ad'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_atomic.c.07b7dfec'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_hal_gen1.c.05b97bf3'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_cq.c.a4507095'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_msg.c.252f72b6'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_trx_ctxt.c.7481c73d'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `cmarwu.c.e065cc37'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_ep.c.96bd95c6'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_am.c.91fd4121'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_proto_gen1.c.817f1274'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_mq.c.2b16d895'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_mq_utils.c.19d3ad4e'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_diags.c.8ee35ec6'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_mr.c.c144fbe3'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_timer.c.1b606306'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_uuid.c.feb483d4'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_sysfs.c.4cc15b4b'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm2_hal.c.816ec2f3'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_init.c.8ca60d56'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_epstate.c.5680bb59'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_syslog.c.f4685391'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ptl_rcvthread.c.06127306'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_recv.c.5df5ce99'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_mq_recv.c.3379ddca'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_cntr.c.f02e2815'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `unpack.c.5af14dc4'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_debug.c.47ad4842'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_utils.c.29f92dd2'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_tidflow.c.33a63ec0'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_path_rec.c.0e8eae29'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_stats.c.971fa64e'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `am_reqrep.c.862ec2b0'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm3_revision.c.869793f5'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_ep_connect.c.402e1db2'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_service_gen1.c.301f0fae'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm.c.15bc9f4b'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_proto_connect.c.4ae07caf'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_wait.c.31531468'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psmx3_util.c.6849a3c2'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ips_crc32.c.aaf9931c'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_mpool.c.3282e25c'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `psm_mq.c.a83075c2'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_utils_gen1.c.bb465054'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `am_reqrep_shmem.c.8fdc8a45'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `ptl.c.b69a94c9'
/usr/bin/ld: src/.libs/libfabric.so: undefined reference to `opa_dwordcpy_x86_64.c.2d1e7ef0'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:10123: util/fi_info] Error 1
make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/libfabric-1.12.0'
make: *** [Makefile:6169: all] Error 2

@jsquyres
Copy link
Member

@kloczek Also see #6661

@kloczek
Copy link
Author

kloczek commented Mar 26, 2021

Sorry to say that but stripping optimisation flags is not the right way to solve that :/
Please add proper definition of the public symbols to define libraries ABI.

@acgoldma
Copy link
Contributor

While not a 100% fix (which we are working on, but don't know time frame), I have limited the impact of our linker 'workaround' in #6655. This patch has also been staged with my other fixes (so far) on the v1.12.x branch in #6657.

@kloczek
Copy link
Author

kloczek commented Mar 26, 2021

Instead wasteing time on commiting temporary "woraroud" was possible to just put in README that LTO issue is not solved for now.

@shefty
Copy link
Member

shefty commented Apr 3, 2021

Has the LTO build issue been resolved? This is fixed in v1.12.1, correct? There's a separate open issue to cleanup the psm3 build, but has this specific problem been fixed?

@github-actions
Copy link
Contributor

This issue is stale because it has been open 360 days with no activity. Remove stale label or comment, otherwise it will be closed in 7 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants