Parallel Build Failing on Distributed Filesystem

[j-w-jones]

I'm trying to compile OpenBLAS 0.3.12 using GCC 10.2.0.

When it gets to the linktest part it fails with unresolved symbols. And when I look in the .so file using 'nm' I can see a load of symbols are undefined.

perl ./gensymbol linktest x86_64 _ 0 0 0 0 0 0 "" "" 1 0 1 1 1 1 > linktest.c
gcc -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=80 -DMAX_PARALLEL_NUMBER=1 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION="0.3.12" -msse3 -mssse3 -msse4.1 -march=skylake-avx512 -mavx2 -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME="" -DCHAR_CNAME="" -DNO_AFFINITY -I.. -shared -o ../libopenblas_skylakexp-r0.3.12.so
-Wl,--whole-archive ../libopenblas_skylakexp-r0.3.12.a -Wl,--no-whole-archive
-Wl,-soname,libopenblas.so.0 -lm -lpthread -lgfortran -lm -lpthread -lgfortran
gcc -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=80 -DMAX_PARALLEL_NUMBER=1 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION="0.3.12" -msse3 -mssse3 -msse4.1 -march=skylake-avx512 -mavx2 -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME= -DASMFNAME= -DNAME=_ -DCNAME= -DCHAR_NAME="" -DCHAR_CNAME="" -DNO_AFFINITY -I.. -w -o linktest linktest.c ../libopenblas_skylakexp-r0.3.12.so -L/opt/software/base/gcc/10.2.0/lib/gcc/x86_64-pc-linux-gnu/10.2.0 -L/opt/software/base/gcc/10.2.0/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/opt/software/base/gcc/10.2.0/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../.. -lgfortran -lm -lquadmath -lm -lc && echo OK.
/tmp/ccaHbXUr.o: In function main': linktest.c:(.text.startup+0xf88): undefined reference to slagge'
linktest.c:(.text.startup+0xf8f): undefined reference to slagsy_' linktest.c:(.text.startup+0xf96): undefined reference to slahilb_'
linktest.c:(.text.startup+0xf9d): undefined reference to slakf2_' linktest.c:(.text.startup+0xfa4): undefined reference to slaran_'
linktest.c:(.text.startup+0xfab): undefined reference to slarge_' linktest.c:(.text.startup+0xfb2): undefined reference to slarnd_'

However, if I turn off parallel make then it builds fine.

I have 40 cores in my server so could there be an issue with the make dependencies causing the dynamic library to be build from the statis library before the static library has finished being built?

[martin-frbg]

This is a bit weird, but starting a make in exports while the static build is still running looks like the only explanation. (Wonder why that has not come up before, unless everybody is silently limiting their make to just a handful of parallel jobs).
Could you check if adding shared to the .NOTPARALLEL in line 40 of the toplevel Makefile fixes this ?

[j-w-jones]

Hi Martin I have tried that, but it didn’t make a difference I’m afraid. I have run ‘nm’ on libopenblas_skylakexp-r0.3.12.so from a sequential build and a parallel build. They’re attached. You can see that some symbols are undefined in the parallel one, but defined in the sequential one, for example: “slagge_”. Regards, Jason

…

-- Dr Jason W Jones Associate Professor College of Engineering Swansea University Singleton Park Swansea UK SA2 8PP Tel: +44-1792-295869 [cid:image001.jpg@01D6B377.8A6F5300] From: Martin Kroeker <notifications@github.com> Sent: 05 November 2020 12:35 To: xianyi/OpenBLAS <OpenBLAS@noreply.github.com> Cc: Jones J.W. <J.W.Jones@Swansea.ac.uk>; Author <author@noreply.github.com> Subject: Re: [xianyi/OpenBLAS] Parallel Build Failing (#2973) This is a bit weird, but starting a make in exports while the static build is still running looks like the only explanation. (Wonder why that has not come up before, unless everybody is silently limiting their make to just a handful of parallel jobs). Could you check if adding shared to the .NOTPARALLEL in line 40 of the toplevel Makefile fixes this ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fxianyi%2FOpenBLAS%2Fissues%2F2973%23issuecomment-722350475&data=04%7C01%7CJ.W.Jones%40Swansea.ac.uk%7Caedb59580626423e5e6208d8818747e1%7Cbbcab52e9fbe43d6a2f39f66c43df268%7C0%7C1%7C637401765309498952%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=VGOIwGif0WNFc489ezkvI6a9X5mRG%2FOqwjDqwKxmO44%3D&reserved=0>, or unsubscribe<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMGLFLZBZW6RTRFOOZD65DSOKLXTANCNFSM4TLIMH3A&data=04%7C01%7CJ.W.Jones%40Swansea.ac.uk%7Caedb59580626423e5e6208d8818747e1%7Cbbcab52e9fbe43d6a2f39f66c43df268%7C0%7C1%7C637401765309498952%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=3MB2ziAmUsh2a%2Fr1oaDp84lcBm2bX1cLs345UbOwqfs%3D&reserved=0>.

[martin-frbg]

Hi Jason, unfortunately the nm output was not attached. The symbols you mentioned above, slagge, slagsy et al. all belong to
TESTING/MATGEN, I'm wondering if we need .NOTPARALLEL for the entire lapack-netlib hierarchy (or at least in the Makefile in
lapack-netlib/TESTING/MATGEN). Still strange that this has not come up before - some of the CI jobs run on Epyc or ThunderX
hardware, need to check if these use make -j though...

[martin-frbg]

Not reproduced on a 96core ARM server, nor on a 48core AMD Epyc

[brada4]

Longer build log would be necessary, especially errors around slag* functions.
Intel makes no 40-core CPU I assume it is 20core some-lake platinum?

[j-w-jones]

Hi Andrew I redid the built in parallel but in /tmp, which is a normal SAS disk and it compiled fine. The original builds I was doing were on a large Lustre filesystem. I know Lustre tends to cache more and flush to the disk far less but that shouldn’t affect any files that have been closed, i.e. the process using them has finished. It does mean two processes writing to the same file, even if the actual writes are separated in time is far less likely to succeed than for a normal disk. I did try to see if I could get ‘make’ to print out which build steps where in which threads, or maybe print a timestamp for each build but I couldn’t find anything. The server has 2 x 20 Intel Xeon Gold 6230 cpus. Cheers, Jason From: Andrew <notifications@github.com> Sent: 06 November 2020 13:05 To: xianyi/OpenBLAS <OpenBLAS@noreply.github.com> Cc: Jones J.W. <J.W.Jones@Swansea.ac.uk>; Author <author@noreply.github.com> Subject: Re: [xianyi/OpenBLAS] Parallel Build Failing (#2973) Longer build log would be necessary, especially errors around slag* functions. Intel makes no 40-core CPU I assume it is 20core some-lake platinum? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fxianyi%2FOpenBLAS%2Fissues%2F2973%23issuecomment-723070268&data=04%7C01%7CJ.W.Jones%40Swansea.ac.uk%7C7fd23f81827447dfc5ac08d882549100%7Cbbcab52e9fbe43d6a2f39f66c43df268%7C0%7C0%7C637402647001036297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vlf4zQpIPlTsBZIsldHZBG95%2FuFDh%2BZbKhvHofbGA6I%3D&reserved=0>, or unsubscribe<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMGLFOJB3JB2NJ3T2J5LBDSOPX65ANCNFSM4TLIMH3A&data=04%7C01%7CJ.W.Jones%40Swansea.ac.uk%7C7fd23f81827447dfc5ac08d882549100%7Cbbcab52e9fbe43d6a2f39f66c43df268%7C0%7C0%7C637402647001036297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=L%2BxB8c5chj5Go43VWte0xgFb7v7UJbhboHJidaXuUUo%3D&reserved=0>.

[martin-frbg]

That would appear to be a consistency issue with your distributed filesystem then (or perhaps gmake's inability to handle such a case). I still have not seen this with the drone.io CI and whatever fs backend they use, where I believe any trivial cases of missing make dependencies should show up. Is it always the same (small?) set of missing functions your (don't) see ?

[brada4]

In general make relies on very accurate timestamps, in case of NFS and seemingly lustre those are set by backend which may have time off more than timestamp resolution. Probably you got some warnings in regard to timestamps when building.
One solution is to assure very accurate time synchronisation, dev/shm is kind of gratis filesystem with local timestamps, whough mounting /tmp same way looks more legit.

[j-w-jones]

Hi Andrew, The timestamps are fine – make never complains about this. I have had this in the past with NFS mounted filesystems where the client and server clocks were out of sync. I am wondering if two threads are updating the archive library at similar times (for example within a second of each other) and a normal, local disk deals with it better. Anyway, I guess you can close this issue. If I do have time to investigate further and find anything I will let you know. Cheers, Jason From: Andrew <notifications@github.com> Sent: 10 November 2020 11:55 To: xianyi/OpenBLAS <OpenBLAS@noreply.github.com> Cc: Jones J.W. <J.W.Jones@Swansea.ac.uk>; Author <author@noreply.github.com> Subject: Re: [xianyi/OpenBLAS] Parallel Build Failing (#2973) In general make relies on very accurate timestamps, in case of NFS and seemingly lustre those are set by backend which may have time off more than timestamp resolution. Probably you got some warnings in regard to timestamps when building. One solution is to assure very accurate time synchronisation, dev/shm is kind of gratis filesystem with local timestamps, whough mounting /tmp same way looks more legit. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fxianyi%2FOpenBLAS%2Fissues%2F2973%23issuecomment-724654729&data=04%7C01%7CJ.W.Jones%40Swansea.ac.uk%7C5dcf87a5c29d4101fd5108d8856f6c54%7Cbbcab52e9fbe43d6a2f39f66c43df268%7C0%7C0%7C637406060886441843%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BAiP3dItJ7RkYX3DJ%2BzPJnEwCPSP8%2B%2FAmIeY9324LZo%3D&reserved=0>, or unsubscribe<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMGLFK7VFWCVL3BNAAIH5LSPESYBANCNFSM4TLIMH3A&data=04%7C01%7CJ.W.Jones%40Swansea.ac.uk%7C5dcf87a5c29d4101fd5108d8856f6c54%7Cbbcab52e9fbe43d6a2f39f66c43df268%7C0%7C0%7C637406060886451836%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=OGzBSCHg04MFasKJzr3yjbqeaQZlFO3iBXp4VRzJc3Y%3D&reserved=0>.

[martin-frbg]

Unfortunately I do not have access to a lustre-based DFS, so the best option seems to be to mention this topic in the wiki and/or README. (Need to look into Makefile debugging with something like https://github.com/rocky/remake)

[brada4]

make complains only if it sees source files timestamped in future, e.g. you do something CI style on the cluster - expand tarball on the node living in future, then compile from same place on other node which sees unlikely future files. It is not only NFS problem, sometimes reboot and misguided timezone setups trigger that.

You will diagnose unwillingly building next software package. Lustre says to use NTP, please ask your admin to check.
https://wiki.lustre.org/Operating_System_Configuration_Guidelines_For_Lustre#Date_and_Time_Synchronization_with_NTP

[j-w-jones]

Hi Andrew I am the system admin and the cluster does use NTP. Lustre would be throwing errors regularly if clocks were not synced. Regards, Jason From: Andrew <notifications@github.com> Sent: 10 November 2020 15:39 To: xianyi/OpenBLAS <OpenBLAS@noreply.github.com> Cc: Jones J.W. <J.W.Jones@Swansea.ac.uk>; Author <author@noreply.github.com> Subject: Re: [xianyi/OpenBLAS] Parallel Build Failing (#2973) make complains only if it sees source files timestamped in future, e.g. you do something CI style on the cluster - expand tarball on the node living in future, then compile from same place on other node which sees unlikely future files. It is not only NFS problem, sometimes reboot and misguided timezone setups trigger that. You will diagnose unwillingly building next software package. Lustre says to use NTP, please ask your admin to check. https://wiki.lustre.org/Operating_System_Configuration_Guidelines_For_Lustre#Date_and_Time_Synchronization_with_NTP<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.lustre.org%2FOperating_System_Configuration_Guidelines_For_Lustre%23Date_and_Time_Synchronization_with_NTP&data=04%7C01%7CJ.W.Jones%40Swansea.ac.uk%7Ca50d45ef770c4ac59c8908d8858ec4c0%7Cbbcab52e9fbe43d6a2f39f66c43df268%7C0%7C0%7C637406195529886471%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Xs%2BQ5n5YPjAq3ifkXNjJwGIuVR0AKMA491aZV3nLgJA%3D&reserved=0> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fxianyi%2FOpenBLAS%2Fissues%2F2973%23issuecomment-724782641&data=04%7C01%7CJ.W.Jones%40Swansea.ac.uk%7Ca50d45ef770c4ac59c8908d8858ec4c0%7Cbbcab52e9fbe43d6a2f39f66c43df268%7C0%7C0%7C637406195529886471%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KZhIo5J48zqBEFwBqoAsmjr76tnh2MWFcEFdian682Y%3D&reserved=0>, or unsubscribe<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMGLFJVHTTO5S4ZWML2S73SPFNBXANCNFSM4TLIMH3A&data=04%7C01%7CJ.W.Jones%40Swansea.ac.uk%7Ca50d45ef770c4ac59c8908d8858ec4c0%7Cbbcab52e9fbe43d6a2f39f66c43df268%7C0%7C0%7C637406195529896465%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=TCpesuixzjt9255JlcHXzJHnWcsdrHV9cz7saMhpEiY%3D&reserved=0>.

[brada4]

You can always provide native OS package built on a better filesystem. OpenBLAS does not do anything specific with make to push it out of normal operation. NFS with bad times is known to fail, Lustre may or may not fail same way.
Full log (like script) should tell more about destinies of missing files.
Please delete original message if replying this via e-mail

[martin-frbg]

Err, @brada4, can you give this a rest please ?