do not rename GCC's include-fixed directory by default + add script to regenerate it #3254

Flamefire · 2024-03-13T09:32:32Z

(created using eb --new-pr)

This adds an EC option to control the renaming of the include-fixed directory which is OFF by default because this leads to failures when GCC is "too new" relative to the OS, i.e. we actually want the fixincludes part of the GCC installation.

However as described in #10666 OS upgrades might break those GCC installations when the source headers (of GLIBC) change in a way that make them incompatible with the earlier output of fixincludes.

However GCC comes with a nice script to (re)generate that folder. Hence I think it makes more sense to make that easier to use by providing a wrapper in $GCCROOT/easybuild that calls that script (it is in some subfolder of libexec and depends on the architecture and version) and also takes care of file permissions of the include-fixed folder

If that is considered too much we could at least add a symlink to the script into the easybuild folder.

Fixes #3253

…o regenerate it

Micket

lgtm. Will need to test the script

Micket · 2024-03-13T23:50:26Z

Should be relatively safe for anyone to revert the changes manually in an existing installation right?

cd $EBROOTGCCCORE/lib/gcc/linux/*/ (e.g. software/GCCcore/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0)
mv include-fixed.renamed-by-easybuild include-fixed
optionally rm include/{sys,}limits.h

(or rebuild GCCcore)

You'd also want the script for the future OS/glibc updates.

Flamefire · 2024-03-14T09:29:17Z

lgtm. Will need to test the script

Note that the permissions afterwards won't be exactly as original because of the files created by mkheaders. For a test installation with group-writeable-installdir the group-write permissions were gone. Can't have it all... 🤷

Should be relatively safe for anyone to revert the changes manually in an existing installation right?

More or less yes. If the OS has changed in the meantime then this will run into the original issue.

(or rebuild GCCcore)

And afterwards rename the folder. Or rebuild with this PR.

There already is software/GCCcore/13.2.0/libexec/gcc/x86_64-pc-linux-gnu/13.2.0/install-tools/mkheaders which can be invoked manually even for current installations

You'd also want the script for the future OS/glibc updates.

Correct. The newly added script in GCCROOT/easybuild just makes doing this slightly more convenient. Note that your glob won't work because there is additonally a nvptx-* (or so) folder next to the *linux-gnu folder

Micket · 2024-03-15T00:17:32Z

Test report by @Micket

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-13.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/2b936c228384b531f85a12814090566f for a full test report.

Micket · 2024-03-15T01:14:09Z

Test report by @Micket

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/1b6c005890abc3e3c4921538cc343ed5 for a full test report.

migueldiascosta · 2024-03-15T04:35:44Z

Test report by @migueldiascosta

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-13.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
sms - Linux CentOS Linux 7.6.1810, x86_64, AMD EPYC 7601 32-Core Processor, Python 3.6.8
See https://gist.github.com/migueldiascosta/1fe813b23c69afe5c3141a102680f121 for a full test report.

akesandgren · 2024-03-15T08:02:16Z

Test report by @akesandgren

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-13.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
b-cn1603.hpc2n.umu.se - Linux Ubuntu 22.04, x86_64, AMD EPYC 7313 16-Core Processor, Python 3.10.12
See https://gist.github.com/akesandgren/489e31e8a1fba469d53d1c11626ff19e for a full test report.

verdurin · 2024-03-15T13:30:52Z

Test report by @verdurin

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-13.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
easybuild-c7.novalocal - Linux CentOS Linux 7.9.2009, x86_64, Intel Xeon Processor (Skylake, IBRS), Python 3.6.8
See https://gist.github.com/verdurin/efb7b72d63400af2ecdeb27e79643620 for a full test report.

boegel

I'm working on a commit to tackle the suggested changes...

easybuild/easyblocks/g/gcc.py

…k + minor style tweaks

boegel · 2024-03-16T10:38:22Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node3136.skitty.os - Linux RHEL 8.8, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 3.6.8
See https://gist.github.com/boegel/92e55f25f8f1b5a0502fab0f66e4f4de for a full test report.

edit: this was with a copy of the easyconfig with rename_include_fixed set to True

boegel · 2024-03-16T12:31:37Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-9.3.0.eb
SUCCESS GCCcore-10.3.0.eb
SUCCESS GCCcore-11.4.0.eb
SUCCESS GCCcore-12.3.0.eb
SUCCESS GCCcore-13.2.0.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
node3515.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/d258a3d8cc5fc1083247c51c526ad803 for a full test report.

boegel · 2024-03-16T18:55:42Z

@boegelbot please test @ generoso
CORE_CNT=16
EB_ARGS="GCCcore-9.2.0.eb GCCcore-9.3.0.eb GCCcore-9.4.0.eb GCCcore-9.5.0.eb GCCcore-11.3.0.eb GCCcore-9.1.0.eb GCCcore-10.1.0.eb GCCcore-10.2.0.eb GCCcore-10.3.0.eb GCCcore-11.1.0.eb GCCcore-11.2.0.eb GCCcore-11.4.0.eb GCCcore-12.1.0.eb GCCcore-12.2.0.eb GCCcore-12.3.0.eb GCCcore-13.1.0.eb GCCcore-13.2.0.eb"

boegelbot · 2024-03-16T19:00:08Z

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=3254 EB_ARGS="GCCcore-9.2.0.eb GCCcore-9.3.0.eb GCCcore-9.4.0.eb GCCcore-9.5.0.eb GCCcore-11.3.0.eb GCCcore-9.1.0.eb GCCcore-10.1.0.eb GCCcore-10.2.0.eb GCCcore-10.3.0.eb GCCcore-11.1.0.eb GCCcore-11.2.0.eb GCCcore-11.4.0.eb GCCcore-12.1.0.eb GCCcore-12.2.0.eb GCCcore-12.3.0.eb GCCcore-13.1.0.eb GCCcore-13.2.0.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3254 --ntasks="16" ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 13119

Test results coming soon (I hope)...

- notification for comment with ID 2002088054 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegel · 2024-03-16T19:07:34Z

@boegelbot please test @ jsc-zen3
CORE_CNT=16
EB_ARGS="GCCcore-10.2.0.eb GCCcore-10.3.0.eb GCCcore-11.2.0.eb GCCcore-11.3.0.eb GCCcore-12.2.0.eb GCCcore-12.3.0.eb GCCcore-13.2.0.eb GCCcore-9.3.0.eb"

boegelbot · 2024-03-16T19:13:07Z

@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3254 EB_ARGS="GCCcore-10.2.0.eb GCCcore-10.3.0.eb GCCcore-11.2.0.eb GCCcore-11.3.0.eb GCCcore-12.2.0.eb GCCcore-12.3.0.eb GCCcore-13.2.0.eb GCCcore-9.3.0.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3254 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 3786

Test results coming soon (I hope)...

- notification for comment with ID 2002092305 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-03-16T23:57:58Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-10.2.0.eb
SUCCESS GCCcore-10.3.0.eb
SUCCESS GCCcore-11.2.0.eb
SUCCESS GCCcore-11.3.0.eb
SUCCESS GCCcore-12.2.0.eb
SUCCESS GCCcore-12.3.0.eb
SUCCESS GCCcore-13.2.0.eb
SUCCESS GCCcore-9.3.0.eb

Build succeeded for 8 out of 8 (8 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.3, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/657d1a8b3338f0f253f747a6db1325c7 for a full test report.

Micket · 2024-03-17T01:15:47Z

Test report by @Micket

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-12.3.0.eb
SUCCESS GCCcore-13.2.0.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/1f4bab6bd4e24aedf8059a529822c5fb for a full test report.

boegelbot · 2024-03-17T06:32:21Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-9.2.0.eb
SUCCESS GCCcore-9.3.0.eb
SUCCESS GCCcore-9.4.0.eb
SUCCESS GCCcore-9.5.0.eb
SUCCESS GCCcore-11.3.0.eb
SUCCESS GCCcore-9.1.0.eb
SUCCESS GCCcore-10.1.0.eb
SUCCESS GCCcore-10.2.0.eb
SUCCESS GCCcore-10.3.0.eb
SUCCESS GCCcore-11.1.0.eb
SUCCESS GCCcore-11.2.0.eb
SUCCESS GCCcore-11.4.0.eb
SUCCESS GCCcore-12.1.0.eb
SUCCESS GCCcore-12.2.0.eb
SUCCESS GCCcore-12.3.0.eb
SUCCESS GCCcore-13.1.0.eb
SUCCESS GCCcore-13.2.0.eb

Build succeeded for 17 out of 17 (17 easyconfigs in total)
cnx1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/0f5d53608689f6234f55b5c656d115c8 for a full test report.

Don't rename GCCs include-fixed directory by default and add script t…

3de8516

…o regenerate it

Flamefire mentioned this pull request Mar 13, 2024

{lib}[GCCcore/13.2.0] CUDA v12.4.0, NCCL v2.20.5, magma v2.7.2, ... w/ CUDA 12.4.0 easybuilders/easybuild-easyconfigs#20107

Merged

boegel added enhancement bug fix change labels Mar 13, 2024

boegel modified the milestones: 4.x, release after 4.9.0 Mar 13, 2024

Micket previously approved these changes Mar 13, 2024

View reviewed changes

add check for validity of easyconfig parameter names

a9be5e0

boegel dismissed Micket’s stale review via a9be5e0 March 16, 2024 08:45

boegel requested changes Mar 16, 2024

View reviewed changes

boegel force-pushed the 20240313103229_new_pr_gcc branch from b74f41f to b8cd764 Compare March 16, 2024 09:49

fix name of rename_include_fixed easyconfig parameter in GCC easybloc…

be1afb1

…k + minor style tweaks

boegel force-pushed the 20240313103229_new_pr_gcc branch from b8cd764 to be1afb1 Compare March 16, 2024 09:50

boegel changed the title ~~Don't rename GCCs include-fixed directory by default and add script to regenerate it~~ do not rename GCC's include-fixed directory by default + add script to regenerate it Mar 16, 2024

boegel approved these changes Mar 16, 2024

View reviewed changes

boegel merged commit e65a6d1 into easybuilders:develop Mar 17, 2024
47 checks passed

Flamefire deleted the 20240313103229_new_pr_gcc branch March 17, 2024 11:14

Flamefire mentioned this pull request Mar 19, 2024

{perf}[gompi/2023b] Score-P v8.4 w/ CUDA 12.4.0 easybuilders/easybuild-easyconfigs#20146

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

do not rename GCC's include-fixed directory by default + add script to regenerate it #3254

do not rename GCC's include-fixed directory by default + add script to regenerate it #3254

Flamefire commented Mar 13, 2024 •

edited by boegel

Micket left a comment

Micket commented Mar 13, 2024

Flamefire commented Mar 14, 2024 •

edited

Micket commented Mar 15, 2024

Micket commented Mar 15, 2024

migueldiascosta commented Mar 15, 2024

akesandgren commented Mar 15, 2024

verdurin commented Mar 15, 2024

boegel left a comment

boegel commented Mar 16, 2024 •

edited

boegel commented Mar 16, 2024

boegel commented Mar 16, 2024

boegelbot commented Mar 16, 2024

boegel commented Mar 16, 2024

boegelbot commented Mar 16, 2024

boegelbot commented Mar 16, 2024

Micket commented Mar 17, 2024

boegelbot commented Mar 17, 2024

do not rename GCC's include-fixed directory by default + add script to regenerate it #3254

do not rename GCC's include-fixed directory by default + add script to regenerate it #3254

Conversation

Flamefire commented Mar 13, 2024 • edited by boegel

Micket left a comment

Choose a reason for hiding this comment

Micket commented Mar 13, 2024

Flamefire commented Mar 14, 2024 • edited

Micket commented Mar 15, 2024

Overview of tested easyconfigs (in order)

Micket commented Mar 15, 2024

Overview of tested easyconfigs (in order)

migueldiascosta commented Mar 15, 2024

Overview of tested easyconfigs (in order)

akesandgren commented Mar 15, 2024

Overview of tested easyconfigs (in order)

verdurin commented Mar 15, 2024

Overview of tested easyconfigs (in order)

boegel left a comment

Choose a reason for hiding this comment

boegel commented Mar 16, 2024 • edited

Overview of tested easyconfigs (in order)

boegel commented Mar 16, 2024

Overview of tested easyconfigs (in order)

boegel commented Mar 16, 2024

boegelbot commented Mar 16, 2024

boegel commented Mar 16, 2024

boegelbot commented Mar 16, 2024

boegelbot commented Mar 16, 2024

Overview of tested easyconfigs (in order)

Micket commented Mar 17, 2024

Overview of tested easyconfigs (in order)

boegelbot commented Mar 17, 2024

Overview of tested easyconfigs (in order)

Flamefire commented Mar 13, 2024 •

edited by boegel

Flamefire commented Mar 14, 2024 •

edited

boegel commented Mar 16, 2024 •

edited