cos7 CDTs and new compilers

[beckermr]

As part of our effort to add support for cos7 to linux-64, we'd like to do some refactoring of the structure of our CDTs. We (@isuruf and myself) are looking for community help in this work.

To do:

1. make sure old CDTs are mutually exclusive with the new ones

• patch the repo data to make all current CDTs conflict with the sysroot packages (ENH make old-style CDTs conflict with new sysroot conda-forge-repodata-patches-feedstock#60)
• patch the CDT repo data on defaults so they conflict with the sysroot package (ENH make CDTs conflict with sysroot packages from conda-forge AnacondaRecipes/repodata-hotfixes#94)
• patch repodata so that new sysroots don't interact with old compilers (ENH patch latest compilers for new sysroots as well conda-forge-repodata-patches-feedstock#66, ENH make sure old compilers don't use things with the new sysroot conda-forge-repodata-patches-feedstock#65)

2. build CDTS

• setup one-off CDT builds on conda-forge (ENH CI setup and an example cdt-builds#2)
• build all of the old-style CDTs
• build all of the new-style ones too
• make all CDTs use no_hoist: true in the source sections
• new-style CDTs should use the new conda sysroot (i.e., {{ arch }}-conda-linux-gnu) sysroot
• new-style CDTs should depend on the correct version of the sysroot package (i.e., cos6 depends on 2.12 and cos7 depends on 2.17)
• old-style CDTs need to have the proper run_constrained
• see https://hackmd.io/T5q0GPa1QfKfQ6irNWAe5Q?edit for notes from the CDT discussion

3. move to only conda-forge CDTs

• enable strict channel priority in the ci setup scripts (to make sure we always use our CDTs, ENH strict channel priority for linux conda-forge-ci-setup-feedstock#110)

4. turn on new compilers

• finally remove the symlinks and CDT-location variants in the new compilers (ENH single sysroot 2.17 linux-sysroot-feedstock#26, ENH single sysroot 2.12 linux-sysroot-feedstock#27)
• make sure to add a sysroot run dep to gcc_impl, gxx_impl and gfortran_impl so that the repo data patches above don't break the new compilers (ENH add extra sysroot deps to mark builds ctng-compilers-feedstock#32)

cc @isuruf @jakirkham @kkraus14 @nehaljwani @mingwandroid @jjhelmus

xref: #900
closes conda-forge/ctng-compilers-feedstock#28

[beckermr]

cc @brianv0 too

[kkraus14]

cc @mike-wendt

[mingwandroid]

I need to rebuild all the COS6 CDTs (I did this a while back, those are on anaconda.org/rdonnelly) due to a bug (missing feature kinda really) in the construction of the old ones (to do with conda-build not having any way to be told not to 'hoist' the top level dir from archives it unpacks).

This is fixed now. Please make sure your CDT meta.yamls have:

source:
  - url: http://mirror.centos.org/centos/6.10/os/x86_64/Packages/mesa-libEGL-11.0.7-4.el6.x86_64.rpm
    sha256: 1b5638595636d26590e1a2140c9d64a452b63d985a507197206d0e85b7796753
    no_hoist: true

[beckermr]

thanks @mingwandroid!

[beckermr]

Hi all. We need a plan to break this work up and get it done. Can you all join the next core meeting? We’ll have an extra one next week at the usual time.

[beckermr]

I've made a few PRs with a pretty hacky script.

conda-forge/staged-recipes#12014
conda-forge/staged-recipes#12015
conda-forge/staged-recipes#12016

I'd be great if you all can look them over!

[jakirkham]

Those all appear to be closed. Is that expected or am I looking in the wrong place?

[beckermr]

I made some improvements and so closed them. Sorry!

The recipe for zip 3 on x86_64 was just merged. You can check that one out.

[isuruf]

Should we be moving everything from defaults or just ones that we actually use in conda-forge? I don't think it makes sense to move everything and increase the maintenance burden.

[jakirkham]

Either way seems fine to me. The maintenance burden here is pretty low. These are typically built once and that's it.

[jakirkham]

So what other CDTs do we still need @beckermr? Do you also have a link to the script somewhere?

[beckermr]

I will push the script later. There are ~250 cdts in defaults. Idk which ones we use

[beckermr]

Here is the CDT list: https://github.com/beckermr/misc/blob/master/work/conda_forge_cdt_scripts/cdt_list.yaml

I made this list by grabbing all CDTs in the aggregate repo from anaconda.

Here is the script: https://github.com/beckermr/misc/blob/master/work/conda_forge_cdt_scripts/make_cdt_pr.py

Pls do not make PRs with it yet or push to my fork.

[jakirkham]

Ok should we find some time to align next week? Either at the core meeting or separately?

[beckermr]

I planned to do this at the core meeting

[beckermr]

I’ve archived the zip feedstock so nobody has to maintain it.

@isuruf can you send a list of cdts we need?

[beckermr]

Here is my best guess for the CDTs we need: https://github.com/beckermr/misc/blob/master/work/conda_forge_cdt_scripts/needed_cdt_list.yaml

The script that computed them is here: https://github.com/beckermr/misc/blob/master/work/conda_forge_cdt_scripts/compute_needed_cdts.py

[beckermr]

draft of the CDT builds is here: conda-forge/cdt-builds#2

[beckermr]

I have made the last PR needed here. it may or may not be merged and is not strictly needed. closing this issue!