More information on CDT (Core Dependency Tree) packages?

[mithro]

Sorry if this is the wrong repository to log this issue, feel free to move to the right location.

The Host section of Defining metadata (meta.yaml) page talks about a thing called CDT (which apparently stands for Core Dependency Tree). I however can't seem to find any more information about CDT after a bunch of Googling. This leaves me with a couple of questions;

• Is CDT a conda thing? The docs seem to imply it might be a more general term used outside of conda?
• How does one figure out what is provided in the CDT and how are they are named? The docs mention;

  The Anaconda Distribution chose not to provide X11 or GL packages, so we use CDT packages for X11. Conda-forge chose to provide X11 and GL packages.

• Are the CDT dependencies automatically added to a packages requirements: output? The example seems to imply you have to add them yourself?

I have a couple of packages which need to link against libusb and it seems. In older versions of conda, we got this from the system host (and it worked mostly by accident). It seems like this should be explicitly set as a requirement by a CDT definition in newer conda builds? I however didn't know what to put in my requirements section. I was guessing something like {{ cdt('libusb-devel') }} or something?

[mingwandroid]

You are not forced to use our compilers but I would encourage it if you want to achieve the optimum level of compatibility with AD packages.

All that 'compiler' does is to use ours by default if it finds no compiler package(s) identified in conda_build_config.yaml. The keys to change are: c_compiler, cxx_compiler and fortran_compiler, be aware that the conda subdir (osx-64, linux-32 etc) get prepended to these package names. You can create empty packages here if you wish and just use gcc or g++ in your recipe.

Regarding CDT packages, I agree we should document them properly, but as a brief explanation, AD promises compatibility with CentOS6 (and some compatibility with SLES11). We use repackaged RPMs from CentOS6 moved into the sysroot of our pseudo-cross compilers.

To see which packages use CDTs you can recursively clone https://github.com/AnacondaRecipes/aggregate.git and search for cdt in the yaml files (I'd estimate about 5-10% of our packages use them directly, basically the X11 stack).

If achieving compatibility with AD (and CentOS6) is something you wish to achieve then you have two choices when it comes to libusb:

1. Run conda skeleton cran --distro cos6 --recursive libusb-devel --architecture x86_64 and conda skeleton cran --distro cos6 --recursive libusb-devel --architecture i686 then build that package and list it as a requirements/build in your package's recipe's requirements/build section so they get put in the same env. as the compilers.
2. Create a conda recipe for libusb, and add that to requirements/host though for things that touch device hardware I'm always a bit wary, still try it, if it works then great. I found this recipe which has been updated to the new compiler specification syntax, maybe it could be of use? https://github.com/ryanvolz/libusb-feedstock

[mingwandroid]

CDT packages are a conda thing yes. It gives enough isolation that we do not need to force people to use a specific linux distro (or docker or chroots or other isolation - though they are free to combine our isolation with those systems for even better isolation!), however if a build system explicitly adds /usr/lib to the linker search path then it may find your system packages. I am considering adding something whereby the toolchain binaries check an env. var and ignore /usr/include and '/usr/lib*`, but ..

.. to help avoid this recent conda-build 3 releases raise an error when any system libraries are detected as having been dynamically linked. You can disable that error by adding --no-error-overlinking to the conda-build command-line, but only do that if you plan to use system compilers and accept the risks therein.

.. and I think that the isolation and error are probably good enough (this --error-overlinking option is really good for keeping your dependencies correct).

If you ever need to do 'binary repackaging' as we do when we create these CDT packages, you can use build/missing_dso_whitelist to prevent --error-overlinking from complaining about those DSOs showing up (or just pass --no-error-overlinking when you build them).

[mithro]

Thanks for that info! It's super helpful.

I'm definitely trying to move my packages to be "more compatible" which I believe means moving towards your new compilers + CDT system. QQ What does AD stand for? Anaconda Distro?

FYI It might be worth just dumping this explanation above in CDT section?

(Just a FYI, I'm in the process of updating our packaging of bare metal cross-compilers and FPGA tools for our Python based hardware description language project. See https://github.com/timvideos/litex-buildenv -- we are using conda to provide a self contained SDK style environment.)

[mingwandroid]

I abbreviate Anaconda Distribution to AD, yeah. Saves my fingers a bit.

Just a FYI, I'm in the process of updating our packaging of bare metal cross-compilers and FPGA tools for our Python based hardware description language project. See https://github.com/timvideos/litex-buildenv -- we are using conda to provide a self contained SDK style environment.)

Yes, I remember our discussion about static compilers. I think the issues on our end are all fixed for that now, or are you still unable to build them fully statically?

[mithro]

We were able to statically link the compilers but then ran into ContinuumIO/anaconda-issues#8203 (Due to glibc 2.12 limitation, static executables that use time(), cpuinfo() and maybe a few others cannot be run on systems that do not support or use vsyscall=emulate #8203). Sadly, back in March I wasn't able to find the time to finish the vsyscall removal stuff and then ran out of free time.

I'm now trying to test if using the AD compilers means we don't need to statically link and can just bypass the above issue.

I'm also hoping to port our cross compilers to be based on your packages which used crosstool-ng. (I'm also hoping that it will make it easier for us to provide Mac + Windows packages too.)

[mingwandroid]

I'd love to have the time to build out a matrix of cross-compilers so you can build conda-packages for Linux, macOS and Windows (using clang-cl backend with either the clang-gcc or the clang-cl frontend) on any of the other OSes. Testing would need careful consideration, as would make check in build.sh, but I was able to use QEMU, binfmt-misc and dejagnu to cross compile packages from glibc-based x86_64 Linux to a uclibc-ng based arm6 target. I built enough to run miniconda and it worked fine.

I also ported macOS and iPhone compilers to run on Linux and Windows a good few years ago. That's how I got into crosstool-ng stuff actually, adding llvm+clang to it, but that work has long since rotted.

[mithro]

From a building packages point of view, I only really care about building on Linux (IE Travis actually :-). For people using the packages, I would like to support as many platforms as possible but have historically restricted it to Linux due to not having enough time.

If you ever want some FPGA hardware to play with, I would be more than happy to send you some to play with! However, I'm guessing you already have way to many time commitments.

[mingwandroid]

Yeah, I have far more tasks to do than time to do them, as always, but many thanks for the offer!

[mithro]

I noticed your crosstool-ng-feedstock recently changed to using the repository at https://github.com/continuumio/crosstool-ng.git -- Should I be basing my ct-ng patches on top of that repo?

[mingwandroid]

If you have changes you think are more appropriate for crosstool-ng upstream then make a PR there first, otherwise, having your patches that you would like to submit upstream come before our patches will make it easier for if / when you do submit them upstream (apologies if you know this workflow already, I find it helps with submission and rebasing to sort my patches in order of my perception of their upstream-a-bility, but that only really matters if I get round to attempting to upstream).

Yes, we moved it to be more official and to standardise on the master branch and also forked it directly from the upstream project instead of using the diorcety fork.

If your patches are too Anaconda-specific then it's fine to submit a PR to us. If it's in support of esoteric platforms that you need to support then so long as they don't break intel linux then I personally don't mind carrying them, esp. if it opens AD up to other platforms.

We'd not build these esoteric compiler binaries though unless there was a strong demand (which I guess esoteric excludes), we'd leave that up to you instead.

[mingwandroid]

Oh, one important consideration when building the compilers is to do it on CentOS6 using the system compilers. If you don't do this then the compilers may not run on other distros (or even compile, I've only rarely built them on any other distro and just during development).

This is the only package that needs to be built on CentOS6 though.

[mithro]

What is the status of compiling crosstool-ng compilers using the anaconda compilers? I see comments about "fixme bootstrap" :-)

[mingwandroid]

What is the status of compiling crosstool-ng compilers using the anaconda compilers? I see comments about "fixme bootstrap" :-)

It might work fine, comments have probably rotted, but use the crosstool-ng-boostrap recipe on CentOS6 (the other one is for building it using our own packages and compilers - but just for the crosstool-ng config binaries themselves which is pretty irrelevant in the grand scheme of things).

Our compilers are bootstrapped though internally since crosstool-ng handles that. The final gcc is built using the stage-2 gcc.

So build crosstool-ng-boostrap then once you've got that, build ctng-compilers-feedstock and then finally ctng-compilers-activation-feedstock (or ctng-dbg-compilers-activation-feedstock .. then to use those, add the following keys to conda_build_config.yaml):

cxx_compiler:
  - gxx-dbg                   # [linux]
c_compiler:
  - gcc-dbg                   # [linux]
fortran_compiler:
  - gfortran-dbg

If you run into trouble building them the give me a shout.

[mingwandroid]

The system packages you need to install on CentOS6 are:

yum install -y patch which file gperf help2man gawk texinfo m4 wget tar bison flex gcc gcc-c++ ncurses-devel git autoconf perl-Thread-Queue

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!