runtime depends on extremely recent verisons of cmake, blocking adoption on Linux

[omajid]

Description

In order to help make .NET ubiquitous, I am working on packaging and including .NET (Core) into as many Linux distributions as I can. Some of these Linux distributions are included in the official support matrix of .NET:

https://github.com/dotnet/core/blob/master/release-notes/5.0/5.0-supported-os.md

May of the Linux distributions listed in the matrix above include much older versions of cmake than what dotnet/runtime requires to build. For example:

• CentOS 7: 2.8.12.2
• CentOS 8: 3.11.4
• Debian 9: 3.7.2
• Debian 10: 3.13.4
• OpenSuSE Leap 15.0: 3.10.2
• RHEL 7: 2.8.12.2
• RHEL 8: 3.11.4
• Ubuntu 18.04: 3.10.2
• Ubuntu 19.10: 3.13.4

Because of this, it becomes almost impossible to build or package .NET in those distributions :( That prevents us from easily including the upcoming .NET 5 in these distributions.

To work around this, we would have to revert a long series of non-trivial commits: like dotnet/coreclr#26980

If it matters, for packaging on Linux, we are consuming dotnet/runtime indirectly via dotnet/source-build.

Configuration

Here's a dockerfile that reproduces the problem:

FROM docker.io/centos:8

RUN dnf install --setopt tsflags=nodocs --refresh -y \
        dnf-plugins-core &&  \
    dnf config-manager --set-enabled PowerTools  && \
    dnf install --setopt tsflags=nodocs --refresh -y \
        automake \
        clang \
        cmake \
        curl-devel \
        findutils \
        git \
        glibc-langpack-en \
        hostname \
        krb5-devel \
        libicu-devel \
        libtool \
        lldb-devel \
        llvm \
        lttng-ust-devel \
        make \
        openssl-devel \
        python3 \
        tar \
        wget \
        which \
        zlib-devel && \
    dnf upgrade -y && \
    dnf clean all -y

CMD git clone https://github.com/dotnet/runtime && \
        cd runtime && \
        git submodule update --init --recursive && \
        ./build.sh clr

$ podman build -t d -f Dockerfile .
$ podman run -it d
Cloning into 'runtime'...                                                                                                                                                                                                                     
remote: Enumerating objects: 32, done.                                                                                                                                                                                                        
...
  Setting up directories for build                                                                                                                                                                                                            
  Checking prerequisites...                                                                                                                                                                                                                   
  Please install cmake v3.14.5 or newer from https://www.cmake.org/download/.                                                                                                                                                                 
/runtime/src/coreclr/runtime.proj(32,5): error MSB3073: The command ""/runtime/src/coreclr/build-runtime.sh" -x64 -debug -os Linux" exited with code 1.
....
    0 Warning(s)
    13 Error(s)

Time Elapsed 00:03:53.27
Build failed (exit code '1').

Regression?

This is a build-time regression.

The situation was not so bad in the 3.1 timeframe where the minimum version required was much lower.

The situation seems to have gotten much worse with .NET 5: RHEL 8, Debian 9 and Ubuntu 18.04 all have incompatible versions of cmake.

Other information

• Do you know of any workarounds?

One possible option would be to upgrade or package up alternate versions of cmake in those Linux distributions. That becomes more challenging the older the Linux distribution we are looking at is. It's harder both technically (compatibility impact, more recent versions of cmake require a more recent version of other dependencies) and process-wise (how do you convince others that adding new packages to a X year old distribution is a good idea now?)

Some distributions, like RHEL 7 use "alternate" packaging mechanisms (such as software collections) that allow using a custom cmake for building .NET 5, but have a significant usability impact for the end-user.

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[omajid]

cc @tmds @dleeapho

[ghost]

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

[jkotas]

cc @jkoritzinsky

[stephentoub]

This is a much more minor concern than the one cited, but this also makes it harder to get started contributing to dotnet/runtime.

[tmds]

For building on RHEL 7 (and CentOS 7), we have a version 3.6.2 available. This is not the default cmake, but separate build maintained by the llvm team.

That would be our desired minimum version, and support all distros mentioned in the first comment.

[tmds]

@jkoritzinsky @jkotas can we target 3.6.2?

[jkotas]

It would required undoing dotnet/coreclr#26980 and all follow up changes.

[jkoritzinsky]

There’s significant dependency now on CMake generator expressions introduced in newer versions of CMake as well as more recent commands that accept generator expressions to support our DAC/Crossgen builds. It’s possible to refactor the builds to not use these commands, but it will take quite a bit of work to lower our minimum to 3.6.2.

[janvorli]

It would also require reworking the change #36847 that relies on an ability to use object libraries in target_link_libraries. I believe that support for that came in cmake 3.12.

[omajid]

It would also require reworking the change #36847 that relies on an ability to use object libraries in target_link_libraries. I believe that support for that came in cmake 3.12.

I can confirm: runtime's use of target_link_libraries is generating errors on 3.11 (RHEL 8).

[omajid]

it will take quite a bit of work to lower our minimum to 3.6.2.

I am happy to help with the work. I don't think I could do it alone, though: I don't have access to Windows platforms for development.

[wfurt]

I'm using cmake's packages on Ubuntu without any problems. I think that covers Debian as well. That pretty much leaves only redhat platforms. I'm wondering if it would make more sense to push forward and provide newer cmake/cmake3 on those distributions. It seems like we will need to spent non trivial effort to go back. And work forward may benefit other projects as well.

[stephentoub]

I'm using cmake's packages on Ubuntu without any problems

I'm using Ubuntu 18.04.4 LTS on WSL2. The cmake version is 3.10.2, and there's nothing newer I could find in the package manager (cmake is already the newest version (3.10.2-1ubuntu2.18.04.1)). I had to download cmake from the website and install it manually to be able to dev in dotnet/runtime.

[wfurt]

You need to add new repository. https://apt.kitware.com (probably can skip first step)
With that, everything should work fine with package manager.

[jkoritzinsky]

Kitware provides an APT feed with the most recent version of CMake for Ubuntu at https://apt.kitware.com. We link to it in the getting started docs.

[wfurt]

BTW on Centos7 I use cmake3-3.14.7-1.el7.x86_64. There is still some trickery which can be improved as the binary is cmake3 and not cmake as current build expects. #35593 did some minimal fixes.

[tmds]

To ship .NET as part of a distro, no 3rd party binaries may be involved in the building. Everything must come from the distro. This is to ensure the final result is completely open and reproducible. When 3rd party binaries are used in the build, packages are not allowed in the default repositories.

To support different distros, that means lowering the version.

[jkoritzinsky]

For the older distributions, would it be possible to build the newer version of cmake from source based on tools available in the distro and use that to build dotnet/runtime?

[tmds]

There is a trade-off between having cmake features, and supporting different distros.
With the current required minimum of 3.14, almost no distributions are supported.

.NET package maintainers could maintain another set of cmake packages.
The preferred way of working is to be compatible with what the system has.
Having to build cmake to build .NET` should be the exception.

[wfurt]

While I hate to upgrade frequently, I think packages from Kitware should be sufficient for developers and contributors.
I could make it work without much troubles and I really don't care about the source. .NET is distributed in similar way.
If this is really about official build, can this be solved with source-build @tmds?

[tmds]

If this is really about official build, can this be solved with source-build @tmds?

This is pushing the issue outward, from runtime, to source-build, and finally package maintainers.
It makes sense to push some thing outward, up to some extent.

imo, we should aim to be compatible at runtime with a large set of distros, and avoid pushing work downstream (especially package mantainers).

I think packages from Kitware should be sufficient for developers and contributors.

This works. Personally, I like using 'open' packages as much as possible.

[danmoseley]

Cc @dleeapho

[RheaAyase]

You need to add new repository. https://apt.kitware.com (probably can skip first step)
With that, everything should work fine with package manager.

For the older distributions, would it be possible to build the newer version of cmake from source based on tools available in the distro and use that to build dotnet/runtime?

etc

I think that there is missing communication here somewhere.

This is not about older distributions, this is about enterprise distributions (and a few others that are not as cutting edge as Fedora.)

Further, in the Open Source world we can not use 3rd party packages (or 3rd party binaries of any kind,) so "kitware" or other solutions are absolutely out of the question. Building cmake from source just to build dotnet also sounds very dismissive of the problem at hand. We're already doing something like this with other things in RHEL and it's not maintainable to keep piling up one on top of another.

---

For the sake of a bit of retrospective / exploring what we can do to also prevent these things in the future - the source of this issue is that these changes were not discussed with the wider community, Red Hat or other Linux representatives were not consulted. We would have pointed it out that it's a bad idea before any commits have happened. All the dependencies and especially glibc, cmake, etc... these need to be considered carefully and with more points of view than just one build pipeline (which isn't open to others.)

While dotnet is a Microsoft product, it's also an open source one now. Official builds are not just the ones that Microsoft produces. Microsoft itself supports builds done in RHEL (CentOS) and other distributions. This problem means that these supported builds ... can not be built. And now, now we have major problems getting the next version of dotnet included in the next version of RHEL which will manifest either in crazy crunch times, or missed deadlines. And missed deadlines for products with exceptions to miss deadlines.

[omajid]

I would like to provide some additional context that might help clarify things.

The ultimate goal of the source-build effort is to package .NET into as many Linux distributions as possible. If/when we are successful, .NET will be available out of the box in those distributions, just like may other languages like C, Go, Java, Perl, Python, and Ruby.

To be included in distributions, generally packages are only allowed to use what's already in the distribution. If a distribution doesn't have cmake at all, for example, there's no way we could package .NET in that distribution. We would first have to add cmake and then try and package .NET in that distribution.

The list of distributions I described above all already have an older version of cmake. So we can't fix the problem by adding cmake. We would have to upgrade cmake. That has an unknown (at least to me) compatibility impact. I expect that upgrading the version of cmake in, for example, RHEL 8 or Debian 9 is going to be quite a challenge. We would have to show that all programs that worked with cmake 3.x.y continue to build the same with cmake 3.14.5 (or newer). There's a very high bar for making what could be a breaking change to one of the fundamental build tools being used in a distribution. And we have to meet this bar for all the distributions we want to add .NET to.

There's also the question of timing. Even if an upgrade was accepted (I dont think it will be in many such enterprise or stable distributions), a change can take a long, long time. If it takes 1 year from now for Debian 9 to get the new cmake version in the main package repositories, it will be too late for use in .NET 5.

Perhaps you might understand the impact of this issue more broadly if you mentally replaced cmake with glibc in this discussion. We don't ask users to install a separate glibc. We don't ask distributions to upgrade their version of glibc. We look at our support matrix and try and support all the glibc versions available there.

@wfurt said:

I'm using cmake's packages on Ubuntu without any problems. I think that covers Debian as well.

I hope it now makes sense why it doesn't even work for the versions of Debian and Ubuntu I listed in my initial comment above. Those versions of Debian and Ubuntu do not have the required version of cmake in the main package repositories. Using an external package repository is not acceptable for Debian.

I'm wondering if it would make more sense to push forward and provide newer cmake/cmake3 on those distributions.

There's a very high compatibility bar that needs to be crossed. And it needs to be met for every distribution that we want to package .NET in. I know .NET cares mostly about run-time compatibility. But Linux distributions generally care about as much for build-time compatibility. Updating cmake is far from a trivial operation.

It seems like we will need to spent non trivial effort to go back.

Agreed. I can help with this.

@wfurt said:

You need to add new repository. https://apt.kitware.com (probably can skip first step) With that, everything should work fine with package manager.

@jkoritzinsky said:

Kitware provides an APT feed with the most recent version of CMake for Ubuntu at https://apt.kitware.com. We link to it in the getting started docs.

That's true, but it's also non-usable in a distribution context. Most distributions (certainly the case for enterprise distributions like RHEL) wont break the trust model by using binaries compiled by someone else.

For the older distributions, would it be possible to build the newer version of cmake from source based on tools available in the distro and use that to build dotnet/runtime?

A general upgrade for every distribution might be next to impossible. Doing this as part of .NET Core build might something we could do as a last resort. That might still violate some distribution policies. For example last time I tried to use a new version of cmake in RHEL, I had to bootstrap it using a prebuilt cmake binary. This would not work for Fedora (but Fedora is not in the list of distributions that has and old version of cmake).

@wfurt said:

While I hate to upgrade frequently, I think packages from Kitware should be sufficient for developers and contributors.

That's true for developers and contributors, but not builders and Linux distributions. We can't use binaries produced by a third-party (except in very specific circumstances, like bootstrapping).

I could make it work without much troubles and I really don't care about the source. .NET is distributed in similar way.

Community linux distributions that focus on Openness and Freedom want to be able to build everything from source and only ship things they have themselves built from source. That lets them be sure those users who do care about the source are satisifed.

Enterprise linux distributions (as well as various *BSDs, I think) care about sources because it lets them avoid risk; if an open source project can't fix an major issue, then they can step in and fix things themselves.

Both models mean that being able to build everything from source is important. That's why there's a source-build project in the first place.

If this is really about official build, can this be solved with source-build @tmds?

I am trying to think what a solution that lives only on source-build-side would look like. Source-build will have to, some how, support older versions of cmake and build runtime based on that.

One way to do that would be to build a newer cmake version as part of source-build. I touched on this earlier in my reply: there may be issues with this.

Another way source-build could support older versions of cmake would be to carry patches locally to remove the dependency on newer cmake version. That would be, effectively, doing the same work that everyone wants to avoid here. Worse, it would be specific to source-build. It would be likely that some build bugs and issues will exist only in source-build. I would much rather do the changes here, in runtime, where everyone can collaborate and share issues and fixes.

[jkoritzinsky]

Has something changed since the conversation we had in dotnet/dotnet-buildtools-prereqs-docker#185 about using a newer CMake version?

[omajid]

Has something changed since the conversation we had in dotnet/dotnet-buildtools-prereqs-docker#185 about using a newer CMake version?

It's my mistake I didn't look into all the default versions back then.

I thought the conversation was essentially about RHEL 7/CentOS 7 going from 3.14.x to 3.15.x.

3.1.5.x didn't seem any worse than 3.14.x in terms of distribution inclusion. RHEL 7, as you can see from the matrix above, only has 2.x I figured that the version of 3.x wouldn't matter.

I didn't realize that the matrix was affecting RHEL 8 as well as many, many, many other distributions as well.

[janvorli]

Thank you @omajid and @RheaAyase for providing all the details. It seems to me that it is inevitable to move back to using an older cmake.

[jkoritzinsky]

We should be able to preserve most of the new design we have if we use functions to add the right defines. It'll just be tedious to convert all of the add_compile_definitions and add_link_options to target_compile_definitions and target_link_libraries that are attached to specific targets.

[jashook]

Based on the feedback here we will change our supported Cmake version from Linux to 3.6. @jkoritzinsky has started this work with #39044. In order to avoid regressing the build using older version of Cmake we will add in a CI job which specifically builds with 3.6 on CentOS.

For non-source build builds we will continue to use higher versions of CMake in the same way we use clang9 now.

/cc @dotnet/runtime-infrastructure @jkoritzinsky @omajid @jkotas @janvorli

Please let us know if there is anything else required.

[jashook]

Note that this will not force us to rebuild all of the dotnet-build-prereqs images, and will not require people who already have 3.14 installed to remove and install a different version.

[am11]

in the Open Source world we can not use 3rd party packages (or 3rd party binaries of any kind,) so "kitware" or other solutions are absolutely out of the question.

Most distributions (certainly the case for enterprise distributions like RHEL) wont break the trust model by using binaries compiled by someone else.

I agree that we should relax the cmake version requirement, if it is feasible for repo maintainers. (we would still need to special case the check in libraries for iOS build, which requires 3.14.5)

However, I am still trying to understand the overarching concern. There are also other prebuilt binaries downloaded during the build of runtime in the realm of "build tools". For example, AFAIK, when dotnet-install.sh script runs during the coreclr/libraries build, there is no way to skip downloading the pre-built dotnet(1) binary from blob storage to the effect: use my local dotnet(1) or download dotnet(1) from my company's trusted server. Given that, comparatively how bad is it to acquire the portable cmake binary from Kitware's official server with these two lines:

# works on all glibc+musl-libc based linux distros
curl -SLO https://github.com/Kitware/CMake/releases/download/v3.16.5/cmake-3.16.5-Linux-x86_64.sh
sudo sh cmake-3.16.5-Linux-x86_64.sh --skip-license --prefix=runtime/.tools
# runtime/build.sh

Note that in case of windows, cmake.exe is downloaded as part of the build.cmd invocation from blob storage, see eng/common/native/install-tool.ps1, so in theory as an alternative solution to this problem, Linux portable binaries could also be placed next to it in the same blob storage and use similar tool acquisition mechanism as done in windows build?

[jkotas]

@am11 Source build that this discussion is about has these goals: https://github.com/dotnet/source-build#goals . Source build is not using the default build flow to achieve these goals.

comparatively how bad is it to acquire the portable cmake binary from Kitware's official server with these two lines:

If we have done that, we would also need to acquire the cmake sources during source build and build cmake from these sources. It is doable in theory, but quite expensive. We believe that simply lowering the min cmake version is cheaper overall.

[omajid]

@am11 said:

However, I am still trying to understand the overarching concern. There are also other prebuilt binaries downloaded during the build of runtime in the realm of "build tools".

This has been solved in the source-build project. In Fedora, we are using source-build to build .NET Core 3.1 in a way that absolutely nothing is downloaded. You can see our completely off-line builds here: https://koji.fedoraproject.org/koji/packageinfo?packageID=30766. Like any other distro-package, only things that already exist in the distro are available to these builds.

For example, AFAIK, when dotnet-install.sh script runs during the coreclr/libraries build, there is no way to skip downloading the pre-built dotnet(1) binary from blob storage to the effect: use my local dotnet(1) or download dotnet(1) from my company's trusted server.

The main source-build build script supports using a user-provided SDK to build: https://github.com/dotnet/source-build/blob/1eba1a673a160ae4635270fdcf823da4bfe66d3d/build.sh#L12

The build-from-source-tarball script also supports the same flag: https://github.com/dotnet/source-build/blob/1eba1a673a160ae4635270fdcf823da4bfe66d3d/support/tarball/build.sh#L10

I am not quite 100% sure how this works, but you should be able to follow the code to see how it gets coreclr to build using a user-supplied SDK.

[akoeplinger]

For the older distributions, would it be possible to build the newer version of cmake from source based on tools available in the distro and use that to build dotnet/runtime?

A general upgrade for every distribution might be next to impossible. Doing this as part of .NET Core build might something we could do as a last resort. That might still violate some distribution policies. For example last time I tried to use a new version of cmake in RHEL, I had to bootstrap it using a prebuilt cmake binary. This would not work for Fedora (but Fedora is not in the list of distributions that has and old version of cmake).

@omajid according to CMake you can bootstrap just from the tarball: https://cmake.org/install/

I tried this in a minimal CentOS 7 Docker container using these steps and was able to bootstrap without any downloaded binaries other than the tarball from https://github.com/Kitware/CMake/releases/download/v3.17.3/cmake-3.17.3.tar.gz (I unplugged network after installing gcc/make):

$ yum install gcc gcc-c++ make openssl-devel
$ tar -xvf cmake-3.17.3.tar.gz
$ cd cmake-3.17.3
$ ./bootstrap
$ make
...
$ export PATH=$PWD/bin:$PATH

$ cmake -version
cmake version 3.17.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).

It doesn't sound too crazy to me to do this as part of source-build rather than tying us to ancient versions of cmake in dotnet/runtime.

As @am11 mentioned we need newer versions for iOS/Android/WebAssembly as older versions had bugs, so we need to at least check for newer cmake when targetting those platforms, but it would be nice not needing to.

[dleeapho]

It doesn't sound too crazy to me to do this as part of source-build rather than tying us to ancient versions of cmake in dotnet/runtime.

Source-build and source-build bootstrapping is already quite complex and costly adding cmake bootstrap adds to that complexity significantly. That additional cost would be a significant detractor for maintainers wanting to bring in .NET into their distro.

We also are in the midst of a major effort to make source-build more maintainable dotnet/source-build#1510 This effort will go a long way in surfacing difficult issues like this one earlier. Indeed, that source-build effort makes any additional effort to bootstrap cmake prohibitive.
cc @dseefeld

[ViktorHofer]

@jkoritzinsky should this issue be closed now?

[jkoritzinsky]

Yes I think we can close this.