Improvement for ARM/Linux CI

[wateret]

We have following issues with CI/ARM/Linux. The sub-bullets are our plans.

• It is hard to make changes in the CI environment.
  • The rootfs for cross-build is from the CI machine which is not public. It would be nicer to build the rootfs during build process because it is the same way any developer would do.
  • We'd like to use Docker for cross-build. When build environment(toolchain or something) is changed, we don't need to work on cloned CI. We can make changes in Dockerfile.
• Adding x86/Linux build and test
  • This would be another good reason for using Docker.

cc: @hqueue @gkhanna79

[hqueue]

I agree with @wateret on building rootfs in the same way developers do
and also agree using docker for CI is a good idea when considering maintanence.
I think this is related to core-setup issue https://github.com/dotnet/core-setup/issues/1400 and we should considering CI and docker together if we decided to use docker for CI.

For coreclr, there are docker images used for pipilebuild as mentioned by @gkhanna79 at https://github.com/dotnet/core-setup/issues/1400#issuecomment-276823017.

• for Ubuntu 1404, try chcosta/dotnetcore:ubuntu1404_cross_prereqs_v1
• for Ubuntu 1604, try chcosta/dotnetcore:ubuntu1604_cross_prereqs_v1

@wateret What do you think of using above image for coreclr CI ? Can we start with them?

[hqueue]

@gkhanna79 I think using existing docker image can be a candidate. Furthermore I hope we can configure docker image as needed by making PR here or other repo of dotnet, for example installing new packages to introduce new feature in CI.
AFAIK to enable configuration by developers, Dockerfile for above images may need to be maintained somewhere in dotnet too.
What do you think of it? I would like to hear any options you have in mind.

[gkhanna79]

I would suggest that you should experiment using the above Docker images for bringing up CI in core-Setup since CoreCLR already has it. Once that is confirmed to work, you can attempt to revamp CorECLR and CoreFX CI.

Furthermore I hope we can configure docker image as needed by making PR here or other repo of dotne

This is an interesting question. My current understanding is that our central Docker images (which I mentioned above) are indeed maintained as DockerFiles in a repo but everytime we update them, since they affect official builds, we manual build and validate them before publishing to the above repository.

@dleeapho and @MichaelSimons are the folks who own and evolve are Docker infrastructure and can comment on how feasible it is for you to make PR to those files.

[jyoungyun]

@gkhanna79 @dleeapho @MichaelSimons Where can I see the DockerFiles for the docker images what it is mensioned above? If we can access the DockerFiles, we can use it including Core-setup builds.

[wateret]

@hqueue I agree. I'll start with the central docker images mentioned.

for Ubuntu 1404, try chcosta/dotnetcore:ubuntu1404_cross_prereqs_v1
for Ubuntu 1604, try chcosta/dotnetcore:ubuntu1604_cross_prereqs_v1

[MichaelSimons]

@jyoungyun, Unfortunately at the moment, the Dockerfiles are not in an open repo. We are actively working on moving them to the open so that everyone can contribute to them. I expect this work to be done over the next couple of weeks. In the meantime I have copied the Dockerfiles for you to reference.

chcosta/dotnetcore:ubuntu1404_cross_prereqs_v1

FROM chcosta/dotnetcore:ubuntu1404_coredeps_v1

# Install the base toolchain we need to build anything (clang, cmake, make and the like).
RUN apt-get update \
    && apt-get install -y wget \
    && echo "deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty-3.6 main" | tee /etc/apt/sources.list.d/llvm.list \
    && wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key | apt-key add - \
    && apt-get update \
    && apt-get install -y \
        binfmt-support \
        binutils-arm-linux-gnueabihf \
        clang-3.6 \
        cmake \
        debootstrap \
        lldb-3.6 \
        lldb-3.6-dev \
        llvm-3.6 \
        make \
        qemu \
        qemu-user-static \
    && rm -rf /var/lib/apt/lists/*

chcosta/dotnetcore:ubuntu1404_coredeps_v1

FROM ubuntu:14.04

# Install tools used by the VSO build automation.  nodejs-legacy is a Debian specific
# package that provides 'node' on the path (which azure cli needs).
RUN apt-get update \
    && apt-get install -y \
        git \
        nodejs \
        nodejs-legacy \
        npm \
        tar \
        zip \
    && rm -rf /var/lib/apt/lists/* \
    && npm install -g azure-cli \
    && npm cache clean

# Dependencies for CoreCLR and CoreFX
RUN apt-get update \
    && apt-get install -y \
        gettext \
        libcurl4-openssl-dev \
        libicu-dev \
        liblttng-ust-dev \
        libssl-dev \
        libunwind8 \
        libunwind8-dev \
        uuid-dev \
    && rm -rf /var/lib/apt/lists/*

chcosta/dotnetcore:ubuntu1604_cross_prereqs_v1

FROM chcosta/dotnetcore:ubuntu1604_coredeps_v1

# Install the base toolchain we need to build anything (clang, cmake, make and the like).
RUN apt-get update \
    && apt-get install -y wget \
    && echo "deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty-3.6 main" | tee /etc/apt/sources.list.d/llvm.list \
    && wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key | apt-key add - \
    && apt-get update \
    && apt-get install -y \
        binfmt-support \
        binutils-arm-linux-gnueabihf \
        clang-3.6 \
        cmake \
        debootstrap \
        lldb-3.6 \
        lldb-3.6-dev \
        llvm-3.6 \
        make \
        qemu \
        qemu-user-static \
    && rm -rf /var/lib/apt/lists/*

chcosta/dotnetcore:ubuntu1604_coredeps_v1

FROM ubuntu:16.04

# Install tools used by the VSO build automation.  nodejs-legacy is a Debian specific
# package that provides `node' on the path (which azure cli needs).
RUN apt-get update \
    && apt-get install -y \
        git \
        nodejs \
        nodejs-legacy \
        npm \
        tar \
        zip \
    && rm -rf /var/lib/apt/lists/* \
    && npm install -g azure-cli \
    && npm cache clean

# Dependencies for CoreCLR and CoreFX
RUN apt-get update \
    && apt-get install -y \
        gettext \
        libcurl4-openssl-dev \
        libicu-dev \
        libkrb5-dev \
        liblttng-ust-dev \
        libssl-dev \
        libunwind8 \
        libunwind8-dev \
        uuid-dev \
    && rm -rf /var/lib/apt/lists/*

[jyoungyun]

@MichaelSimons Thanks for sharing! Please share with us when the move is complete. If we have to modify the DockerFiles, we will add a comment here.

[jyoungyun]

@MichaelSimons I added some pkgs for armel and tizen build.

FROM chcosta/dotnetcore:ubuntu1404_cross_prereqs_v1                             
                                                                                
RUN apt-get update \                                                            
    && apt-get install -y \                                                     
        binutils-arm-linux-gnueabi \                                            
        libxml2-utils \                                                         
        rpm2cpio \                                                              
    && rm -rf /var/lib/apt/lists/*   

I uploaded the docker image to docker hub (https://hub.docker.com/r/t2wish/dotnetcore/). The binutils-arm-linux-gnueabi and rpm2cpio can be removed if we do not create rootfs in docker image. And the libxml2-utils is installed by https://github.com/dotnet/core-setup/blob/master/cross/armel/tizen-fetch.sh.

[hqueue]

@gkhanna79 I've made a PR of experimental ARM CI for corefx using docker images at dotnet/corefx#15900 and you can find out below two builds are finished without failure.

• Innerloop Linux ARM Emulator HardFP Release Cross Build — Build finished. (https://ci.dot.net/job/dotnet_corefx/job/master/job/linuxarmemulator_softfp_cross_release_prtest/18/)
• Innerloop Linux ARM Emulator SoftFP Release Cross Build — Build finished. (https://ci.dot.net/job/dotnet_corefx/job/master/job/linuxarmemulator_hardfp_cross_release_prtest/16/)

I started working on corefx, because corefx has much simpler CI structure than coreclr and we need corefx artifacts to run tests in coreclr CI.

Short summary for experimental ARM CI.

• ARM CI for arm and armel
• Construct rootfs outside of docker, because we faced various error when constructing rootfs using debootstrap. For example, qemu-debootstrap is not working correctly inside docker image.
• Use docker images
  For arm, we used chcosta/dotnetcore:ubuntu1404_cross_prereqs_v1
  For armel, we used t2wish/dotnetcore:ubuntu1404_cross_prereqs_v2 provided by @jyoungyun above.

What do you think of this approach? If you think this is ok, we will refine code and start to apply this to core-setup, coreclr and corefx together.

/cc: @wateret

[hqueue]

@gkhanna79 Some update.

As a first step, we write a shell script (arm32_ci_script.sh) which will be used for CI of arm and armel and made three test PRs as follows.

• [ARM/CI] ARM CI script for arm and armel corefx#15900 : We invoked CI manually for arm and armel and it works.
• [ARM/CI] ARM CI script for arm and armel coreclr#9445 : We replace old CI for arm(which is actually armel) with docker and it works.
• [ARM/CI] Add ARM CI script for cross build core-setup#1512: Introduce CI script

It make use of docker as described in a previous comment above.

Please take a look into scripts/arm32_ci_script.sh in dotnet/corefx#15900 (https://github.com/dotnet/corefx/pull/15900/files#diff-9b74089ede263293591a48bb7430a9a4. )

There is a small difference in arm32_ci_script.sh between corefx and other two repo.
We will update arm32_ci_script.sh in three repo together with feedback.

When arm32_ci_script.sh is ready, we will update netci.groovy to enable CI.

[gkhanna79]

@jyoungyun Per the discussion in https://github.com/dotnet/coreclr/issues/9553#issuecomment-279319202, the alignment is towards constructing armel rootfs on the fly and not be in the Docker image. Given that, is your proposed change above (to the Docker image) still required?

[gkhanna79]

qemu-debootstrap is not working correctly inside docker image.

@hqueue I believe you maybe running into the same issue as I did. Making qemu-debootstrap requires the following:

1. Pre-reqs be installed on host machine as well
2. Cross build with docker must be done as sudo and docker container be constructed with "--privileged" argument.

Overall, the approach seems reasonable to me. I have left some comments on dotnet/corefx#15900 for you to consider. Also, I would suggest you bring it first up, with allrequired semantics in place, in CoreFX first and then follow up with the other repos.

[jyoungyun]

@gkhanna79 Yes, the pre-reqs are still required. The pre-reqs is for generating tizen rootfs so if we generate rootfs in Docker image, we still need those packages.

[hqueue]

Cross build with docker must be done as sudo and docker container be constructed with "--privileged" argument.

@gkhanna79 Thanks! It works :) I've updated dotnet/corefx#15900. Please take a look.

[hqueue]

@wateret Can we close this ?, because new ARM CI using Docker is now enabled as in dotnet/coreclr#10185.