Added support for 'Current' channel Dotnet Core in Dockerfile, using …

[Vladekk]

…local install for buildbot.

Apt-get prerequisite libs added, env vars set.
Package cache is pre-populated to speed-up first run of real apps

[bcomnes]

We could probably do this.

@rybit Thoughts?

[jhgoodwin]

I think this would be great. I was considering using an intermediate pipeline so my builds would not have to know how to install dotnet (not to mention the extra wait time). This would greatly simplify things for anyone using dotnet core.

[bcomnes]

I'll include this in the xenial image which is coming out soon.

[jhgoodwin]

@bcomnes fyi, I did a naive build using a dotnet tool. The site is created with Wyam, but the error returned seems like a missing library. From the error, it doesn't seem like it's a normal error for dotnet core. Based on the commit for v3.0.2, it seems that this change is in that version already:
Note the line below:

7:07:04 PM: A fatal error occurred, the required library libhostfxr.so could not be found.

In full context:

7:06:45 PM: Build ready to start
7:06:47 PM: build-image version: 324ec043422499a87b63cac1f1dabeefe6dca19d
7:06:47 PM: build-image tag: v3.0.2
7:06:47 PM: buildbot version: ef2e26260c41679f4cdeaebbf93370345c9fecf7
7:06:47 PM: Fetching cached dependencies
7:06:48 PM: Failed to fetch cache, continuing with build
7:06:48 PM: Starting to prepare the repo for build
7:06:48 PM: No cached dependencies found. Cloning fresh repo
7:06:48 PM: git clone https://github.com/jhgoodwin/site-johngoodwin.com
7:06:49 PM: Preparing Git Reference refs/heads/master
7:06:49 PM: Starting build script
7:06:49 PM: Installing dependencies
7:06:51 PM: v10.15.3 is already installed.
7:06:52 PM: Now using node v10.15.3 (npm v6.4.1)
7:06:52 PM: Attempting ruby version 2.6.2, read from environment
7:06:54 PM: Using ruby version 2.6.2
7:06:54 PM: Using PHP version 5.6
7:06:54 PM: Started restoring cached go cache
7:06:54 PM: Finished restoring cached go cache
7:06:54 PM: unset GOOS;
7:06:54 PM: unset GOARCH;
7:06:54 PM: export GOROOT='/opt/buildhome/.gimme/versions/go1.12.linux.amd64';
7:06:54 PM: export PATH="/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}";
7:06:54 PM: go version >&2;
7:06:54 PM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.12.linux.amd64.env';
7:06:54 PM: go version go1.12 linux/amd64
7:06:54 PM: Installing missing commands
7:06:54 PM: Verify run directory
7:06:54 PM: Executing user command: dotnet tool install -g Wyam.Tool && wyam build
7:07:04 PM: You can invoke the tool using the following command: wyam
7:07:04 PM: Tool 'wyam.tool' (version '2.2.4') was successfully installed.
7:07:04 PM: A fatal error occurred, the required library libhostfxr.so could not be found.
7:07:04 PM: If this is a self-contained application, that library should exist in [/opt/buildhome/.dotnet/tools/.store/wyam.tool/2.2.4/wyam.tool/2.2.4/tools/netcoreapp2.1/any/].
7:07:04 PM: If this is a framework-dependent application, install the runtime in the default location [/usr/share/dotnet] or use the DOTNET_ROOT environment variable to specify the runtime location.
7:07:04 PM: Caching artifacts
7:07:04 PM: Started saving pip cache
7:07:04 PM: Finished saving pip cache
7:07:04 PM: Started saving emacs cask dependencies
7:07:04 PM: Finished saving emacs cask dependencies
7:07:04 PM: Started saving maven dependencies
7:07:04 PM: Finished saving maven dependencies
7:07:04 PM: Started saving boot dependencies
7:07:04 PM: Finished saving boot dependencies
7:07:04 PM: Started saving go dependencies
7:07:04 PM: Finished saving go dependencies
7:07:15 PM: Cached node version v10.15.3
7:07:15 PM: Error running command: Build script returned non-zero exit code: 131
7:07:15 PM: Failing build: Failed to build site
7:07:16 PM: failed during stage 'building site': Build script returned non-zero exit code: 131
7:07:16 PM: Finished processing build request in 28.322645573s
7:07:16 PM: Shutting down logging, 0 messages pending

[bcomnes]

Im completely unfamiliar with dotnet core. @Vladekk @jhgoodwin any ideas on how to address?

[jhgoodwin]

@bcomnes Interesting.
If I use the netlify docker image, I don't seem to be able to repro this issue locally:

$ docker run --rm -it netlify/build:xenial bash
... lots of downloaded layers ...
Digest: sha256:cba778d5dc6b80d3276b216cdbbb42007cf0eb06993ea4234bea8245416daca1
Status: Downloaded newer image for netlify/build:xenial
$ cd /tmp
$ git clone https://github.com/jhgoodwin/site-johngoodwin.com.git
Cloning into 'site-johngoodwin.com'...
Username for 'https://github.com': jhgoodwin
Password for 'https://jhgoodwin@github.com': 
remote: Enumerating objects: 138, done.
remote: Counting objects: 100% (138/138), done.
remote: Compressing objects: 100% (125/125), done.
remote: Total 138 (delta 8), reused 138 (delta 8), pack-reused 0
Receiving objects: 100% (138/138), 3.39 MiB | 2.88 MiB/s, done.
Resolving deltas: 100% (8/8), done.
$ cd site-johngoodwin.com/
$ dotnet tool install -g Wyam.Tool
You can invoke the tool using the following command: wyam
Tool 'wyam.tool' (version '2.2.4') was successfully installed.
$ wyam build
... no error about libhostfxr.so ...

Any thoughts on why I can't repro locally using these steps?

Side note, I noticed there's a bunch of extra stuff on this image. It's about 2GB, and just the /tmp folder had about 445MB of data in it.

[bcomnes]

Oh good catch, I bet we can clean that up.

And we can't reproduce locally, Hrmm..... not ringing any bells for me. I wonder if its a permissions issue. Where does -g install to?

[bcomnes]

One idea I just had: You can run the exact image we use in production by using the squash tag.

docker run --rm -it netlify/build:v3.0.2-squash bash

[jhgoodwin]

From inside the container:

$ which wyam
/opt/buildhome/.dotnet/tools/wyam

[jhgoodwin]

@bcomnes ,
I retried the build locally using the squash - same result as the non-squashed - no repro.
When I tried editing the build commands to dump exports, whoami, pwd, and such, still no repro. Is it possible to get more details about the pre-build steps so I can recreate the same scenario?

So far, I put this into the build step, but it does not do the same thing as local:

cd /tmp && git clone https://github.com/jhgoodwin/wyam-blog-template && cd wyam-blog-template && dotnet tool install -g Wyam.Tool && wyam build

For repro inside docker, locally, I'm doing this:

docker run --rm -it netlify/build:v3.0.2-squash bash -c "cd /tmp && git clone https://github.com/jhgoodwin/wyam-blog-template && cd wyam-blog-template && dotnet tool install -g Wyam.Tool && wyam build"

It does not perform the same steps. The repo I pasted in the command is public, so needs no auth to read it. Any help on why they don't output the same result would be greatly appreciated.

[jhgoodwin]

I think I found the problem!

If I add this line to the beginning of my build:

export DOTNET_ROOT=/opt/buildhome/.dotnet

And it gets past this error!

I find this interesting, because you have this in your Dockerfile near line 451:

ENV DOTNET_ROOT "/opt/buildhome/.dotnet"

It was mentioned in this issue https://github.com/dotnet/cli/issues/9114 so, that gave me some interest to check the exports for this environment variable.

I haven't done any testing to see if there's a better way to fix this right in the Dockerfile definition, but it's a good start.

[mattferderer]

This looks merged into the xenial branch. Does this mean you can use .NET Core if using Ubuntu Xenial as your build image?

[MysterDru]

@mattferderer I can confirm builds can use dotnet core on the Xenial build image. I am using the following build script to build a wyam static site.

https://gist.github.com/MysterDru/c4d1fd5a905c92d0d89d6936b2d7a725