Problem building iotedge packages from Azure Build Pipeline

[gri6507]

I am trying to build the itoedge packages following https://github.com/Azure/iotedge/blob/master/edgelet/doc/devguide.md#building-os-packages inside an Azure YAML Build Pipeline (Ubuntu base image). The relevant steps from the pipeline are exactly the same as the ones from the instructions. The relevant YAML code is

- task: Bash@3
  displayName: Build package
  inputs:
    targetType: 'inline'
    script: |
      set -eux
  
      # Add the certificates to resolve the remote git repo URL. Add Curl and GPG to add the repo to get Moby
      DEBIAN_FRONTEND=noninteractive apt-get update
      DEBIAN_FRONTEND=noninteractive apt-get install -yq ca-certificates curl gpg
      update-ca-certificates
  
      # Add the microsoft repo so that we can get Moby (aka docker) runtime. See https://docs.microsoft.com/en-us/azure/iot-edge/how-to-install-iot-edge-linux
      curl https://packages.microsoft.com/config/ubuntu/$(DIST_RELEASE)/prod.list > /etc/apt/sources.list.d/microsoft-prod.list
      curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > /etc/apt/trusted.gpg.d/microsoft.gpg
      DEBIAN_FRONTEND=noninteractive apt-get update
      DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends moby-engine moby-cli
  
      # From https://github.com/Azure/iotedge/blob/master/edgelet/doc/devguide.md
      git clone --recurse-submodules https://github.com/Azure/iotedge
      cd iotedge
      PACKAGE_OS=$(DIST_VENDOR)$(DIST_RELEASE) PACKAGE_ARCH=amd64 ./edgelet/build/linux/package.sh

However, when I run this, I get an error inside package.sh which says cat: /__w/75/s/edgelet/version.txt: No such file or directory. To troubleshoot this, I've added some debug prints to the top of package.sh so that it now looks like this

#!/bin/bash

set -e

echo "@@@ package.sh - me = $0"
echo "@@@ package.sh - dirname = $(dirname "$0")"

# Get directory of running script
DIR="$(cd "$(dirname "$0")" && pwd)"
echo "@@@ package.sh - DIR = $DIR"

BUILD_REPOSITORY_LOCALPATH="$(realpath "${BUILD_REPOSITORY_LOCALPATH:-$DIR/../../..}")"
echo "@@@ package.sh - BUILD_REPOSITORY_LOCALPATH = $BUILD_REPOSITORY_LOCALPATH"

When I run the build pipeline with this modification, this sheds some more light on the error, producing the following output

@@@ package.sh - me = ./edgelet/build/linux/package.sh
@@@ package.sh - dirname = ./edgelet/build/linux
@@@ package.sh - DIR = /__w/75/s/iotedge/edgelet/build/linux
@@@ package.sh - BUILD_REPOSITORY_LOCALPATH = /__w/75/s
cat: /__w/75/s/edgelet/version.txt: No such file or directory

Clearly, something went wrong with generating BUILD_REPOSITORY_LOCALPATH from DIR.

[gri6507]

I think I found the first piece of the puzzle. Azure build pipelines define their own BUILD_REPOSITORY_LOCALPATH variable. If I unset that variable in the YAML pipeline before calling package.sh then it gets further (but not to completion - see below). May I recommend that the variable used in package.sh be changed to a different name, perhaps one with IOTEDGE in it?

As for the remaining problem, it appears that the docker command fails to run because /project in the container is empty. I am still investigating why that is the case and will update here once I know more

[gri6507]

I've tracked down the problem. ./edgelet/build/linux/package.sh internally runs docker. Since Azure pipelines are always started with volume mount of /var/run/docker.sock, this means that a docker invocation inside the pipeline uses mount directories with respect to the base OS, rather than the environment inside which it is running.

The way I am working around that now is by adding an optional argument to pass to docker inside ./edgelet/build/linux/package.sh which I can then leverage to volume mount the correct path. Please consider adding this as another improvement

[arsing]

May I recommend that the variable used in package.sh be changed to a different name, perhaps one with IOTEDGE in it?

I think you already realized, but just to be clear, using the same env var is intentional.

I've tracked down the problem.

Well you could just configure your job to run on the host instead of a container. That's how our builds work for example.

https://github.com/Azure/iotedge/blob/release/1.1/builds/misc/packages.yaml#L17-L18

[micahl]

Hi gri6507, what's the reason you'd like to setup an Azure Build Pipeline for the iotedge packages?

[gri6507]

Hello Micah Lewis (@micahl) - Microsoft only hosts iotedge packages for Tier 1 OSs. I need to run it on multiple Tier 2 OSs. As a result, I need to build my own packages.

[micahl]

gri6507 - with each release there are a set of packages built and made available that could be used on some of those Tier 2 OSs. For example, for the 1.1 release there are rpms that can be used on CentOS/RHEL 7 as well as .debs for various Debian based systems. Do none of those work for you? If not, which OS's are you targeting?

[gri6507]

Micah Lewis (@micahl) - Yes, I know that the release publishes the binary packages for Tier2 OSs. However, Microsoft does not, for example, host an APT repo for Ubuntu 20.04.

For example, https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/ does not have the iotedge package whereas https://packages.microsoft.com/ubuntu/18.04/prod/pool/main/ does

[veyalla]

You can install 18.04 iotedge packages on 20.04 OS (Tier 2) without any changes.