$PATH does not contain paths defined on /etc/environment

[marcuslopes]

Description

When running a task inside a self-hosted Ubuntu build agent (either manual or automatic scaleset), running echo $PATH on a self hosted agent returns a very limited number of paths.
Connecting to the agent via SSH and running echo $PATH directly will return the full range of paths inside the PATH variable though.

I am not sure if this can help in the troubleshooting, but I have noticed that even though my image was generated via this repo, there are comment lines and possibly extra config in MS-hosted agents in files like ~/.profile that are not present in my self-hosted agent. I wonder if there would be some extra configuration necessary that is done outside of the image generation scripts available in this repo.

Example of differences between my image's ~/.profile and the MS-hosted one:
self-hosted:

# ~/.profile: executed by Bourne-compatible login shells.

if [ "$BASH" ]; then
  if [ -f ~/.bashrc ]; then
    . ~/.bashrc
  fi
fi

mesg n || true

MS-hosted:

# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.

# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022

# if running bash
if [ -n "$BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
	. "$HOME/.bashrc"
    fi
fi

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$PATH"
fi

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

Virtual environments affected

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11
• Windows Server 2016
• Windows Server 2019

Image version and build link

Ubuntu 18
Image version: releases/ubuntu18/20210606 commit 58b026c
Agent version: 2.187.2

Ubuntu 20
Image version: releases/ubuntu20/20210606 commit a26b241
Agent version: 2.188.3

Packer: 1.6.2, 1.7.2

Is it regression?

No response

Expected behavior

Running a shell script task echo $PATH on a self hosted Ubuntu agent should return all of the paths defined inside the /etc/environment file:

PATH=/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:$HOME/.local/bin:/opt/pipx_bin:/usr/share/rust/.cargo/bin:$HOME/.config/composer/vendor/bin:/usr/local/.ghcup/bin:$HOME/.dotnet/tools:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

Actual behavior

Running a shell script task echo $PATH on a self hosted Ubuntu agent returns a very limited number of paths:

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

If I perform the following steps, $PATH gets filled as expected though:

1. Connect to the virtual machine
2. Open the agent folder /opt/vsts/a1
3. Run the following commands:

./env.sh
sudo ./svc.sh stop
sudo ./svc.sh start

Repro steps

1. Build a VM image using the Ubuntu image versions provided
2. Create a virtual machine on a scaleset using the newly generated image (either manual or automatic scaleset via Azure DevOps)
3. (If manual scaleset only) , install a vsts agent on the virtual machine using a release from the vsts agent repo (https://github.com/microsoft/azure-pipelines-agent)
4. Create a new pipeline (ex.: YAML) and run script echo $PATH

[MaksimZhukov]

Hello @marcuslopes! Thank you for your report. We will take a look

[MaksimZhukov]

Hello @marcuslopes. Thank you for your patience!
It seems that the issue is most likely related to the Azure pipeline agent, since, as you mentioned, the PATH environment variable with correct value exists on the image and is accessible via SSH connection. The fact that your issue can be solved by running the env.sh script and restarting the service (sudo ./svc.sh stop sudo ./svc.sh start) may also indicate that the problem may be caused by the Azure pipeline agent or probably its incorrect initial configuration.

Could you please create an issue in the microsoft/azure-pipelines-agent repository?

[marcuslopes]

Thank you for your time Maksim.
Following your advice, I created an issue in the azure-pipelines-agent: microsoft/azure-pipelines-agent#3461

[MaksimZhukov]

Thank you!
I'm going to close the issue, but feel free to contact us if you have any questions.

[ChristopheLav]

We continue to investigate with Marcus and we found that we have more than one issue. The PATH is not correct ($HOME was not replaced with the user account name) because we miss to execute the post generation scripts like documented here). So, we setup an VMSS extension to do the job when a VM instance is started.

The main issue remain but we think this can be useful for others :)

[ChristopheLav]

Hi @MaksimZhukov, can you jump into the discussion here microsoft/azure-pipelines-agent#3461 and help us? Thank you!

[kuleshovilya]

@MaksimZhukov @ChristopheLav We looked into the issue, and there is not much we can do on the agent side.
The issue seems to be stemming from the fact that /etc/environment is loaded on login, but script sets it after that, and agent is installed during that same login session, thus making it unable to read the /etc/env variables(since the agent loads PATH var itself)
I think the changes to the deployment scripts would be required to fix this issue.

[tonyskidmore]

I came across this thread after building an image using virtual-environments to a Shared Image Gallery and then following Azure virtual machine scale set agents to create a new "Azure virtual machine scale set" ADO Agent Pool and noticed that ansible could not be found when running a pipeline.

@ChristopheLav describes the issue precisely in microsoft/azure-pipelines-agent#3461 but both this issue and #3461 are both now closed so I am wondering how this problem will be handled to a resolution?

Both the virtual-environments image that I created and the ADO Agent Pool are deployed as per default and following Microsoft documentation and subsequently they do not work as expected. For now I have used the prependpath workaround:

- script: |
    echo "##vso[task.prependpath]/home/azureuser/.local/bin:/opt/pipx_bin:/usr/share/rust/.cargo/bin:/home/azureuser/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/azureuser/.dotnet/tools:"
  displayName: 'prependpath to fix incorrect path'

However, adding this to every pipeline is not a solution. Can this issue be re-opened?

[miketimofeev]

@tonyskidmore Unfortunately, we can't see any way to fix this in our scripts as the root cause seems to be in the VMSS Custom Script Extension behavior.

[tonyskidmore]

I have been working on a three part blog series on Azure DevOps Self-Hosted VMSS Agents. I looked at this issue again while working through that and came up with a workaround that seems to work for me, based on various comments in the history of this issue. Specifically, in the PATH issue in Part 2 I mention what I did to workaround this. Would be interested to get any feedback.

[HoLengZai]

Hi @miketimofeev, @kuleshovilya, @MaksimZhukov
I post a workaround to avoid any change on runner-images as it's not related to the runner-images but on how the VM pipeline agent extension for linux deploy the pipeline agent on the VM
microsoft/azure-pipelines-agent#3461 (comment)

@marcuslopes @ChristopheLav
It is also possible to add the .env file in /agent folder on the runner-image but I think it's better to create that .env file in the custom script Linux extension as it's really specific for the VM pipeline agent extension behaviour/install process