Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure Pipelines build system not recognized with non-Windows jobs #2432

Closed
gitfool opened this Issue Jan 7, 2019 · 9 comments

Comments

Projects
None yet
2 participants
@gitfool
Copy link
Contributor

commented Jan 7, 2019

See the logs for all jobs in Cake.Dungeon/#20190107.3:

There are a couple of problems; one is that TF_BUILD is expected in TFBuildProvider.cs#L46 but is missing in the Docker job environment variables, and the other is that AGENT_NAME is always defined but not as expected in TFBuildProvider.cs#L82.

(The Mac job is currently broken for other reasons I haven't looked into yet.)

@devlead

This comment has been minimized.

Copy link
Member

commented Jan 7, 2019

Wonder, is that an error in Azure Pipelines or per design?

Checking docs at https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=vsts

Variable Description
TF_BUILD Set to True if the script is being run by a build task. This variable is agent-scoped. It can be used as an environment variable in a script and as a parameter in a build task, but not as part of the build number or as a version control tag.

Normally it's not available on the system but set by the process invoker on each task
https://github.com/Microsoft/azure-pipelines-agent/blob/148dbe4bb953da97a423ad899626c773371b3083/src/Agent.Sdk/ProcessInvoker.cs#L195

            // Set the TF_BUILD env variable.
            _proc.StartInfo.Environment["TF_BUILD"] = "True";

Looking at the docker bits it looks like they're just copying the host variables but not setting the extra "TF_BUILD=True"
https://github.com/Microsoft/azure-pipelines-agent/blob/30d2e8b9c2afe9cdb98d16016968616780cea933/src/Agent.Worker/Container/DockerCommandManager.cs#L123

So likely docker process is getting the variable, but not the container itself.

@gitfool maybe worth raising an issue on agent repo to ask.

@gitfool

This comment has been minimized.

Copy link
Contributor Author

commented Jan 7, 2019

@devlead Done and already confirmed as a bug on their side. Re the second problem, since the AGENT_NAME is no longer simply Hosted Agent, I suggest changing the relevant part of TFBuildProvider.IsHostedAgent from:

_environment.GetEnvironmentVariable("AGENT_NAME") == "Hosted Agent"

to:

_environment.GetEnvironmentVariable("AGENT_NAME").StartsWith("Hosted")

Either that or switching to or also checking for the SYSTEM_SERVERTYPE environment variable, which I noticed has the value Hosted for all Azure Pipelines builds. The first option would be backwards compatible so would be preferred?

@devlead

This comment has been minimized.

Copy link
Member

commented Jan 7, 2019

Isn't this from your build log?

2019-01-07T07:33:02.2586978Z AGENT_NAME                                                          Hosted Agent

Looks like Hosted Agent?

@gitfool

This comment has been minimized.

Copy link
Contributor Author

commented Jan 7, 2019

Yes, for the Docker job, but the other logs use different hosts; currently the Linux host is Hosted Ubuntu 1604 2, and I've seen other variants, so the Linux job is detected as running on TFS, rather than VSTS / Azure Pipelines.

@gitfool

This comment has been minimized.

Copy link
Contributor Author

commented Jan 7, 2019

I just noticed another problem. The TFBuildAgentInfo.MachineName is not set - a single " is how I log and distinguish empty strings - as it expects the AGENT_MACHINE_NAME environment variable, but it seems to be called AGENT_MACHINENAME instead.

@devlead

This comment has been minimized.

Copy link
Member

commented Jan 7, 2019

Yes, for the Docker job, but the other logs use different hosts; currently the Linux host is Hosted Ubuntu 1604 2

I see, well yes then startswith makes sense probably _environment.GetEnvironmentVariable("AGENT_NAME")?.StartsWith("Hosted") as it will be null if the environment variable is not found.

I just noticed another problem. The TFBuildAgentInfo.MachineName is not set - a single " is how I log and distinguish empty strings - as it expects the AGENT_MACHINE_NAME environment variable, but it seems to be called AGENT_MACHINENAME instead.

AGENT_MACHINENAME seems to be correct spelling according to docs.

Variable Description
Agent.MachineName The name of the machine on which the agent is installed

https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=vsts

@gitfool

This comment has been minimized.

Copy link
Contributor Author

commented Jan 7, 2019

Okay, I'll submit a PR with those two changes...

@devlead

This comment has been minimized.

Copy link
Member

commented Jan 7, 2019

@gitfool the mac build you can likely fix by adding a .NET installer task into yaml, example of such yaml

variables:
  DOTNET_SDK_VERSION: '2.1.502'

jobs:
- job: MacOS
  displayName: Build on MacOS

  pool:
    vmImage: 'macOS 10.13'

  steps:
  - task: DotNetCoreInstaller@0
    displayName: 'Use .NET Core SDK $(DOTNET_SDK_VERSION)'
    inputs:
      version: '$(DOTNET_SDK_VERSION)'

  - bash: ./build.sh
    displayName: 'Execute Cake Bootstrapper'

  - task: PublishTestResults@2
    displayName: 'Publish Test Results'   
    inputs:
      testRunner: xUnit
      testResultsFiles: '**/test_result.xml'
      mergeTestResults: true



- job: Windows
  displayName: Build on Windows

  pool:
    vmImage: 'VS2017-Win2016'

  steps:
  - task: DotNetCoreInstaller@0
    displayName: 'Use .NET Core SDK $(DOTNET_SDK_VERSION)'
    inputs:
      version: '$(DOTNET_SDK_VERSION)'


  - powershell: ./build.ps1
    displayName: 'Execute Cake PowerShell Bootstrapper'

  - task: PublishTestResults@2
    displayName: 'Publish Test Results'
    inputs:
      testRunner: XUnit
      testResultsFiles: '**/test_result.xml'
      mergeTestResults: true


- job: Linux
  displayName: Build on Linux

  pool:
    vmImage: 'Ubuntu 16.04'

  steps:
  - task: DotNetCoreInstaller@0
    displayName: 'Use .NET Core SDK $(DOTNET_SDK_VERSION)'
    inputs:
      version: '$(DOTNET_SDK_VERSION)'


  - bash: ./build.sh
    displayName: 'Execute Cake Bash Bootstrapper'

  - task: PublishTestResults@2
    displayName: 'Publish Test Results'
    inputs:
      testRunner: XUnit
      testResultsFiles: '**/test_result.xml'
      mergeTestResults: true


- job: Docker
  displayName: Build Docker Image

  pool:
    vmImage: 'Ubuntu 16.04'

  steps:
  - task: Docker@1
    displayName: 'Build image'
@gitfool

This comment has been minimized.

Copy link
Contributor Author

commented Jan 9, 2019

@devlead thanks for the tip; it works on Mac again!

FTR, it only started failing when I switched to GitVersion.Tool. From the logs the Mac agent had up to .NET Core 2.2.0 installed. I always update to the latest tools, so I added a DotNetCoreInstaller task to install .NET Core SDK 2.2.102 and the issue went away. 😈

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.