RunPipeline action failed

[kennetlindberg]

When running the build pipeline on a self hosted (azure) machine, this has happened multiple times. I have not been able to see the logic behind what is causing it. Sometimes it feels that it could be related to the version of BC in the container being built.

When I look in the Event Viewer of the build machine, I see the following event that seem to be related.
Handler for GET /v1.41/plugins/bc4925104438/json returned error: plugins are not supported on this platform

This is the error message from the Action:

RunPipeline action failed. Error: Docker Build failed with exit code 1 Stacktrace: at New-BcImage, C:\Users\vmadmin\AppData\Local\Temp\ac333ae3-13cb-442d-a689-8d7ec83db6d6\BcContainerHelper\ContainerHandling\New-NavImage.ps1: line 705 at New-BcContainer, C:\Users\vmadmin\AppData\Local\Temp\ac333ae3-13cb-442d-a689-8d7ec83db6d6\BcContainerHelper\ContainerHandling\New-NavContainer.ps1: line 605 at <ScriptBlock>, C:\Users\vmadmin\AppData\Local\Temp\ac333ae3-13cb-442d-a689-8d7ec83db6d6\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 693 at <ScriptBlock>, C:\Users\vmadmin\AppData\Local\Temp\ac333ae3-13cb-442d-a689-8d7ec83db6d6\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 915 at <ScriptBlock>, C:\Users\vmadmin\AppData\Local\Temp\ac333ae3-13cb-442d-a689-8d7ec83db6d6\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 845 at <ScriptBlock>, C:\Users\vmadmin\AppData\Local\Temp\ac333ae3-13cb-442d-a689-8d7ec83db6d6\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 827 at Run-AlPipeline, C:\Users\vmadmin\AppData\Local\Temp\ac333ae3-13cb-442d-a689-8d7ec83db6d6\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 789 at <ScriptBlock>, C:\actions-runner\_work\_actions\microsoft\AL-Go-Actions\v3.0\RunPipeline\RunPipeline.ps1: line 367 at <ScriptBlock>, C:\actions-runner\_work\_temp\d7794a38-2a94-4d82-8009-454534d84906.ps1: line 2 at <ScriptBlock>, <No file>: line 1

[freddydk]

Is there enough disk space and memory on these self-hosted runners?
How did you create them?

[freddydk]

Is there enough disk space and memory on these self-hosted runners?

[kennetlindberg]

Yes, I checked because I had that problem earlier, and there is plenty of disk space (90 GB free) and 16 GB memory (3.4 GB used).

[freddydk]

How are they created?
Which docker version is installed?

I use https://aka.ms/getbuildagent to create my agents (or actually a PS script) - and if they fail - I re-run the script, which removes the old agents and creates new ones.

Hasn't failed for months though.

[kennetlindberg]

Yes, I used that link to create the machine. But I can create a new one and see if it works better.

[freddydk]

How many agents did you select?

[kennetlindberg]

I believe I have added one agent manually after I created the machine. But I am really unsure about the details since this was a while back. Now I'm even unsure if I actually used the template when I think more about it. I'll try to create a new machine and see if that solves the problem.

[freddydk]

In the template you can also specify the disk space I think.
I use this script to create two beefy self-hosted agents in my org.
It will automatically remove old agents, remove old VMs, create new VMs and register agents:

$subscriptionId = $FreddysSubscription
try {
    Set-AzContext -Subscription $subscriptionId
} catch {
    Connect-AzAccount -Environment AzureCloud -TenantId $tenantId
    Set-AzContext -Subscription $subscriptionId
}

# Resource Group
$resGroup                 = "bcapprunner"
$resLocation              = "Sweden Central"
$org                      = "BusinessCentralApps"
$removeResGroupIfExist    = $true
$machines                 = 2
$offset                   = 0

if ($removeResGroupIfExist) {
    # Remove offline runners
    $runners = (gh api /orgs/$org/actions/runners | ConvertFrom-Json).runners
    if ($runners | Where-Object { $_.busy }) {
        Read-Host "Some agents are listed as busy. Press ENTER to proceed"
    }
}

$resourceGroup = Get-AzResourceGroup -name $resGroup -ErrorAction Ignore
if ($resourceGroup -and $removeResGroupIfExist) {
    Write-Host "Removing Resource Group $resGroup"
    Remove-AzResourceGroup -Name $resGroup -Force
    $resourceGroup = $null
}

if ($removeResGroupIfExist) {
    # Remove offline runners
    $runners = (gh api /orgs/$org/actions/runners | ConvertFrom-Json).runners
    $runners | Where-Object { $_.status -eq "offline" } | ForEach-Object {
        $id = $_.id
        gh api -X DELETE /orgs/$org/actions/runners/$id
    }
}

if (!($resourceGroup)) {
    $resourceGroup = New-AzResourceGroup -Name $resGroup -Location $resLocation -Force
}

1..$machines | ForEach-Object {
    # Deployment
    $no                       = $offset + $_
    $branch                   = "dev"
    $template                 = "buildagent"
    
    # ARM template
    $templateUri = "https://raw.githubusercontent.com/microsoft/nav-arm-templates/$branch/$template.json"
    
    $registrationToken = (gh api -X POST /orgs/$org/actions/runners/registration-token | ConvertFrom-Json).token
    
    $Parameters = @{
        "VmName"              = "$resgroup$no"
        "Remotedesktopaccess" = "-"
        "OperatingSystem"     = "Windows Server 2022"
        "VmSize"              = "Standard_D32as_v5"
        "OSDiskSize"          = 2000
        "StorageAccountType"  = "Premium_LRS"
        "AdminPassword"       = [SecureString]$AzureVmPasswordSecret.SecretValue
        "Count"               = 4
        "Token"               = $registrationToken
        "Organization"        =  "https://github.com/$org"
        "LabelsOrPool"        = "bc-build"
        "AgentUrl"            = "https://github.com/actions/runner/releases/download/v2.298.2/actions-runner-win-x64-2.298.2.zip"
        "RunInsideDocker"     = "No"
        "FinalSetupScriptUrl" = "additional-installforbuildagent.ps1"
    }
    
    $err = $resourceGroup | Test-AzResourceGroupDeployment -TemplateUri $templateUri -TemplateParameterObject $Parameters
    if ($err) {
        $err
        throw "stop"
    }
    $resourceGroup | New-AzResourceGroupDeployment -TemplateUri $templateUri -TemplateParameterObject $Parameters -Name $Parameters.vmName -ErrorAction Ignore
}

[freddydk]

Should be the same result with the template though

[freddydk]

BTW - I actually never remote desktop into the build machines, I never troubleshoot - I just recreate.

[trmsbj]

We're having this exact issue, but with an on-prem action runner in a VM instead of Azure.
The VM can start up a container with New-BCContainer, but fails in the github pipeline action runner.
Are there no way to see what the error actually is? The command just seems to return exit code 1.

[freddydk]

I don't think I get more info than the text returned after the error.
If you can create containers manually, but AL-Go cannot - then I would look at the output form both and compare.
Maybe hyperv vs. process isolation?

[trmsbj]

The container is created just fine and I can see the processes from the container in taskmanager, so it would appear it's running process isolation like the test container I created.
It fails on the 'New-BcImage' commandlet it would appear, I'd like to run that manually on the server, how can I find what parameters it's being executed with?

[trmsbj]

Nevermind, appears it's working now! I enabled the Hyper-V role and then it worked.