windows based runner vm fails to provision in azure

[rgl]

The windows based runner vm fails to provision because the CustomScriptExtension extension fails with exit code 1.

The pool is created as (full example is at https://github.com/rgl/terraform-azure-garm/tree/wip-windows-runner):

./garm-cli pool create \
  --enabled true \
  --org "$org_id" \
  --min-idle-runners 0 \
  --max-runners 2 \
  --tags garm-org-azure-amd64-windows-2022 \
  --runner-prefix rgl-garm-windows-org \
  --provider-name azure \
  --os-arch amd64 \
  --os-type windows \
  --flavor Standard_F2s_v2 \
  --image MicrosoftWindowsServer:WindowsServer:2022-Datacenter:latest \
  --extra-specs '{
    "storage_account_type": "StandardSSD_LRS",
    "disk_size_gb": 127
  }'

But the runner vm ends up failing the provisioning.

Here's the stripped down error message:

    <S S="Error">User runner already exists._x000D__x000A_</S>
    <S S="Error">At C:\run.ps1:333 char:3_x000D__x000A_</S>
    <S S="Error">+         Throw $Message_x000D__x000A_</S>
    <S S="Error">+         ~~~~~~~~~~~~~~_x000D__x000A_</S>
    <S S="Error">    + CategoryInfo          : OperationStopped: (User runner already exists.:String) [], RuntimeException_x000D__x000A_</S>
    <S S="Error">    + FullyQualifiedErrorId : User runner already exists._x000D__x000A_</S>
    <S S="Error"> _x000D__x000A_</S>

Here's the full error message:

#< CLIXML
<Objs Version="1.1.0.1"
    xmlns="http://schemas.microsoft.com/powershell/2004/04">
    <Obj S="progress" RefId="0">
        <TN RefId="0">
            <T>System.Management.Automation.PSCustomObject</T>
            <T>System.Object</T>
        </TN>
        <MS>
            <I64 N="SourceId">1</I64>
            <PR N="Record">
                <AV>Preparing modules for first use.</AV>
                <AI>0</AI>
                <Nil />
                <PI>-1</PI>
                <PC>-1</PC>
                <T>Completed</T>
                <SR>-1</SR>
                <SD></SD>
            </PR>
        </MS>
    </Obj>
    <Obj S="progress" RefId="1">
        <TNRef RefId="0" />
        <MS>
            <I64 N="SourceId">2</I64>
            <PR N="Record">
                <AV>Preparing modules for first use.</AV>
                <AI>0</AI>
                <Nil />
                <PI>-1</PI>
                <PC>-1</PC>
                <T>Completed</T>
                <SR>-1</SR>
                <SD></SD>
            </PR>
        </MS>
    </Obj>
    <Obj S="progress" RefId="2">
        <TNRef RefId="0" />
        <MS>
            <I64 N="SourceId">3</I64>
            <PR N="Record">
                <AV>Preparing modules for first use.</AV>
                <AI>0</AI>
                <Nil />
                <PI>-1</PI>
                <PC>-1</PC>
                <T>Completed</T>
                <SR>-1</SR>
                <SD></SD>
            </PR>
        </MS>
    </Obj>
    <Obj S="progress" RefId="3">
        <TNRef RefId="0" />
        <MS>
            <I64 N="SourceId">3</I64>
            <PR N="Record">
                <AV>Preparing modules for first use.</AV>
                <AI>0</AI>
                <Nil />
                <PI>-1</PI>
                <PC>-1</PC>
                <T>Completed</T>
                <SR>-1</SR>
                <SD></SD>
            </PR>
        </MS>
    </Obj>
    <Obj S="progress" RefId="4">
        <TNRef RefId="0" />
        <MS>
            <I64 N="SourceId">4</I64>
            <PR N="Record">
                <AV>Reading web response</AV>
                <AI>174593042</AI>
                <Nil />
                <PI>-1</PI>
                <PC>-1</PC>
                <T>Processing</T>
                <SR>-1</SR>
                <SD>Reading response stream... (Number of bytes read: 0)</SD>
            </PR>
        </MS>
    </Obj>
    <Obj S="progress" RefId="5">
        <TNRef RefId="0" />
        <MS>
            <I64 N="SourceId">4</I64>
            <PR N="Record">
                <AV>Reading web response</AV>
                <AI>174593042</AI>
                <Nil />
                <PI>-1</PI>
                <PC>-1</PC>
                <T>Completed</T>
                <SR>-1</SR>
                <SD>Reading web response completed. (Number of bytes read: 0)</SD>
            </PR>
        </MS>
    </Obj>
    <S S="Error">User runner already exists._x000D__x000A_</S>
    <S S="Error">At C:\run.ps1:333 char:3_x000D__x000A_</S>
    <S S="Error">+         Throw $Message_x000D__x000A_</S>
    <S S="Error">+         ~~~~~~~~~~~~~~_x000D__x000A_</S>
    <S S="Error">    + CategoryInfo          : OperationStopped: (User runner already exists.:String) [], RuntimeException_x000D__x000A_</S>
    <S S="Error">    + FullyQualifiedErrorId : User runner already exists._x000D__x000A_</S>
    <S S="Error"> _x000D__x000A_</S>
</Objs>

[gabriel-samfira]

quick question. Are you using a custom image that already has a user called "runner" ?

[gabriel-samfira]

We updated the userdata for windows in this PR:

cloudbase/garm-provider-common#46

The TL;DR is that we now use a regular user, added to the local Administrators group and with SeServiceLoginRight privilege set. This should cover the most use cases for any runner.

I don't think we retry creating the user, so one potential explanation is that the image you're using already has the "runner" user. One quick way to test is to overwrite the install template by running:

garm-cli pool update --extra-specs-file $HOME/specs.json <pool ID>

and in $HOME/specs.json you can add:

{
	"runner_install_template": "I3BzMV9zeXNuYXRpdmUKUGFyYW0oCglbUGFyYW1ldGVyKE1hbmRhdG9yeT0kZmFsc2UpXQoJW3N0cmluZ10kVG9rZW49Int7LkNhbGxiYWNrVG9rZW59fSIKKQoKJEVycm9yQWN0aW9uUHJlZmVyZW5jZT0iU3RvcCIKCmZ1bmN0aW9uIFN0YXJ0LUV4ZWN1dGVXaXRoUmV0cnkgewogICAgW0NtZGxldEJpbmRpbmcoKV0KICAgIHBhcmFtKAogICAgICAgIFtQYXJhbWV0ZXIoTWFuZGF0b3J5PSR0cnVlKV0KICAgICAgICBbU2NyaXB0QmxvY2tdJFNjcmlwdEJsb2NrLAogICAgICAgIFtpbnRdJE1heFJldHJ5Q291bnQ9MTAsCiAgICAgICAgW2ludF0kUmV0cnlJbnRlcnZhbD0zLAogICAgICAgIFtzdHJpbmddJFJldHJ5TWVzc2FnZSwKICAgICAgICBbYXJyYXldJEFyZ3VtZW50TGlzdD1AKCkKICAgICkKICAgIFBST0NFU1MgewogICAgICAgICRjdXJyZW50RXJyb3JBY3Rpb25QcmVmZXJlbmNlID0gJEVycm9yQWN0aW9uUHJlZmVyZW5jZQogICAgICAgICRFcnJvckFjdGlvblByZWZlcmVuY2UgPSAiQ29udGludWUiCiAgICAgICAgJHJldHJ5Q291bnQgPSAwCiAgICAgICAgd2hpbGUgKCR0cnVlKSB7CiAgICAgICAgICAgIHRyeSB7CiAgICAgICAgICAgICAgICAkcmVzID0gSW52b2tlLUNvbW1hbmQgLVNjcmlwdEJsb2NrICRTY3JpcHRCbG9jayAtQXJndW1lbnRMaXN0ICRBcmd1bWVudExpc3QKICAgICAgICAgICAgICAgICRFcnJvckFjdGlvblByZWZlcmVuY2UgPSAkY3VycmVudEVycm9yQWN0aW9uUHJlZmVyZW5jZQogICAgICAgICAgICAgICAgcmV0dXJuICRyZXMKICAgICAgICAgICAgfSBjYXRjaCBbU3lzdGVtLkV4Y2VwdGlvbl0gewogICAgICAgICAgICAgICAgJHJldHJ5Q291bnQrKwoKICAgICAgICAgICAgICAgIGlmICgkXy5FeGNlcHRpb24gLWlzIFtTeXN0ZW0uTmV0LldlYkV4Y2VwdGlvbl0pIHsKICAgICAgICAgICAgICAgICAgICAkd2ViUmVzcG9uc2UgPSAkXy5FeGNlcHRpb24uUmVzcG9uc2UKCQkJCQkjIFNraXAgcmV0cnkgb24gRXJyb3I6IDRYWCAoZS5nLiA0MDEgVW5hdXRob3JpemVkLCA0MDQgTm90IEZvdW5kIGV0Yy4pCiAgICAgICAgICAgICAgICAgICAgaWYgKCR3ZWJSZXNwb25zZSAtYW5kICR3ZWJSZXNwb25zZS5TdGF0dXNDb2RlIC1nZSA0MDAgLWFuZCAkd2ViUmVzcG9uc2UuU3RhdHVzQ29kZSAtbHQgNTAwKSB7CiAgICAgICAgICAgICAgICAgICAgICAgICMgU2tpcCByZXRyeSBvbiA0eHggZXJyb3JzCiAgICAgICAgICAgICAgICAgICAgICAgIFdyaXRlLU91dHB1dCAiRW5jb3VudGVyZWQgbm9uLXJldHJ5YWJsZSBlcnJvciAoNHh4KTogJCgkXy5FeGNlcHRpb24uTWVzc2FnZSkiCiAgICAgICAgICAgICAgICAgICAgICAgICRFcnJvckFjdGlvblByZWZlcmVuY2UgPSAkY3VycmVudEVycm9yQWN0aW9uUHJlZmVyZW5jZQogICAgICAgICAgICAgICAgICAgICAgICB0aHJvdwogICAgICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgICAgIH0KCiAgICAgICAgICAgICAgICBpZiAoJHJldHJ5Q291bnQgLWd0ICRNYXhSZXRyeUNvdW50KSB7CiAgICAgICAgICAgICAgICAgICAgJEVycm9yQWN0aW9uUHJlZmVyZW5jZSA9ICRjdXJyZW50RXJyb3JBY3Rpb25QcmVmZXJlbmNlCiAgICAgICAgICAgICAgICAgICAgdGhyb3cKICAgICAgICAgICAgICAgIH0gZWxzZSB7CiAgICAgICAgICAgICAgICAgICAgaWYgKCRSZXRyeU1lc3NhZ2UpIHsKICAgICAgICAgICAgICAgICAgICAgICAgV3JpdGUtT3V0cHV0ICRSZXRyeU1lc3NhZ2UKICAgICAgICAgICAgICAgICAgICB9IGVsc2VpZiAoJF8pIHsKICAgICAgICAgICAgICAgICAgICAgICAgV3JpdGUtT3V0cHV0ICRfCiAgICAgICAgICAgICAgICAgICAgfQogICAgICAgICAgICAgICAgICAgIFN0YXJ0LVNsZWVwIC1TZWNvbmRzICRSZXRyeUludGVydmFsCiAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgIH0KICAgICAgICB9CiAgICB9Cn0KCmZ1bmN0aW9uIEdldC1SYW5kb21TdHJpbmcgewogICAgW0NtZGxldEJpbmRpbmcoKV0KICAgIFBhcmFtKAogICAgICAgIFtpbnRdJExlbmd0aD0xMwogICAgKQogICAgUFJPQ0VTUyB7CiAgICAgICAgaWYoJExlbmd0aCAtbHQgNikgewogICAgICAgICAgICAkTGVuZ3RoID0gNgogICAgICAgIH0KICAgICAgICAkc3BlY2lhbCA9IEAoNDQsIDQ1LCA0NiwgNjQpCiAgICAgICAgJG51bWVyaWMgPSA0OC4uNTcKICAgICAgICAkdXBwZXIgPSA2NS4uOTAKICAgICAgICAkbG93ZXIgPSA5Ny4uMTIyCgogICAgICAgICRwYXNzd2QgPSBbU3lzdGVtLkNvbGxlY3Rpb25zLkdlbmVyaWMuTGlzdFtvYmplY3RdXShOZXctb2JqZWN0ICJTeXN0ZW0uQ29sbGVjdGlvbnMuR2VuZXJpYy5MaXN0W29iamVjdF0iKQogICAgICAgIGZvcigkaT0wOyAkaSAtbHQgJExlbmd0aC00OyAkaSsrKXsKICAgICAgICAgICAgJGMgPSBnZXQtcmFuZG9tIC1pbnB1dCAoJHNwZWNpYWwgKyAkbnVtZXJpYyArICR1cHBlciArICRsb3dlcikKICAgICAgICAgICAgJHBhc3N3ZC5BZGQoW2NoYXJdJGMpCiAgICAgICAgfQoKICAgICAgICAkcGFzc3dkLkFkZChbY2hhcl0oZ2V0LXJhbmRvbSAtaW5wdXQgJG51bWVyaWMpKQogICAgICAgICRwYXNzd2QuQWRkKFtjaGFyXShnZXQtcmFuZG9tIC1pbnB1dCAkc3BlY2lhbCkpCiAgICAgICAgJHBhc3N3ZC5BZGQoW2NoYXJdKGdldC1yYW5kb20gLWlucHV0ICR1cHBlcikpCiAgICAgICAgJHBhc3N3ZC5BZGQoW2NoYXJdKGdldC1yYW5kb20gLWlucHV0ICRsb3dlcikpCgogICAgICAgICRSYW5kb20gPSBOZXctT2JqZWN0IFJhbmRvbQogICAgICAgIHJldHVybiBbc3RyaW5nXTo6am9pbigiIiwoJHBhc3N3ZHxTb3J0LU9iamVjdCB7JFJhbmRvbS5OZXh0KCl9KSkKICAgIH0KfQoKQWRkLVR5cGUgLVR5cGVEZWZpbml0aW9uIEAiCnVzaW5nIFN5c3RlbTsKdXNpbmcgU3lzdGVtLlJ1bnRpbWUuSW50ZXJvcFNlcnZpY2VzOwp1c2luZyBTeXN0ZW0uVGV4dDsKCnB1YmxpYyBjbGFzcyBHcmFudFN5c1ByaXZpbGVnZXMKewogICAgW1N0cnVjdExheW91dChMYXlvdXRLaW5kLlNlcXVlbnRpYWwpXQogICAgcHVibGljIHN0cnVjdCBMU0FfVU5JQ09ERV9TVFJJTkcKICAgIHsKICAgICAgICBwdWJsaWMgdXNob3J0IExlbmd0aDsKICAgICAgICBwdWJsaWMgdXNob3J0IE1heGltdW1MZW5ndGg7CiAgICAgICAgcHVibGljIEludFB0ciBCdWZmZXI7CiAgICB9CgogICAgW1N0cnVjdExheW91dChMYXlvdXRLaW5kLlNlcXVlbnRpYWwpXQogICAgcHVibGljIHN0cnVjdCBMU0FfT0JKRUNUX0FUVFJJQlVURVMKICAgIHsKICAgICAgICBwdWJsaWMgaW50IExlbmd0aDsKICAgICAgICBwdWJsaWMgSW50UHRyIFJvb3REaXJlY3Rvcnk7CiAgICAgICAgcHVibGljIEludFB0ciBPYmplY3ROYW1lOwogICAgICAgIHB1YmxpYyB1aW50IEF0dHJpYnV0ZXM7CiAgICAgICAgcHVibGljIEludFB0ciBTZWN1cml0eURlc2NyaXB0b3I7CiAgICAgICAgcHVibGljIEludFB0ciBTZWN1cml0eVF1YWxpdHlPZlNlcnZpY2U7CiAgICB9CgogICAgW0RsbEltcG9ydCgiYWR2YXBpMzIuZGxsIiwgU2V0TGFzdEVycm9yPXRydWUpXQogICAgcHVibGljIHN0YXRpYyBleHRlcm4gdWludCBMc2FPcGVuUG9saWN5KAogICAgICAgIHJlZiBMU0FfVU5JQ09ERV9TVFJJTkcgU3lzdGVtTmFtZSwKICAgICAgICByZWYgTFNBX09CSkVDVF9BVFRSSUJVVEVTIE9iamVjdEF0dHJpYnV0ZXMsCiAgICAgICAgdWludCBEZXNpcmVkQWNjZXNzLAogICAgICAgIG91dCBJbnRQdHIgUG9saWN5SGFuZGxlCiAgICApOwoKICAgIFtEbGxJbXBvcnQoImFkdmFwaTMyLmRsbCIsIFNldExhc3RFcnJvcj10cnVlKV0KICAgIHB1YmxpYyBzdGF0aWMgZXh0ZXJuIHVpbnQgTHNhQWRkQWNjb3VudFJpZ2h0cygKICAgICAgICBJbnRQdHIgUG9saWN5SGFuZGxlLAogICAgICAgIEludFB0ciBBY2NvdW50U2lkLAogICAgICAgIExTQV9VTklDT0RFX1NUUklOR1tdIFVzZXJSaWdodHMsCiAgICAgICAgdWludCBDb3VudE9mUmlnaHRzCiAgICApOwoKICAgIFtEbGxJbXBvcnQoImFkdmFwaTMyLmRsbCIpXQogICAgcHVibGljIHN0YXRpYyBleHRlcm4gdWludCBMc2FDbG9zZShJbnRQdHIgUG9saWN5SGFuZGxlKTsKCiAgICBbRGxsSW1wb3J0KCJhZHZhcGkzMi5kbGwiKV0KICAgIHB1YmxpYyBzdGF0aWMgZXh0ZXJuIHVpbnQgTHNhTnRTdGF0dXNUb1dpbkVycm9yKHVpbnQgc3RhdHVzKTsKCiAgICBwdWJsaWMgY29uc3QgdWludCBQT0xJQ1lfQUxMX0FDQ0VTUyA9IDB4MDBGMEZGRjsKCiAgICBwdWJsaWMgc3RhdGljIHVpbnQgR3JhbnRQcml2aWxlZ2UoYnl0ZVtdIHNpZCwgc3RyaW5nW10gcmlnaHRzKQogICAgewogICAgICAgIExTQV9PQkpFQ1RfQVRUUklCVVRFUyBsb2EgPSBuZXcgTFNBX09CSkVDVF9BVFRSSUJVVEVTKCk7CiAgICAgICAgTFNBX1VOSUNPREVfU1RSSU5HIHN5c3RlbU5hbWUgPSBuZXcgTFNBX1VOSUNPREVfU1RSSU5HKCk7CgogICAgICAgIEludFB0ciBwb2xpY3lIYW5kbGU7CiAgICAgICAgdWludCByZXN1bHQgPSBMc2FPcGVuUG9saWN5KHJlZiBzeXN0ZW1OYW1lLCByZWYgbG9hLCBQT0xJQ1lfQUxMX0FDQ0VTUywgb3V0IHBvbGljeUhhbmRsZSk7CiAgICAgICAgaWYgKHJlc3VsdCAhPSAwKQogICAgICAgIHsKICAgICAgICAgICAgcmV0dXJuIExzYU50U3RhdHVzVG9XaW5FcnJvcihyZXN1bHQpOwogICAgICAgIH0KCiAgICAgICAgTFNBX1VOSUNPREVfU1RSSU5HW10gdXNlclJpZ2h0cyA9IG5ldyBMU0FfVU5JQ09ERV9TVFJJTkdbcmlnaHRzLkxlbmd0aF07CiAgICAgICAgZm9yIChpbnQgaSA9IDA7IGkgPCByaWdodHMuTGVuZ3RoOyBpKyspCiAgICAgICAgewogICAgICAgICAgICBieXRlW10gYnl0ZXMgPSBFbmNvZGluZy5Vbmljb2RlLkdldEJ5dGVzKHJpZ2h0c1tpXSk7CiAgICAgICAgICAgIEludFB0ciBwdHIgPSBNYXJzaGFsLkFsbG9jSEdsb2JhbChieXRlcy5MZW5ndGgpOwogICAgICAgICAgICBNYXJzaGFsLkNvcHkoYnl0ZXMsIDAsIHB0ciwgYnl0ZXMuTGVuZ3RoKTsKCiAgICAgICAgICAgIHVzZXJSaWdodHNbaV0uQnVmZmVyID0gcHRyOwogICAgICAgICAgICB1c2VyUmlnaHRzW2ldLkxlbmd0aCA9ICh1c2hvcnQpYnl0ZXMuTGVuZ3RoOwogICAgICAgICAgICB1c2VyUmlnaHRzW2ldLk1heGltdW1MZW5ndGggPSAodXNob3J0KShieXRlcy5MZW5ndGgpOwogICAgICAgIH0KCiAgICAgICAgSW50UHRyIHNpZFB0ciA9IE1hcnNoYWwuQWxsb2NIR2xvYmFsKHNpZC5MZW5ndGgpOwogICAgICAgIE1hcnNoYWwuQ29weShzaWQsIDAsIHNpZFB0ciwgc2lkLkxlbmd0aCk7CgogICAgICAgIHJlc3VsdCA9IExzYUFkZEFjY291bnRSaWdodHMocG9saWN5SGFuZGxlLCBzaWRQdHIsIHVzZXJSaWdodHMsICh1aW50KXJpZ2h0cy5MZW5ndGgpOwogICAgICAgIExzYUNsb3NlKHBvbGljeUhhbmRsZSk7CgogICAgICAgIGZvcmVhY2ggKHZhciByaWdodCBpbiB1c2VyUmlnaHRzKQogICAgICAgIHsKICAgICAgICAgICAgTWFyc2hhbC5GcmVlSEdsb2JhbChyaWdodC5CdWZmZXIpOwogICAgICAgIH0KICAgICAgICBNYXJzaGFsLkZyZWVIR2xvYmFsKHNpZFB0cik7CgogICAgICAgIHJldHVybiBMc2FOdFN0YXR1c1RvV2luRXJyb3IocmVzdWx0KTsKICAgIH0KfQoiQCAtTGFuZ3VhZ2UgQ1NoYXJwCgpmdW5jdGlvbiBJbnZva2UtRmFzdFdlYlJlcXVlc3QgewoJW0NtZGxldEJpbmRpbmcoKV0KCVBhcmFtKAoJCVtQYXJhbWV0ZXIoTWFuZGF0b3J5PSRUcnVlLFZhbHVlRnJvbVBpcGVsaW5lPSR0cnVlLFBvc2l0aW9uPTApXQoJCVtTeXN0ZW0uVXJpXSRVcmksCgkJW1BhcmFtZXRlcihQb3NpdGlvbj0xKV0KCQlbc3RyaW5nXSRPdXRGaWxlLAoJCVtIYXNodGFibGVdJEhlYWRlcnM9QHt9LAoJCVtzd2l0Y2hdJFNraXBJbnRlZ3JpdHlDaGVjaz0kZmFsc2UKCSkKCVBST0NFU1MKCXsKCQlpZighKFtTeXN0ZW0uTWFuYWdlbWVudC5BdXRvbWF0aW9uLlBTVHlwZU5hbWVdJ1N5c3RlbS5OZXQuSHR0cC5IdHRwQ2xpZW50JykuVHlwZSkKCQl7CgkJCSRhc3NlbWJseSA9IFtTeXN0ZW0uUmVmbGVjdGlvbi5Bc3NlbWJseV06OkxvYWRXaXRoUGFydGlhbE5hbWUoIlN5c3RlbS5OZXQuSHR0cCIpCgkJfQoKCQlpZighJE91dEZpbGUpIHsKCQkJJE91dEZpbGUgPSAkVXJpLlBhdGhBbmRRdWVyeS5TdWJzdHJpbmcoJFVyaS5QYXRoQW5kUXVlcnkuTGFzdEluZGV4T2YoIi8iKSArIDEpCgkJCWlmKCEkT3V0RmlsZSkgewoJCQkJdGhyb3cgIlRoZSAiIk91dEZpbGUiIiBwYXJhbWV0ZXIgbmVlZHMgdG8gYmUgc3BlY2lmaWVkIgoJCQl9CgkJfQoKCQkkZnJhZ21lbnQgPSAkVXJpLkZyYWdtZW50LlRyaW0oJyMnKQoJCWlmICgkZnJhZ21lbnQpIHsKCQkJJGRldGFpbHMgPSAkZnJhZ21lbnQuU3BsaXQoIj0iKQoJCQkkYWxnb3JpdGhtID0gJGRldGFpbHNbMF0KCQkJJGhhc2ggPSAkZGV0YWlsc1sxXQoJCX0KCgkJaWYgKCEkU2tpcEludGVncml0eUNoZWNrIC1hbmQgJGZyYWdtZW50IC1hbmQgKFRlc3QtUGF0aCAkT3V0RmlsZSkpIHsKCQkJdHJ5IHsKCQkJCXJldHVybiAoVGVzdC1GaWxlSW50ZWdyaXR5IC1GaWxlICRPdXRGaWxlIC1BbGdvcml0aG0gJGFsZ29yaXRobSAtRXhwZWN0ZWRIYXNoICRoYXNoKQoJCQl9IGNhdGNoIHsKCQkJCVJlbW92ZS1JdGVtICRPdXRGaWxlCgkJCX0KCQl9CgoJCSRjbGllbnQgPSBuZXctb2JqZWN0IFN5c3RlbS5OZXQuSHR0cC5IdHRwQ2xpZW50CgkJZm9yZWFjaCAoJGsgaW4gJEhlYWRlcnMuS2V5cyl7CgkJCSRjbGllbnQuRGVmYXVsdFJlcXVlc3RIZWFkZXJzLkFkZCgkaywgJEhlYWRlcnNbJGtdKQoJCX0KCQkkdGFzayA9ICRjbGllbnQuR2V0U3RyZWFtQXN5bmMoJFVyaSkKCQkkcmVzcG9uc2UgPSAkdGFzay5SZXN1bHQKCQlpZigkdGFzay5Jc0ZhdWx0ZWQpIHsKCQkJJG1zZyA9ICJSZXF1ZXN0IGZvciBVUkwgJ3swfScgaXMgZmF1bHRlZC4gVGFzayBzdGF0dXM6IHsxfS4iIC1mIEAoJFVyaSwgJHRhc2suU3RhdHVzKQoJCQlpZigkdGFzay5FeGNlcHRpb24pIHsKCQkJCSRtc2cgKz0gIkV4Y2VwdGlvbiBkZXRhaWxzOiB7MH0iIC1mIEAoJHRhc2suRXhjZXB0aW9uKQoJCQl9CgkJCVRocm93ICRtc2cKCQl9CgkJJG91dFN0cmVhbSA9IE5ldy1PYmplY3QgSU8uRmlsZVN0cmVhbSAkT3V0RmlsZSwgQ3JlYXRlLCBXcml0ZSwgTm9uZQoKCQl0cnkgewoJCQkkdG90UmVhZCA9IDAKCQkJJGJ1ZmZlciA9IE5ldy1PYmplY3QgQnl0ZVtdIDFNQgoJCQl3aGlsZSAoKCRyZWFkID0gJHJlc3BvbnNlLlJlYWQoJGJ1ZmZlciwgMCwgJGJ1ZmZlci5MZW5ndGgpKSAtZ3QgMCkgewoJCQkJJHRvdFJlYWQgKz0gJHJlYWQKCQkJCSRvdXRTdHJlYW0uV3JpdGUoJGJ1ZmZlciwgMCwgJHJlYWQpOwoJCQl9CgkJfQoJCWZpbmFsbHkgewoJCQkkb3V0U3RyZWFtLkNsb3NlKCkKCQl9CgkJaWYoISRTa2lwSW50ZWdyaXR5Q2hlY2sgLWFuZCAkZnJhZ21lbnQpIHsKCQkJVGVzdC1GaWxlSW50ZWdyaXR5IC1GaWxlICRPdXRGaWxlIC1BbGdvcml0aG0gJGFsZ29yaXRobSAtRXhwZWN0ZWRIYXNoICRoYXNoCgkJfQoJfQp9CgpmdW5jdGlvbiBJbXBvcnQtQ2VydGlmaWNhdGUoKSB7CglbQ21kbGV0QmluZGluZygpXQoJcGFyYW0gKAoJCVtwYXJhbWV0ZXIoTWFuZGF0b3J5PSR0cnVlKV0KCQkkQ2VydGlmaWNhdGVEYXRhLAoJCVtwYXJhbWV0ZXIoTWFuZGF0b3J5PSRmYWxzZSldCgkJW1N5c3RlbS5TZWN1cml0eS5DcnlwdG9ncmFwaHkuWDUwOUNlcnRpZmljYXRlcy5TdG9yZUxvY2F0aW9uXSRTdG9yZUxvY2F0aW9uPSJMb2NhbE1hY2hpbmUiLAoJCVtwYXJhbWV0ZXIoTWFuZGF0b3J5PSRmYWxzZSldCgkJW1N5c3RlbS5TZWN1cml0eS5DcnlwdG9ncmFwaHkuWDUwOUNlcnRpZmljYXRlcy5TdG9yZU5hbWVdJFN0b3JlTmFtZT0iVHJ1c3RlZFB1Ymxpc2hlciIKCSkKCVBST0NFU1MKCXsKCQkkc3RvcmUgPSBOZXctT2JqZWN0IFN5c3RlbS5TZWN1cml0eS5DcnlwdG9ncmFwaHkuWDUwOUNlcnRpZmljYXRlcy5YNTA5U3RvcmUoCgkJCSRTdG9yZU5hbWUsICRTdG9yZUxvY2F0aW9uKQoJCSRzdG9yZS5PcGVuKFtTeXN0ZW0uU2VjdXJpdHkuQ3J5cHRvZ3JhcGh5Llg1MDlDZXJ0aWZpY2F0ZXMuT3BlbkZsYWdzXTo6UmVhZFdyaXRlKQoJCSRjZXJ0ID0gW1N5c3RlbS5TZWN1cml0eS5DcnlwdG9ncmFwaHkuWDUwOUNlcnRpZmljYXRlcy5YNTA5Q2VydGlmaWNhdGUyXTo6bmV3KCRDZXJ0aWZpY2F0ZURhdGEpCgkJJHN0b3JlLkFkZCgkY2VydCkKCX0KfQoKZnVuY3Rpb24gSW52b2tlLUFQSUNhbGwoKSB7CglbQ21kbGV0QmluZGluZygpXQoJcGFyYW0gKAoJCVtwYXJhbWV0ZXIoTWFuZGF0b3J5PSR0cnVlKV0KCQlbb2JqZWN0XSRQYXlsb2FkLAoJCVtwYXJhbWV0ZXIoTWFuZGF0b3J5PSR0cnVlKV0KCQlbc3RyaW5nXSRDYWxsYmFja1VSTAoJKQoJUFJPQ0VTU3sKCQlJbnZva2UtV2ViUmVxdWVzdCAtVXNlQmFzaWNQYXJzaW5nIC1NZXRob2QgUG9zdCAtSGVhZGVycyBAeyJBY2NlcHQiPSJhcHBsaWNhdGlvbi9qc29uIjsgIkF1dGhvcml6YXRpb24iPSJCZWFyZXIgJFRva2VuIn0gLVVyaSAkQ2FsbGJhY2tVUkwgLUJvZHkgKENvbnZlcnRUby1Kc29uICRQYXlsb2FkKSB8IE91dC1OdWxsCgl9Cn0KCmZ1bmN0aW9uIFVwZGF0ZS1HYXJtU3RhdHVzKCkgewoJW0NtZGxldEJpbmRpbmcoKV0KCXBhcmFtICgKCQlbcGFyYW1ldGVyKE1hbmRhdG9yeT0kdHJ1ZSldCgkJW3N0cmluZ10kTWVzc2FnZSwKCQlbcGFyYW1ldGVyKE1hbmRhdG9yeT0kZmFsc2UpXQoJCVtpbnQ2NF0kQWdlbnRJRD0wLAoJCVtwYXJhbWV0ZXIoTWFuZGF0b3J5PSRmYWxzZSldCgkJW3N0cmluZ10kU3RhdHVzPSJpbnN0YWxsaW5nIiwKCQlbcGFyYW1ldGVyKE1hbmRhdG9yeT0kdHJ1ZSldCgkJW3N0cmluZ10kQ2FsbGJhY2tVUkwKCSkKCVBST0NFU1N7CgkJJGJvZHkgPSBAewoJCQkic3RhdHVzIj0kU3RhdHVzCgkJCSJtZXNzYWdlIj0kTWVzc2FnZQoJCX0KCgkJaWYgKCRBZ2VudElEIC1uZSAwKSB7CgkJCSRib2R5WyJhZ2VudF9pZCJdID0gJEFnZW50SUQKCQl9CgkJSW52b2tlLUFQSUNhbGwgLVBheWxvYWQgJGJvZHkgLUNhbGxiYWNrVVJMICRDYWxsYmFja1VSTCB8IE91dC1OdWxsCgl9Cn0KCmZ1bmN0aW9uIEludm9rZS1HYXJtU3VjY2VzcygpIHsKCVtDbWRsZXRCaW5kaW5nKCldCglwYXJhbSAoCgkJW3BhcmFtZXRlcihNYW5kYXRvcnk9JHRydWUpXQoJCVtzdHJpbmddJE1lc3NhZ2UsCgkJW3BhcmFtZXRlcihNYW5kYXRvcnk9JHRydWUpXQoJCVtpbnQ2NF0kQWdlbnRJRCwKCQlbcGFyYW1ldGVyKE1hbmRhdG9yeT0kdHJ1ZSldCgkJW3N0cmluZ10kQ2FsbGJhY2tVUkwKCSkKCVBST0NFU1N7CgkJVXBkYXRlLUdhcm1TdGF0dXMgLU1lc3NhZ2UgJE1lc3NhZ2UgLUFnZW50SUQgJEFnZW50SUQgLUNhbGxiYWNrVVJMICRDYWxsYmFja1VSTCAtU3RhdHVzICJpZGxlIiB8IE91dC1OdWxsCgl9Cn0KCmZ1bmN0aW9uIEludm9rZS1HYXJtRmFpbHVyZSgpIHsKCVtDbWRsZXRCaW5kaW5nKCldCglwYXJhbSAoCgkJW3BhcmFtZXRlcihNYW5kYXRvcnk9JHRydWUpXQoJCVtzdHJpbmddJE1lc3NhZ2UsCgkJW3BhcmFtZXRlcihNYW5kYXRvcnk9JHRydWUpXQoJCVtzdHJpbmddJENhbGxiYWNrVVJMCgkpCglQUk9DRVNTewoJCVVwZGF0ZS1HYXJtU3RhdHVzIC1NZXNzYWdlICRNZXNzYWdlIC1DYWxsYmFja1VSTCAkQ2FsbGJhY2tVUkwgLVN0YXR1cyAiZmFpbGVkIiB8IE91dC1OdWxsCgkJVGhyb3cgJE1lc3NhZ2UKCX0KfQoKZnVuY3Rpb24gU2V0LVN5c3RlbUluZm8gewogICAgW0NtZGxldEJpbmRpbmcoKV0KICAgIHBhcmFtICgKICAgICAgICBbcGFyYW1ldGVyKE1hbmRhdG9yeT0kdHJ1ZSldCiAgICAgICAgW3N0cmluZ10kQ2FsbGJhY2tVUkwsCiAgICAgICAgW3BhcmFtZXRlcihNYW5kYXRvcnk9JHRydWUpXQogICAgICAgIFtzdHJpbmddJFJ1bm5lckRpciwKCQlbcGFyYW1ldGVyKE1hbmRhdG9yeT0kdHJ1ZSldCiAgICAgICAgW3N0cmluZ10kQmVhcmVyVG9rZW4KICAgICkKCiAgICAjIENvbnN0cnVjdCB0aGUgcGF0aCB0byB0aGUgLnJ1bm5lciBmaWxlCiAgICAkYWdlbnRJbmZvRmlsZSA9IEpvaW4tUGF0aCAkUnVubmVyRGlyICIucnVubmVyIgoKICAgICMgUmVhZCBhbmQgcGFyc2UgdGhlIEpTT04gY29udGVudCBmcm9tIHRoZSAucnVubmVyIGZpbGUKICAgICRhZ2VudEluZm8gPSBDb252ZXJ0RnJvbS1Kc29uIChHZXQtQ29udGVudCAtUmF3IC1QYXRoICRhZ2VudEluZm9GaWxlKQogICAgJEFnZW50SWQgPSAkYWdlbnRJbmZvLmFnZW50X2lkCgogICAgIyBSZXRyaWV2ZSBPUyBpbmZvcm1hdGlvbgogICAgJG9zSW5mbyA9IEdldC1XbWlPYmplY3QgLUNsYXNzIFdpbjMyX09wZXJhdGluZ1N5c3RlbQogICAgJG9zTmFtZSA9ICRvc0luZm8uQ2FwdGlvbgogICAgJG9zVmVyc2lvbiA9ICRvc0luZm8uVmVyc2lvbgoKICAgICMgU3RyaXAgc3RhdHVzIGZyb20gdGhlIGNhbGxiYWNrIFVSTAogICAgaWYgKCRDYWxsYmFja1VybCAtbWF0Y2ggJ14oLiopL3N0YXR1cygvKT8kJykgewogICAgICAgICRDYWxsYmFja1VybCA9ICRtYXRjaGVzWzFdCiAgICB9CgogICAgJFN5c0luZm9VcmwgPSAiJENhbGxiYWNrVXJsL3N5c3RlbS1pbmZvLyIKICAgICRQYXlsb2FkID0gQHsKICAgICAgICBvc19uYW1lICAgID0gJE9TTmFtZQogICAgICAgIG9zX3ZlcnNpb24gPSAkT1NWZXJzaW9uCiAgICAgICAgYWdlbnRfaWQgICA9ICRBZ2VudElkCiAgICB9IHwgQ29udmVydFRvLUpzb24KCiAgICAjIFNlbmQgdGhlIFBPU1QgcmVxdWVzdAogICAgdHJ5IHsKICAgICAgICBJbnZva2UtUmVzdE1ldGhvZCAtVXJpICRTeXNJbmZvVXJsIC1NZXRob2QgUG9zdCAtQm9keSAkUGF5bG9hZCAtQ29udGVudFR5cGUgJ2FwcGxpY2F0aW9uL2pzb24nIC1IZWFkZXJzIEB7ICdBdXRob3JpemF0aW9uJyA9ICJCZWFyZXIgJEJlYXJlclRva2VuIiB9IC1FcnJvckFjdGlvbiBTdG9wCiAgICB9IGNhdGNoIHsKICAgICAgICBXcml0ZS1PdXRwdXQgIkZhaWxlZCB0byBzZW5kIHRoZSBzeXN0ZW0gaW5mb3JtYXRpb24uIgogICAgfQp9CgokR0hSdW5uZXJHcm91cCA9ICJ7ey5HaXRIdWJSdW5uZXJHcm91cH19IgoKZnVuY3Rpb24gSW5zdGFsbC1SdW5uZXIoKSB7CgkkQ2FsbGJhY2tVUkw9Int7LkNhbGxiYWNrVVJMfX0iCglpZiAoISgkQ2FsbGJhY2tVUkwgLW1hdGNoICJeKC4qKS9zdGF0dXMoLyk/JCIpKSB7CgkJJENhbGxiYWNrVVJMID0gIiRDYWxsYmFja1VSTC9zdGF0dXMiCgl9CgoJaWYgKCRUb2tlbi5MZW5ndGggLWVxIDApIHsKCQlUaHJvdyAibWlzc2luZyBjYWxsYmFjayBhdXRoZW50aWNhdGlvbiB0b2tlbiIKCX0KCXRyeSB7CgkJJE1ldGFkYXRhVVJMPSJ7ey5NZXRhZGF0YVVSTH19IgoJCSREb3dubG9hZFVSTD0ie3suRG93bmxvYWRVUkx9fSIKCQlpZigkTWV0YWRhdGFVUkwgLWVxICIiKXsKCQkJVGhyb3cgIm1pc3NpbmcgbWV0YWRhdGEgVVJMIgoJCX0KCgkJIyBDcmVhdGUgdXNlciB3aXRoIGFkbWluaXN0cmF0b3IgcmlnaHRzIHRvIHJ1biBzZXJ2aWNlIGFzCgkJJHVzZXJQYXNzd2QgPSBHZXQtUmFuZG9tU3RyaW5nIC1MZW5ndGggMTAKCQkkc2VjUGFzc3dkID0gQ29udmVydFRvLVNlY3VyZVN0cmluZyAiJHVzZXJQYXNzd2QiIC1Bc1BsYWluVGV4dCAtRm9yY2UKCQlOZXctTG9jYWxVc2VyIC1OYW1lICJnYXJtcnVubmVyIiAtUGFzc3dvcmQgJHNlY1Bhc3N3ZCAtUGFzc3dvcmROZXZlckV4cGlyZXMgLVVzZXJNYXlOb3RDaGFuZ2VQYXNzd29yZAoJCSRwc2NyZWRzID0gTmV3LU9iamVjdCBTeXN0ZW0uTWFuYWdlbWVudC5BdXRvbWF0aW9uLlBTQ3JlZGVudGlhbCAoIi5cZ2FybXJ1bm5lciIsICRzZWNQYXNzd2QpCgkJJGFkbWluR3JwTmFtZSA9IChHZXQtQ2ltSW5zdGFuY2Ugd2luMzJfZ3JvdXAgLUZpbHRlciAnU0lEID0gIlMtMS01LTMyLTU0NCInKS5OYW1lCgkJaWYgKCEkYWRtaW5HcnBOYW1lKSB7CgkJCVRocm93ICJDb3VsZCBub3QgZmluZCBhZG1pbmlzdHJhdG9ycyBncm91cCBuYW1lIgoJCX0KCQlBZGQtTG9jYWxHcm91cE1lbWJlciAtR3JvdXAgJGFkbWluR3JwTmFtZSAtTWVtYmVyIHJ1bm5lcgoJCSRudEFjY3QgPSBOZXctT2JqZWN0IFN5c3RlbS5TZWN1cml0eS5QcmluY2lwYWwuTlRBY2NvdW50KCJnYXJtcnVubmVyIikKCQkkc2lkID0gJG50QWNjdC5UcmFuc2xhdGUoW1N5c3RlbS5TZWN1cml0eS5QcmluY2lwYWwuU2VjdXJpdHlJZGVudGlmaWVyXSkKCQkkc2lkQnl0ZXMgPSBOZXctT2JqZWN0IGJ5dGVbXSAoJHNpZC5CaW5hcnlMZW5ndGgpCgkJJHNpZC5HZXRCaW5hcnlGb3JtKCRzaWRCeXRlcywgMCkKCgkJJHJlc3VsdCA9IFtHcmFudFN5c1ByaXZpbGVnZXNdOjpHcmFudFByaXZpbGVnZSgkc2lkQnl0ZXMsICgiU2VCYXRjaExvZ29uUmlnaHQiLCAiU2VTZXJ2aWNlTG9nb25SaWdodCIpKQoJCWlmICgkcmVzdWx0IC1uZSAwKSB7CgkJICAgIFRocm93ICJGYWlsZWQgdG8gZ3JhbnQgcHJpdmlsZWdlcyIKCQl9CgoJCSRidW5kbGUgPSB3Z2V0IC1Vc2VCYXNpY1BhcnNpbmcgLUhlYWRlcnMgQHsiQWNjZXB0Ij0iYXBwbGljYXRpb24vanNvbiI7ICJBdXRob3JpemF0aW9uIj0iQmVhcmVyICRUb2tlbiJ9IC1VcmkgJE1ldGFkYXRhVVJML3N5c3RlbS9jZXJ0LWJ1bmRsZQoJCSRjb252ZXJ0ZWQgPSBDb252ZXJ0RnJvbS1Kc29uICRidW5kbGUKCQlmb3JlYWNoICgkaSBpbiAkY29udmVydGVkLnJvb3RfY2VydGlmaWNhdGVzLnBzb2JqZWN0LlByb3BlcnRpZXMpewoJCQkkZGF0YSA9IFtTeXN0ZW0uQ29udmVydF06OkZyb21CYXNlNjRTdHJpbmcoJGkuVmFsdWUpCgkJCUltcG9ydC1DZXJ0aWZpY2F0ZSAtQ2VydGlmaWNhdGVEYXRhICRkYXRhIC1TdG9yZU5hbWUgUm9vdCAtU3RvcmVMb2NhdGlvbiBMb2NhbE1hY2hpbmUKCQl9CgoJCSRydW5uZXJEaXIgPSAiQzpcYWN0aW9ucy1ydW5uZXIiCgkJIyBDaGVjayBpZiBhIGNhY2hlZCBydW5uZXIgaXMgYXZhaWxhYmxlCgkJaWYgKC1ub3QgKFRlc3QtUGF0aCAkcnVubmVyRGlyKSkgewoJCQkjIE5vIGNhY2hlZCBydW5uZXIgZm91bmQsIHByb2NlZWQgdG8gZG93bmxvYWQgYW5kIGV4dHJhY3QKCQkJVXBkYXRlLUdhcm1TdGF0dXMgLUNhbGxiYWNrVVJMICRDYWxsYmFja1VSTCAtTWVzc2FnZSAiZG93bmxvYWRpbmcgdG9vbHMgZnJvbSB7eyAuRG93bmxvYWRVUkwgfX0iCgoJCQkkZG93bmxvYWRUb2tlbj0ie3suVGVtcERvd25sb2FkVG9rZW59fSIKCQkJJERvd25sb2FkVG9rZW5IZWFkZXJzPUB7fQoJCQlpZiAoJGRvd25sb2FkVG9rZW4uTGVuZ3RoIC1ndCAwKSB7CgkJCQkkRG93bmxvYWRUb2tlbkhlYWRlcnM9QHsKCQkJCQkiQXV0aG9yaXphdGlvbiI9IkJlYXJlciAkZG93bmxvYWRUb2tlbiIKCQkJCX0KCQkJfQoKCQkJJGRvd25sb2FkUGF0aCA9IEpvaW4tUGF0aCAkZW52OlRNUCAie3sgLkZpbGVOYW1lIH19IgoJCQlTdGFydC1FeGVjdXRlV2l0aFJldHJ5IC1TY3JpcHRCbG9jayB7CgkJCQlJbnZva2UtRmFzdFdlYlJlcXVlc3QgLVVyaSAie3sgLkRvd25sb2FkVVJMIH19IiAtT3V0RmlsZSAkZG93bmxvYWRQYXRoIC1IZWFkZXJzICREb3dubG9hZFRva2VuSGVhZGVycwoJCQl9IC1NYXhSZXRyeUNvdW50IDUgLVJldHJ5SW50ZXJ2YWwgNSAtUmV0cnlNZXNzYWdlICJSZXRyeWluZyBkb3dubG9hZCBvZiBydW5uZXIuLi4iCgoJCQlta2RpciAkcnVubmVyRGlyCgkJCVVwZGF0ZS1HYXJtU3RhdHVzIC1DYWxsYmFja1VSTCAkQ2FsbGJhY2tVUkwgLU1lc3NhZ2UgImV4dHJhY3RpbmcgcnVubmVyIgoJCQlBZGQtVHlwZSAtQXNzZW1ibHlOYW1lIFN5c3RlbS5JTy5Db21wcmVzc2lvbi5GaWxlU3lzdGVtCgkJCVtTeXN0ZW0uSU8uQ29tcHJlc3Npb24uWmlwRmlsZV06OkV4dHJhY3RUb0RpcmVjdG9yeSgkZG93bmxvYWRQYXRoLCAiJHJ1bm5lckRpciIpCgkJfSBlbHNlIHsKCQkJVXBkYXRlLUdhcm1TdGF0dXMgLUNhbGxiYWNrVVJMICRDYWxsYmFja1VSTCAtTWVzc2FnZSAidXNpbmcgY2FjaGVkIHJ1bm5lciBmb3VuZCBhdCAkcnVubmVyRGlyIgoJCX0KCgkJIyBFbnN1cmUgcnVubmVyIGhhcyBmdWxsIGFjY2VzcyB0byBhY3Rpb25zLXJ1bm5lciBmb2xkZXIKCQkkcnVubmVyQUNMID0gR2V0LUFjbCAkcnVubmVyRGlyCgkJJHJ1bm5lckFDTC5TZXRBY2Nlc3NSdWxlKChOZXctT2JqZWN0IFN5c3RlbS5TZWN1cml0eS5BY2Nlc3NDb250cm9sLkZpbGVTeXN0ZW1BY2Nlc3NSdWxlKAoJCSAgICAiZ2FybXJ1bm5lciIsICJGdWxsQ29udHJvbCIsICJDb250YWluZXJJbmhlcml0LE9iamVjdEluaGVyaXQiLCAiTm9uZSIsICJBbGxvdyIKCQkpKSkKCQlTZXQtQWNsIC1QYXRoICRydW5uZXJEaXIgLUFjbE9iamVjdCAkcnVubmVyQWNsCgoJCVVwZGF0ZS1HYXJtU3RhdHVzIC1DYWxsYmFja1VSTCAkQ2FsbGJhY2tVUkwgLU1lc3NhZ2UgImNvbmZpZ3VyaW5nIGFuZCBzdGFydGluZyBydW5uZXIiCgkJY2QgJHJ1bm5lckRpcgoKCQl7ey0gaWYgLlVzZUpJVENvbmZpZyB9fQoJCVVwZGF0ZS1HYXJtU3RhdHVzIC1DYWxsYmFja1VSTCAkQ2FsbGJhY2tVUkwgLU1lc3NhZ2UgImRvd25sb2FkaW5nIEpJVCBjcmVkZW50aWFscyIKCQl3Z2V0IC1Vc2VCYXNpY1BhcnNpbmcgLUhlYWRlcnMgQHsiQWNjZXB0Ij0iYXBwbGljYXRpb24vanNvbiI7ICJBdXRob3JpemF0aW9uIj0iQmVhcmVyICRUb2tlbiJ9IC1VcmkgJE1ldGFkYXRhVVJML2NyZWRlbnRpYWxzL3J1bm5lciAtT3V0RmlsZSAoSm9pbi1QYXRoICRydW5uZXJEaXIgIi5ydW5uZXIiKQoJCXdnZXQgLVVzZUJhc2ljUGFyc2luZyAtSGVhZGVycyBAeyJBY2NlcHQiPSJhcHBsaWNhdGlvbi9qc29uIjsgIkF1dGhvcml6YXRpb24iPSJCZWFyZXIgJFRva2VuIn0gLVVyaSAkTWV0YWRhdGFVUkwvY3JlZGVudGlhbHMvY3JlZGVudGlhbHMgLU91dEZpbGUgKEpvaW4tUGF0aCAkcnVubmVyRGlyICIuY3JlZGVudGlhbHMiKQoKCQlBZGQtVHlwZSAtQXNzZW1ibHlOYW1lIFN5c3RlbS5TZWN1cml0eQoJCSRyc2FEYXRhID0gKHdnZXQgLVVzZUJhc2ljUGFyc2luZyAtSGVhZGVycyBAeyJBY2NlcHQiPSJhcHBsaWNhdGlvbi9qc29uIjsgIkF1dGhvcml6YXRpb24iPSJCZWFyZXIgJFRva2VuIn0gLVVyaSAkTWV0YWRhdGFVUkwvY3JlZGVudGlhbHMvY3JlZGVudGlhbHNfcnNhcGFyYW1zKQoJCSRlbmNvZGVkQnl0ZXMgPSBbU3lzdGVtLlRleHQuRW5jb2RpbmddOjpVVEY4LkdldEJ5dGVzKCRyc2FEYXRhKQoJCSRwcm90ZWN0ZWRCeXRlcyA9IFtTZWN1cml0eS5DcnlwdG9ncmFwaHkuUHJvdGVjdGVkRGF0YV06OlByb3RlY3QoICRlbmNvZGVkQnl0ZXMsICRudWxsLCBbU2VjdXJpdHkuQ3J5cHRvZ3JhcGh5LkRhdGFQcm90ZWN0aW9uU2NvcGVdOjpMb2NhbE1hY2hpbmUgKQoJCVtTeXN0ZW0uSU8uRmlsZV06OldyaXRlQWxsQnl0ZXMoKEpvaW4tUGF0aCAkcnVubmVyRGlyICIuY3JlZGVudGlhbHNfcnNhcGFyYW1zIiksICRwcm90ZWN0ZWRCeXRlcykKCgkJJHNlcnZpY2VOYW1lRmlsZSA9IChKb2luLVBhdGggJHJ1bm5lckRpciAiLnNlcnZpY2UiKQoJCXdnZXQgLVVzZUJhc2ljUGFyc2luZyAtSGVhZGVycyBAeyJBY2NlcHQiPSJhcHBsaWNhdGlvbi9qc29uIjsgIkF1dGhvcml6YXRpb24iPSJCZWFyZXIgJFRva2VuIn0gLVVyaSAkTWV0YWRhdGFVUkwvc3lzdGVtL3NlcnZpY2UtbmFtZSAtT3V0RmlsZSAkc2VydmljZU5hbWVGaWxlCgoJCVVwZGF0ZS1HYXJtU3RhdHVzIC1DYWxsYmFja1VSTCAkQ2FsbGJhY2tVUkwgLU1lc3NhZ2UgIkNyZWF0aW5nIHN5c3RlbSBzZXJ2aWNlIgoJCSRTVkNfTkFNRT0oZ2MgLXJhdyAkc2VydmljZU5hbWVGaWxlKQoJCU5ldy1TZXJ2aWNlIC1OYW1lICIkU1ZDX05BTUUiIC1CaW5hcnlQYXRoTmFtZSAiQzpcYWN0aW9ucy1ydW5uZXJcYmluXFJ1bm5lclNlcnZpY2UuZXhlIiAtRGlzcGxheU5hbWUgIiRTVkNfTkFNRSIgLURlc2NyaXB0aW9uICJHaXRIdWIgQWN0aW9ucyBSdW5uZXIgKCRTVkNfTkFNRSkiIC1TdGFydHVwVHlwZSBBdXRvbWF0aWMgLUNyZWRlbnRpYWwgJHBzY3JlZHMKCQlTdGFydC1TZXJ2aWNlICIkU1ZDX05BTUUiCgkJU2V0LVN5c3RlbUluZm8gLUNhbGxiYWNrVVJMICRDYWxsYmFja1VSTCAtUnVubmVyRGlyICRydW5uZXJEaXIgLUJlYXJlclRva2VuICRUb2tlbgoJCVVwZGF0ZS1HYXJtU3RhdHVzIC1NZXNzYWdlICJydW5uZXIgc3VjY2Vzc2Z1bGx5IGluc3RhbGxlZCIgLUNhbGxiYWNrVVJMICRDYWxsYmFja1VSTCAtU3RhdHVzICJpZGxlIiB8IE91dC1OdWxsCgoJCXt7LSBlbHNlIH19CgkJJEdpdGh1YlJlZ2lzdHJhdGlvblRva2VuID0gU3RhcnQtRXhlY3V0ZVdpdGhSZXRyeSAtU2NyaXB0QmxvY2sgewoJCQlJbnZva2UtV2ViUmVxdWVzdCAtVXNlQmFzaWNQYXJzaW5nIC1IZWFkZXJzIEB7IkFjY2VwdCI9ImFwcGxpY2F0aW9uL2pzb24iOyAiQXV0aG9yaXphdGlvbiI9IkJlYXJlciAkVG9rZW4ifSAtVXJpICRNZXRhZGF0YVVSTC9ydW5uZXItcmVnaXN0cmF0aW9uLXRva2VuLwoJCX0gLU1heFJldHJ5Q291bnQgNSAtUmV0cnlJbnRlcnZhbCA1IC1SZXRyeU1lc3NhZ2UgIlJldHJ5aW5nIGRvd25sb2FkIG9mIEdpdEh1YiByZWdpc3RyYXRpb24gdG9rZW4uLi4iCgkJe3stIGlmIC5HaXRIdWJSdW5uZXJHcm91cCB9fQoJCS4vY29uZmlnLmNtZCAtLXVuYXR0ZW5kZWQgLS11cmwgInt7IC5SZXBvVVJMIH19IiAtLXRva2VuICRHaXRodWJSZWdpc3RyYXRpb25Ub2tlbiAtLXJ1bm5lcmdyb3VwIHt7LkdpdEh1YlJ1bm5lckdyb3VwfX0gLS1uYW1lICJ7eyAuUnVubmVyTmFtZSB9fSIgLS1sYWJlbHMgInt7IC5SdW5uZXJMYWJlbHMgfX0iIC0tbm8tZGVmYXVsdC1sYWJlbHMgLS1lcGhlbWVyYWwgLS1ydW5hc3NlcnZpY2UgLS13aW5kb3dzbG9nb25hY2NvdW50IHJ1bm5lciAtLXdpbmRvd3Nsb2dvbnBhc3N3b3JkICIkdXNlclBhc3N3ZCIKCQl7ey0gZWxzZX19CgkJLi9jb25maWcuY21kIC0tdW5hdHRlbmRlZCAtLXVybCAie3sgLlJlcG9VUkwgfX0iIC0tdG9rZW4gJEdpdGh1YlJlZ2lzdHJhdGlvblRva2VuIC0tbmFtZSAie3sgLlJ1bm5lck5hbWUgfX0iIC0tbGFiZWxzICJ7eyAuUnVubmVyTGFiZWxzIH19IiAtLW5vLWRlZmF1bHQtbGFiZWxzIC0tZXBoZW1lcmFsIC0tcnVuYXNzZXJ2aWNlIC0td2luZG93c2xvZ29uYWNjb3VudCBydW5uZXIgLS13aW5kb3dzbG9nb25wYXNzd29yZCAiJHVzZXJQYXNzd2QiCgkJe3stIGVuZH19CgkJaWYgKCRMQVNURVhJVENPREUpIHsKCQkJVGhyb3cgIkZhaWxlZCB0byBjb25maWd1cmUgcnVubmVyLiBFcnIgY29kZSAkTEFTVEVYSVRDT0RFIgoJCX0KCQkkYWdlbnRJbmZvRmlsZSA9IEpvaW4tUGF0aCAkcnVubmVyRGlyICIucnVubmVyIgoJCSRhZ2VudEluZm8gPSBDb252ZXJ0RnJvbS1Kc29uIChnYyAtcmF3ICRhZ2VudEluZm9GaWxlKQoJCVNldC1TeXN0ZW1JbmZvIC1DYWxsYmFja1VSTCAkQ2FsbGJhY2tVUkwgLVJ1bm5lckRpciAkcnVubmVyRGlyIC1CZWFyZXJUb2tlbiAkVG9rZW4KCQlJbnZva2UtR2FybVN1Y2Nlc3MgLUNhbGxiYWNrVVJMICRDYWxsYmFja1VSTCAtTWVzc2FnZSAicnVubmVyIHN1Y2Nlc3NmdWxseSBpbnN0YWxsZWQiIC1BZ2VudElEICRhZ2VudEluZm8uYWdlbnRJZAoJCXt7LSBlbmQgfX0KCX0gY2F0Y2ggewoJCUludm9rZS1HYXJtRmFpbHVyZSAtQ2FsbGJhY2tVUkwgJENhbGxiYWNrVVJMIC1NZXNzYWdlICRfCgl9Cn0KSW5zdGFsbC1SdW5uZXIK"
}

This is the same windows userdata template you can find here: https://github.com/cloudbase/garm-provider-common/blob/4163ff05a62f7ffa1b8a6814fe5eefa01debfa0a/cloudconfig/templates.go#L207

but with the runner user renamed to garmrunner. Feel free to edit the template.

Let me know if this makes things work. If it does, then it's probably because you already have the runner user in a custom image. If that's the case, we can add a check in the userdata script.

[rgl]

You mean the image set by garm-cli pool create --image MicrosoftWindowsServer:WindowsServer:2022-Datacenter:latest? If so, that's exactly the image I'm using.

[gabriel-samfira]

hah. Let me try out the azure provider with the config you posted. I'll ping back soon.

[gabriel-samfira]

You mean the image set by garm-cli pool create --image MicrosoftWindowsServer:WindowsServer:2022-Datacenter:latest? If so, that's exactly the image I'm using.

Would you mind trying the template I posted above anyway, while I try to reproduce on my side?

[gabriel-samfira]

ahh. I know what's happening. When creating a VM on Azure, we need to specify an admin username. We use the value from the common package.

I will create a PR for the Azure provider to change the default admin username and also add an extra specs field to customize it. Until it merges, you can use the template I posted above. It should avoid the error.

[rgl]

It now fails with the rgl-garm-window\runner is already a member of group Administrators._x000D__x000A_ error:

<S S="Error">rgl-garm-window\runner is already a member of group Administrators._x000D__x000A_</S>
    <S S="Error">At C:\run.ps1:333 char:3_x000D__x000A_</S>
    <S S="Error">+         Throw $Message_x000D__x000A_</S>
    <S S="Error">+         ~~~~~~~~~~~~~~_x000D__x000A_</S>
    <S S="Error">    + CategoryInfo          : OperationStopped: (rgl-garm-window...Administrators.:String) [], RuntimeException_x000D__x000A_</S>
    <S S="Error">    + FullyQualifiedErrorId : rgl-garm-window\runner is already a member of group Administrators._x000D__x000A_</S>
    <S S="Error"> _x000D__x000A_</S>

Its somewhat hard to debug this. To make things easier, I would:

1. drop the go templates altogether, and instead, use standalone .ps1 file(s) that can be tested separately, with proper arguments, instead of go template syntax, that would allow us to more easily inspect them and do static code analyses on them in the github actions workflow.
2. modify the powershell code to have a global trap that shows the line (and stack trace) where the error happened.
3. document how to login into these machines and how to put them in debug mode. maybe this is already document, and I've missed it.

[gabriel-samfira]

The error you got is my fault. I failed to properly edit this line:

Add-LocalGroupMember -Group $adminGrpName -Member runner

I am working on a branch to fix this. But it might take a couple of days. After that I will cut a release for the azure provider, and rebuild the garm container image. Until then, please use the previous stable release of the azure provider. That should still work fine.

We chose templates because it was the simplest way to leverage userdata that is available in all clouds. Using a static install script is less easy for Windows on most clouds, as not all clouds use cloudbase-init. So everyone implemented their own thing. On Amazon for example, we need to wrap the userdata in <powershell></powershell> tags. There is no easy way to write a static file and call it from the userdata.

Regarding the static ps1, you could do that in theory, even now, but you'd have to embed the ps1 as base64 into the userdata and create a wrapper around it to dump it to disk, then call it with parameters.

But if testability is the goal, we could render the final template based on some test context and analyze the resulting setup script.

Sorry for the headache! I'll try to fix this as soon as possible.

[rgl]

ah no worries! I can wait!

as for the script, I think its easier to troubleshoot when its a standalone script, because, one can simply login into the machine, and directly execute it.

making garm dump the final script for the user to see, is a nice to have, but if the script is already standalone in the repository, as one can simple go into the repository and get it. for me, the most important part, its on how easy is to figure out what is the problem while launching a runner.

[gabriel-samfira]

as for the script, I think its easier to troubleshoot when its a standalone script, because, one can simply login into the machine, and directly execute it.

to some extent. Once the script calls into GARM and sets a status of "idle" or "failed", the token used to interact with the API is invalidated and cannot be run again. This is a security measure to ensure that the userdata script can only ever fetch one runner registration token. Otherwise, once a runner is set up and a workflow starts running, it's trivial for potential untrusted code to inspect the userdata of the instance its running on, get the GARM token and use that to generate as many runner registration tokens as they wish. An attacker could then register their own runners in your repo.

So the point is that by the time workflows run, the userdata of the instance and any credentials contained within, become useless.

for me, the most important part, its on how easy is to figure out what is the problem while launching a runner.

We can definitely improve on the error reporting. If you do a garm-cli runner show <runner name>, do you see the error message? There is a try/catch that phones home with the error.

[rgl]

as for the script, I think its easier to troubleshoot when its a standalone script, because, one can simply login into the machine, and directly execute it.

to some extent. Once the script calls into GARM and sets a status of "idle" or "failed", the token used to interact with the API is invalidated and cannot be run again. This is a security measure to ensure that the userdata script can only ever fetch one runner registration token. Otherwise, once a runner is set up and a workflow starts running, it's trivial for potential untrusted code to inspect the userdata of the instance its running on, get the GARM token and use that to generate as many runner registration tokens as they wish. An attacker could then register their own runners in your repo.

ah, got it! glossing over many details, for a bit more, one would need to have a different token, or some way to start a runner in debug mode to receive a different kind of token. so, maybe too much trouble for this.

hummm the threat model of this system seems very interesting! is it documented?

So the point is that by the time workflows run, the userdata of the instance and any credentials contained within, become useless.

for me, the most important part, its on how easy is to figure out what is the problem while launching a runner.

We can definitely improve on the error reporting. If you do a garm-cli runner show <runner name>, do you see the error message? There is a try/catch that phones home with the error.

Oh, I did not realize there was a show command! Thanks for pointing it out! I was getting the error from the Azure Portal, at the VM script extension page.

This is what it returns when there is an error:

$ ./garm-cli runner show "$runner_name"
+-----------------+------------------------------------------------------------------------------------------+
| FIELD           | VALUE                                                                                    |
+-----------------+------------------------------------------------------------------------------------------+
| ID              | 0ccaff47-020b-4747-9803-0bb1bc81d48d                                                     |
| Created At      | 2025-07-30 18:39:46.171766407 +0000 UTC                                                  |
| Updated At      | 2025-07-30 18:42:16.437660538 +0000 UTC                                                  |
| Provider ID     | rgl-garm-windows-org-yQyfzOj5FVZ0                                                        |
| Name            | rgl-garm-windows-org-yQyfzOj5FVZ0                                                        |
| OS Type         | windows                                                                                  |
| OS Architecture | amd64                                                                                    |
| OS Name         | 2022-Datacenter                                                                          |
| OS Version      | latest                                                                                   |
| Status          | running                                                                                  |
| Runner Status   | failed                                                                                   |
| Pool ID         | 1d00dddf-1038-46b2-9d03-b5e2f0fb4faf                                                     |
| Status Updates  | 2025-07-30T18:42:16: rgl-garm-window\runner is already a member of group Administrators. |
+-----------------+------------------------------------------------------------------------------------------+

$ ./garm-cli runner show "$runner_name" --format json | jq
{
  "id": "0ccaff47-020b-4747-9803-0bb1bc81d48d",
  "provider_id": "rgl-garm-windows-org-yQyfzOj5FVZ0",
  "agent_id": 48,
  "name": "rgl-garm-windows-org-yQyfzOj5FVZ0",
  "os_type": "windows",
  "os_name": "2022-Datacenter",
  "os_version": "latest",
  "os_arch": "amd64",
  "status": "running",
  "runner_status": "failed",
  "pool_id": "1d00dddf-1038-46b2-9d03-b5e2f0fb4faf",
  "status_messages": [
    {
      "created_at": "2025-07-30T18:42:16.436901233Z",
      "message": "rgl-garm-window\\runner is already a member of group Administrators.",
      "event_type": "status",
      "event_level": "info"
    }
  ],
  "created_at": "2025-07-30T18:39:46.171766407Z",
  "updated_at": "2025-07-30T18:42:16.437660538Z"
}

Maybe that event_level should have been error or the word "ERROR" should be in the message. Or even have a "Global Error" kinda of error. Something that make it obvious there was an error (even thou the runner_status already says failed).

It also does not show a stack trace. But since this is generated from a template, maybe matching the stack trace lines from the template is not simple or even possible.

I also noticed that we cannot use the runner id in the garm-cli runner show command (the other calls like garm-cli pool update can use an id) instead we have to use the runner name.

Are these minor issues worthy of a GitHub Issue?

Ah, and modifying Add-LocalGroupMember -Group $adminGrpName -Member runner to use garmrunner made it work here! :-)

[gabriel-samfira]

hummm the threat model of this system seems very interesting! is it documented?

I've been meaning to write a design doc with why some things work the way they do. Hopefully I can find some time to do that in a way that won't put people to sleep.

Maybe that event_level should have been error or the word "ERROR" should be in the message. Or even have a "Global Error" kinda of error. Something that make it obvious there was an error (even thou the runner_status already says failed).

Yep. I always meant to use the events for more things and perhaps set proper levels. I've been looking at otel as an option instead of baking our own thing. I added status updates for runners to give some transparency, but trying out otel seemed like too much for the time I had available. But we can certainly set proper event log levels.

I also noticed that we cannot use the runner id in the garm-cli runner show command (the other calls like garm-cli pool update can use an id) instead we have to use the runner name.

The runner names are unique. The ID was used as the primary key because we used the same base model for all database models when I first wrote GARM a few years ago. The goal now is to allow friendly names along side IDs. The main branch for example allows you to use repo/org/enterprise names instead of IDs when creating pools. But to keep things consistent, we should allow referencing runners by both names and IDs.

Are these minor issues worthy of a GitHub Issue?

We can use this issue to track them.

Ah, and modifying Add-LocalGroupMember -Group $adminGrpName -Member runner to use garmrunner made it work here! :-)

That's awesome! I'm hoping to fix this in garm-provider-common and then update the azure provider soon.

[gabriel-samfira]

@rgl @chickenkiller

Would you mind trying the template from: cloudbase/garm-provider-common#49 ?

you can set it in extra specs. I tested it on my side and it seems to work well, but I would like for someone else to validate as well. If it works, I'll merge it and update the azure provider to use it.

[chickenkiller]

@rgl @chickenkiller

Would you mind trying the template from: cloudbase/garm-provider-common#49 ?

Hello @gabriel-samfira I confirm that it works:

+-----------------+------------------------------------------------------------------------------------------------------+
| FIELD           | VALUE                                                                                                |
+-----------------+------------------------------------------------------------------------------------------------------+
| ID              | a5be8138-c9a7-47f7-96b5-fd4dc0eaf7d1                                                                 |
| Created At      | 2025-08-06 06:26:25.951794484 +0000 UTC                                                              |
| Updated At      | 2025-08-06 06:29:08.526938997 +0000 UTC                                                              |
| Provider ID     | garmdev-ncM8br3Bck4W                                                                                 |
| Name            | garmdev-ncM8br3Bck4W                                                                                 |
| OS Type         | windows                                                                                              |
| OS Architecture | amd64                                                                                                |
| OS Name         | Microsoft Windows Server 2022 Datacenter Azure Edition                                               |
| OS Version      | 10.0.20348                                                                                           |
| Status          | running                                                                                              |
| Runner Status   | idle                                                                                                 |
| Pool ID         | 9e6e5227-3f02-4333-9f4c-fcc65c8ea33a                                                                 |
| Status Updates  | 2025-08-06T06:28:52: downloading tools from https://github.mycompany/_services/pipeline              |
|                 | s/_apis/distributedtask/packagedownload/agent/win-x64/2.321.0                                        |
|                 | 2025-08-06T06:28:53: extracting runner                                                               |
|                 | 2025-08-06T06:28:55: configuring and starting runner                                                 |
|                 | 2025-08-06T06:28:55: downloading JIT credentials                                                     |
|                 | 2025-08-06T06:28:55: Creating system service                                                         |
|                 | 2025-08-06T06:29:08: runner successfully installed                                                   |
+-----------------+------------------------------------------------------------------------------------------------------+