Scripts stuck in "Upcoming"

[pacamaster]

Fleet version:
Reported in Fleet Fleet 4.47.0 Go go1.21.7
osquery 5.11.0
Fleetd 1.22.0
Web browser and operating system:
Current version

---

💥  Actual behavior

After running scripts on an endpoint, the script is stuck in "Upcoming."

This doesn't let other scripts behind it run.

The expected behavior is that scripts, that take over 5 minutes to run, timeout and are moved to "Past." Then, scripts behind this one run.

🧑‍💻  Steps to reproduce

1. Enroll a Windows hosts w/ --enable-scripts
2. Run this script that attempts to install several Windows applications
3. See that sometimes the script is stuck in "Upcoming"

[sharon-fdm]

@georgekarrv, assign to MDM team?

[nonpunctual]

@sharon-fdm sorry if I assigned to the wrong team!

[valentinpezon-primo]

Relate to #17180

[nonpunctual]

From customer-preston:

(per request by for a use case for script purging)
Since:
we are doing script-based app management on our own
we have to sometimes download multi-Gb apps
we use one script to install all customer apps
And, on the Fleet side of script rules:
there is currently no way to manually clear or adjust the script queue
scripts will attempt execution 2x
if a script needs to run a second time after this it must be queued again
re-queue does not clear any previous failures or attempts
script history can be obtained with api (or activity feed) but the Fleet UI modal shows the latest (or pending)
This is what happens:
Our app-install scripts timeout
We consider it still running, since it's in the queue, so we do not attempt to run it again
Any other script (for recovery key, MB agent, ...) is queued and NOT RUN since the queue can't be purged
This is obviously a HUGE problem, since App Management is a key part of any MDM value prop, but especially on SMBs + this blocks any other form of script exec
We really need solutions from you on this, starting with but not limited to:
More permissive rules around script run time
Script purging

[noahtalerman]

EDIT: The customer added multiple applications to the following script and found that at around 6 or 7 apps that take awhile to download, the scripts get stuck in "Pending":

$diskName = (get-location).Drive.Name;
    
    function ExecInstallation($wingetDirectory) {
        $tempFolderPath = "$($diskName):\Temp"
        if (-Not (Test-Path -Path $tempFolderPath)) {
            echo "Creating temporary repository - $tempFolderPath"
            New-Item -Path $tempFolderPath -ItemType Directory
        }

        cd $wingetDirectory
        echo "Running $wingetDirectory\winget.exe"
        
        
        cmd.exe /c "winget.exe install --disable-interactivity --silent --accept-package-agreements --accept-source-agreements --force SlackTechnologies.Slack"
    }

    Set-ExecutionPolicy -ExecutionPolicy Bypass -Force
    $expectedGlobalPath = "$($diskName):\Program Files\WindowsApps\Microsoft.DesktopAppInstaller_*_x64_*"

    echo $expectedGlobalPath

    if (Test-Path "$expectedGlobalPath\winget.exe") {
        $wingetDirectory = $expectedGlobalPath
        echo "Global winget directory found for - $wingetDirectory"
        ExecInstallation "$wingetDirectory"
    } else {
        echo "Global winget directory not found	"
        $userFolders = Get-ChildItem -Path "$($diskName):\Users" -Directory

        foreach ($userFolder in $userFolders) {
            $user = $userFolder.Name
            $wingetDirectory = "C:\Users\$user\AppData\Local\Microsoft\WindowsApps\Microsoft.DesktopAppInstaller*"

            if (Test-Path "$wingetDirectory\winget.exe") {
                echo "Winget directory found for user $user - $wingetDirectory"
                ExecInstallation "$wingetDirectory"
                break
            }
        }
    }

[JoStableford]

Related to a Slack conversation

[noahtalerman]

Hey @valentinpezon-primo! When you get the chance, can you please share the exact script y'all ran? (the one that includes the 6 or 7 apps)

This way, we can try to reproduce on our end and get a fix in.

[valentinpezon-primo]

Hi @noahtalerman , sure i can but Its not script related since it's not the same script that get blocked everytime, I will copy paste the scripts that get blocked so you have them anyway 👍

Also, i saw this small note on the fleet ui "Script is running or will run when the host comes online."
Maybe the issue is not related to the queue itself but related to the way you tell your script queue that the host is online ? since being online is the trigger to start the scripts in queue

Here are the scripts :

Script to re-enroll win device when enrollment fails :

$EnrollmentsPath = "HKLM:\SOFTWARE\Microsoft\Enrollments\"



$Enrollments = Get-ChildItem -Path $EnrollmentsPath



$DiscoveryServerFullUrls = @("https://aventa.mdm.getprimo.com/api/mdm/microsoft/discovery")



Foreach ($Enrollment in $Enrollments) {

    $EnrollmentObject = Get-ItemProperty Registry::$Enrollment

    if ($EnrollmentObject."DiscoveryServiceFullURL" -in $DiscoveryServerFullUrls ) {

        $EnrollmentPath = $EnrollmentsPath + $EnrollmentObject."PSChildName"

        Write-Host "Suppression de l'inscription : $EnrollmentPath"

        Remove-Item -Path $EnrollmentPath -Recurse

        Write-Host "Inscription supprimée. Réinscription de l'appareil..."

        C:\Windows\System32\deviceenroller.exe /c /AutoEnrollMDM

        Write-Host "L'appareil a été réinscrit."

    }

}

Script to install app (on a macos):

#!/bin/bash

curl -o Installomator.sh https://raw.githubusercontent.com/Installomator/Installomator/main/Installomator.sh
chmod +x Installomator.sh

sudo ./Installomator.sh microsoftoffice365 DEBUG=0 NOTIFY=silent BLOCKING_PROCESS_ACTION=silent_fail; sudo ./Installomator.sh microsoftteams DEBUG=0 NOTIFY=silent BLOCKING_PROCESS_ACTION=silent_fail; sudo ./Installomator.sh 1password8 DEBUG=0 NOTIFY=silent BLOCKING_PROCESS_ACTION=silent_fail
rm -rf Installomator.sh
    

Script that install multiple apps on windows :

    $diskName = (get-location).Drive.Name;
    
    function ExecInstallation($wingetDirectory) {
        $tempFolderPath = "$($diskName):\Temp"
        if (-Not (Test-Path -Path $tempFolderPath)) {
            echo "Creating temporary repository - $tempFolderPath"
            New-Item -Path $tempFolderPath -ItemType Directory
        }

        cd $wingetDirectory
        echo "Running $wingetDirectory\winget.exe"
        
        
        cmd.exe /c "winget.exe install --disable-interactivity --silent --accept-package-agreements --accept-source-agreements --force Microsoft.Office"; cmd.exe /c "winget.exe install --disable-interactivity --silent --accept-package-agreements --accept-source-agreements --force Microsoft.Teams"; cmd.exe /c "winget.exe install --disable-interactivity --silent --accept-package-agreements --accept-source-agreements --force 1Password.1Password"
    }

    Set-ExecutionPolicy -ExecutionPolicy Bypass -Force
    $expectedGlobalPath = "$($diskName):\Program Files\WindowsApps\Microsoft.DesktopAppInstaller_*_x64_*"

    echo $expectedGlobalPath

    if (Test-Path "$expectedGlobalPath\winget.exe") {
        $wingetDirectory = $expectedGlobalPath
        echo "Global winget directory found for - $wingetDirectory"
        ExecInstallation "$wingetDirectory"
    } else {
        echo "Global winget directory not found	"
        $userFolders = Get-ChildItem -Path "$($diskName):\Users" -Directory

        foreach ($userFolder in $userFolders) {
            $user = $userFolder.Name
            $wingetDirectory = "C:\Users\$user\AppData\Local\Microsoft\WindowsApps\Microsoft.DesktopAppInstaller*"

            if (Test-Path "$wingetDirectory\winget.exe") {
                echo "Winget directory found for user $user - $wingetDirectory"
                ExecInstallation "$wingetDirectory"
                break
            }
        }
    }

[noahtalerman]

@valentinpezon-primo thanks!

it's not the same script that get blocked everytime

Got it.

Also, i saw this small note on the fleet ui "Script is running or will run when the host comes online."
Maybe the issue is not related to the queue itself but related to the way you tell your script queue that the host is online ? since being online is the trigger to start the scripts in queue

Hmm, that's an interesting thought. For the hosts with scripts stuck in "Upcoming," are these hosts offline?

[noahtalerman]

cc @georgekarrv ^^

[valentinpezon-primo]

Hmm, that's an interesting thought. For the hosts with scripts stuck in "Upcoming," are these hosts offline?

No they come back online, at least the UI says they are online, my idea was that maybe the front "knows" but the script queue doesnt..

But I don't want to interfer with your debugging, It's just some random idea

[martinpannier]

Hey @noahtalerman, I think there may be 2 issues that are getting mixed up in one? There has been a lot of focus on scripts timing out. But we are seeing the issue on a fresh device, just enrolled, online — no failed scripts so far, scripts are just not running:

As a reminder, we rely on scripts to install applications.

[noahtalerman]

Thanks @martinpannier!

I think there may be 2 issues that are getting mixed up in one? There has been a lot of focus on scripts timing out. But we are seeing the issue on a fresh device, just enrolled, online — no failed scripts so far, scripts are just not running

This issue covers the "scripts are not just running" bit.

I updated the issue description to clarify this. Does that capture the issue y'all are seeing?

FYI @dantecatalfamo

[dantecatalfamo]

I'm looking into this, but haven't been able to recreate it yet

[martinpannier]

@noahtalerman Yes, perfect thanks!
@dantecatalfamo Happy to do a live call so you can see the problem first hand and run some tests

We would also love to have some mitigation steps if you guys have any idea (up to & including the dreaded "have you tried restarting your computer?")

[noahtalerman]

Happy to do a live call so you can see the problem first hand and run some tests

Hey @martinpannier! I think this is the plan for Tues (4/2) call.

We would also love to have some mitigation steps if you guys have any idea (up to & including the dreaded "have you tried restarting your computer?")

Currently, running these cleanup queries in the Fleet DB is one known workaround:

Delete pending scripts for a single host matching host_id X:

DELETE FROM host_script_results WHERE host_id = X AND exit_code IS NULL

Delete pending scripts for all hosts:

DELETE FROM host_script_results WHERE exit_code IS NULL

Obviously, this is a workaround that neither the IT admin nor the end user can take.

Taking this back to the Fleet team for ideas on workarounds for IT admin / end user.

[fleet-release]

No scripts stuck, stalled,
Fleet flows like river, unblocked.
Clear path for code's call.