Problems with changing "run as user" on Windows with 2.11 RC1 - Was: Check command 'powershell' does not exist. with new Agent: v2.11.0-rc1

[ChristianMoritz]

Describe the bug

If upgraded the Windows Agent on a Test VM to the new Agent v2.11.0rc1
But until now i didnt Upgrade the Icinga itself to the new RC1!!!!

After the Upgrade of the Agent all my Powershell Checks doesnt run any more.
I Only got the
"Unknown State"

for the Checks with the Output:
"Check command 'powershell' does not exist."

On all VMs which running the Agent v2.10,5 i doesnt got the Issue

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Icinga2: r2.10.5-1

• OS Ubuntu 16.04.6 LTS

• PHP: 7.0.33-0ubuntu0.16.04.5

• Icingaweb: 2.7.0

• Disabled features: compatlog debuglog elasticsearch gelf grafana influxdb livestatus opentsdb statusdata syslog

• Enabled features: api checker command graphite ido-mysql mainlog notification perfdata

• Config validation
  root@smon03:/# icinga2 daemon -C
  [2019-08-02 08:59:35 +0200] information/cli: Icinga application loader (version: r2.10.5-1)
  [2019-08-02 08:59:35 +0200] information/cli: Loading configuration file(s).
  [2019-08-02 08:59:35 +0200] information/ConfigItem: Committing config item(s).
  [2019-08-02 08:59:35 +0200] information/ApiListener: My API identity: smon03.intranet.stg
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 3136 Services.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 527 Hosts.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 1 FileLogger.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 6 NotificationCommands.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 3211 Notifications.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 15 HostGroups.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 1 ApiListener.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 10 Downtimes.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 1 GraphiteWriter.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 1 PerfdataWriter.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 13 Comments.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 182 Zones.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 1 ExternalCommandListener.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 180 Endpoints.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 2 ApiUsers.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 4 Users.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 235 CheckCommands.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 2 UserGroups.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 5 ServiceGroups.
  [2019-08-02 08:59:36 +0200] information/ConfigItem: Instantiated 5 TimePeriods.
  [2019-08-02 08:59:37 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
  [2019-08-02 08:59:37 +0200] information/cli: Finished validating the configuration file(s).

[dnsmichi]

Hi,

powershell seems like a custom CheckCommand of yours. Please look into the agent's log whether it fully received the configuration from the master. Best is to enable the debug log and trace it there, including the following check execution.

It may be the case that your zones.conf was modified during upgrade, and the agent denies to receive the synced zone configuration for global-templates or wherever the command is put into.

Also, please share the object powershell from the master. Best is icinga2 object list --type CheckCommand --name powershell.

Related note: The master/satellite should be upgraded to 2.11 first, is that the case in this scenario?

[dnsmichi]

Another thought: Did you modify the ITL command config files on the Windows agent by yourself? Such things are overridden on upgrades.

[dheitsc]

I just tested the same at my environment (same specs as ChristianMoritz, for this test not updated the master to 2.11 RC) and I had not any issues with the execution of my powershell commands. Agent was Windows Server 2016

[dnsmichi]

Cool, thanks for testing 👍 💪

[ChristianMoritz]

I think that first the clients have to be upgraded before you can upgrade the master because otherwise the error comes from the change of the certificates trust and the master would not trust the clients anymore and thus the clients as "unknown" or the like in the monitoring would.

My Powershell Checks Use the default "Check-command" powershell...

with the option ps_command:
& 'C:\Program Files\ICINGA2\sbin\check_something.ps1'

These Powershell Scripts lay on the VM insight the Agent Direcory
(f.e. C:\Program Files\ICINGA2\sbin\ )

here's the Output of:
icinga2 object list --type CheckCommand --name powershell

powershell
Object 'powershell' of type 'CheckCommand':
  % declared in '/var/lib/icinga2/api/packages/director/4d23d650-ba9e-4ac4-a986-                                                                                                                                                             2a17e3b35c57/zones.d/director-global/commands.conf', lines 97:1-97:32
  * __name = "powershell"
  * arguments
    % = modified in '/var/lib/icinga2/api/packages/director/4d23d650-ba9e-4ac4-a                                                                                                                                                             986-2a17e3b35c57/zones.d/director-global/commands.conf', lines 102:5-120:5
    * -args
      * order = 98
      * value = "$ps_args$"
    * -command
      * order = -1
      * required = true
      * skip_key = true
      * value = "$ps_command$"
    * -crit = "$ps_crit$"
    * -warn = "$ps_warn$"
    * ;exit
      * order = 99
      * skip_key = true
      * value = "$$LASTEXITCODE"
  * command = [ "C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe"                                                                                                                                                              ]
    % = modified in '/var/lib/icinga2/api/packages/director/4d23d650-ba9e-4ac4-a                                                                                                                                                             986-2a17e3b35c57/zones.d/director-global/commands.conf', lines 99:5-101:5
  * env = null
  * execute
    % = modified in 'methods-itl.conf', lines 36:3-36:23
    % = modified in 'methods-itl.conf', lines 36:3-36:23
    * arguments = [ "checkable", "cr", "resolvedMacros", "useResolvedMacros" ]
    * deprecated = false
    * name = "Internal#PluginCheck"
    * side_effect_free = false
    * type = "Function"
  * name = "powershell"
  * package = "director"
  * source_location
    * first_column = 1
    * first_line = 97
    * last_column = 32
    * last_line = 97
    * path = "/var/lib/icinga2/api/packages/director/4d23d650-ba9e-4ac4-a986-2a1                                                                                                                                                             7e3b35c57/zones.d/director-global/commands.conf"
  * templates = [ "powershell", "plugin-check-command", "plugin-check-command" ]
    % = modified in '/var/lib/icinga2/api/packages/director/4d23d650-ba9e-4ac4-a                                                                                                                                                             986-2a17e3b35c57/zones.d/director-global/commands.conf', lines 97:1-97:32
    % = modified in 'methods-itl.conf', lines 35:2-35:94
    % = modified in 'methods-itl.conf', lines 35:2-35:94
  * timeout = 60
  * type = "CheckCommand"
  * vars = null
  * zone = "director-global"
root@smon03:/home/administrator#

[ChristianMoritz]

here is the log entry...

[2019-07-26 21:40:58 +0200] warning/PluginCheckTask: Check command for object 'dummy.test.net' (PID: 0, arguments: 'C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe "& 'C:\Program Files\ICINGA2\sbin\pending-updates.ps1'" $LASTEXITCODE') terminated with exit code 127, output: Command C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe "& 'C:\Program Files\ICINGA2\sbin\pending-updates.ps1'" $LASTEXITCODE failed to execute: 5, "Access is denied."

seems like after the upgrade the agent is not allowed to run the check.

The Icinga2 Agent runs as Network Service insight the Windows Services, so it should be allow to run the Powershell.

[R-Sommer]

The docu for update-windows notes The Network Services Account which runs Icinga 2 by default does not have the required permissions to run this check.
I'd assume your powershell script runs into the same/similar issue and before the update of the Agent the Agent run with as different user e.g. LocalSystem.

[dnsmichi]

I think that first the clients have to be upgraded before you can upgrade the master because otherwise the error comes from the change of the certificates trust and the master would not trust the clients anymore and thus the clients as "unknown" or the like in the monitoring would.

Not sure if I can follow. The certificate cipher suite thingy has been fixed, you can apply the workaround inside the api with with the cipher_list attribute. That's not part of the problem here.

Likely it needed a manual intervention, and this now results that the agent received the synced command. Am I right about this step?

The following indicates that the agent now received the check command and executes checks, right?

here is the log entry...

[2019-07-26 21:40:58 +0200] warning/PluginCheckTask: Check command for object 'dummy.test.net' (PID: 0, arguments: 

Same as @R-Sommer already mentioned - the permissions for running the specific script are wrong, and presumingly you didn't specify to setup the "run service as user" during configuration routine.

Can you share how you're installing the Windows agent, either with a script of yours, or a screenshot of the Windows setup wizard?

Are there any errors logged in the Windows event console when executing the command?
What happens if you modify the running user for the service for icinga2?

[dnsmichi]

Hopefully this gets fixed with the Windows permissions on its own. For 2.11 and Icinga 2, I don't see a release blocking issue here. I'm leaving this open for further findings.

[ChristianMoritz]

the Agent install runs during the deployment of the VM's after the VM is properly deployed, domain joined and so on...

powershell.exe -executionpolicy bypass -command \\path to icinga\icinga_agent.ps1
And this is the Icinga_Agent.ps1 File

copy-item -Path \path to icinga\Icinga2Agent -Destination $PSHOME\Modules\ -Recurse
Start-Sleep -m 500
Import-Module Icinga2Agent
Start-Sleep -m 500
$icinga = Icinga2AgentModule -DirectorUrl 'https://smon03.mydomain.org/icingaweb2/director/'
-DirectorAuthToken '00000000000000000000000000000000' -InstallAgentVersion '2.10.5'
-DownloadUrl '\path to icinga' -ParentEndpoints 'smon03.mydomain.org'
-IgnoreSSLErrors -DebugMode
-RunInstaller
$icinga.install();

Get-ChildItem -Path \path to icinga\scripts\ -Recurse | Copy-Item ${env:ProgramFiles}\ICINGA2\sbin\ -Recurse
Start-Sleep -m 500
exit

@R-Sommer
as u told... after the upgrade of the Agent the Agent now was Started as "Network Service".
On all other VM's the Service is running as "Local System account", so ive switched the run as on my Test VM to the same "Local System account" and restarted the Icinga Agent... but still the same result.

[dnsmichi]

Can you share how you've switched the service user and how you've verified this?

Edit: Oh, and please attach pending-updates.ps1 in order to allow others reproduce your problem better.

[R-Sommer]

Maybe a silly question: Does your check had a run after the change of the service user?

There is still a bug in the powershell modul (can't find the according issue right now) which ignores the setting about the "Service User" in the director. Workaround: add this parameter:

-IcingaServiceUser 'LocalSystem'

[dnsmichi]

Tests

Config

Modify the local icinga2.conf and add the following content for checking something which needs higher elevated permissions. Also add an api user to use the debug console without web interface.

Powershell terminal as admin, vim is installed via chocolatey.

C:\ProgramData\icinga2\etc\icinga2> vim .\icinga2.conf

object Host "update" {
  check_command = "update-windows"
  check_interval = 10s
}

object ApiUser "root" {
  password = "icinga"

  permissions = [ "*" ]
}

C:\Program Files\ICINGA2\sbin> .\icinga2.exe daemon -C

C:\Program Files\ICINGA2\sbin> .\icinga2.exe api setup

C:\Program Files\ICINGA2\sbin> net stop icinga2
C:\Program Files\ICINGA2\sbin> net start icinga2

Network-Service User

C:\Program Files\ICINGA2\sbin> .\icinga2.exe console --connect 'https://root:icinga@localhost:5665/'
Icinga 2 (version: v2.11.0-rc1-1-g2b891fd1e)
Type $help to view available commands.
<1> => get_objects(Host)

<4> => get_host("update").last_check_result
{
        active = true
        check_source = "winmif.int.netways.de"
        command = [ "C:\\Program Files\\ICINGA2\\/sbin/check_update.exe" ]
        execution_end = 1565185763.873000
        execution_start = 1565185763.858000
        exit_status = 3.000000
        output = "Zugriff verweigert"
        performance_data = [ ]
        schedule_end = 1565185763.873000
        schedule_start = 1565185763.853000
        state = 3.000000
        ttl = 0.000000
        type = "CheckResult"
        vars_after = {
                attempt = 1.000000
                reachable = true
                state = 3.000000
                state_type = 1.000000
        }
        vars_before = {
                attempt = 1.000000
                reachable = true
                state = 3.000000
                state_type = 1.000000
        }
}
<5> => get_host("update").last_check_result.output
"Zugriff verweigert"

LocalSystem

Change Permissions

Well, Powershell is from hell.

This doesn't work.

$ServiceObject  = get-wmiobject -Class Win32_Service -filter "Name='icinga2'"

$ServiceObject.stopservice() | out-null

$ServiceObject.Change($null,$null,$null,$null,$null,'LocalSystem',$null,$null,$null,$null) | out-null

$ServiceObject.startservice()

This works.

C:\ProgramData\icinga2\etc\icinga2> $svc=Get-CimInstance win32_service -Filter 'Name="icinga2"'
>> $svc|Invoke-CimMethod -MethodName Change -Arguments @{StartName='LocalSystem'}

ReturnValue PSComputerName
----------- --------------
          0


C:\ProgramData\icinga2\etc\icinga2> $svc

ProcessId Name    StartMode State   Status ExitCode
--------- ----    --------- -----   ------ --------
4736      icinga2 Auto      Running OK     0

Verify it

C:\ProgramData\icinga2\etc\icinga2> net stop icinga2
The Icinga 2 service is stopping..
The Icinga 2 service was stopped successfully.

C:\ProgramData\icinga2\etc\icinga2> net start icinga2
The Icinga 2 service is starting.
The Icinga 2 service was started successfully.

<6> => get_host("update").last_check_result.output
"Zugriff verweigert"
<7> => DateTime(get_host("update").last_check).to_string()
"2019-08-07 16:00:49 +0200"
<8> => DateTime(get_host("update").last_check).to_string()
"2019-08-07 16:01:29 +0200"
<9> => get_host("update").last_check_result.output
"UPDATE OK 0 "
<10> =>

Conclusion

Works for me. Therefore I am closing this issue, thanks for testing and providing feedback 👍 Special thanks to @R-Sommer for his ongoing support.

Unfortunately I cannot help you with the Powershell module, that's @LordHepipud 's playground and not part of Icinga 2 itself. @lippserd Yet another issue which did cost me quite some hours.

If you cannot find the issue with the service user, please create a new one in the Powershell module's repo. I'd say that doesn't hurt.