disk-windows incorrect plugin output #5414

ghost · 2017-07-14T10:03:57Z

Expected Behavior

Seeing a correct check_disk.exe output in /icignaweb2.

Current Behavior

The output of disk-windows in /icingaweb2 seems to be cut off after the ::

Possible Solution

The /icingaweb2 should print/parse the correct full output as shown when it's called from the PowerShell directly:

Steps to Reproduce (for bugs)

This one is hard. I installed and removed the server and the clients several times, also did many config edits. So, unfortunately I don't know what exactly causes this error.

Your Environment

Version used (icinga2 --version):

# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.6.3-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Ubuntu
  Platform version: 17.04 (Zesty Zapus)
  Kernel: Linux
  Kernel version: 4.10.0-26-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: a295cf04d827

# apt policy icinga2
icinga2:
  Installiert:           2.6.3-1.zesty
  Installationskandidat: 2.6.3-1.zesty
  Versionstabelle:
 *** 2.6.3-1.zesty 500
        500 http://packages.icinga.org/ubuntu icinga-zesty/main amd64 Packages
        100 /var/lib/dpkg/status
     2.6.0-2 500
        500 http://de.archive.ubuntu.com/ubuntu zesty/universe amd64 Packages

# apt policy icingaweb2
icingaweb2:
  Installiert:           2.4.1-1.zesty
  Installationskandidat: 2.4.1-1.zesty
  Versionstabelle:
 *** 2.4.1-1.zesty 500
        500 http://packages.icinga.org/ubuntu icinga-zesty/main amd64 Packages
        500 http://packages.icinga.org/ubuntu icinga-zesty/main i386 Packages
        100 /var/lib/dpkg/status
     2.4.1-1 500
        500 http://de.archive.ubuntu.com/ubuntu zesty/universe amd64 Packages
        500 http://de.archive.ubuntu.com/ubuntu zesty/universe i386 Packages

Operating System and version:
Enabled features (icinga2 feature list):

# icinga2 feature list
Disabled features: compatlog debuglog gelf graphite influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification

Icinga Web 2 version and modules (System - About):
Config validation (icinga2 daemon -C):

# icinga2 daemon -C
information/cli: Icinga application loader (version: r2.6.3-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: Icinga2
warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
information/ConfigItem: Instantiated 1 ApiUser.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 4 Zones.
information/ConfigItem: Instantiated 1 FileLogger.
information/ConfigItem: Instantiated 4 Endpoints.
information/ConfigItem: Instantiated 2 NotificationCommands.
information/ConfigItem: Instantiated 36 Notifications.
information/ConfigItem: Instantiated 208 CheckCommands.
information/ConfigItem: Instantiated 1 Downtime.
information/ConfigItem: Instantiated 4 Hosts.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 2 HostGroups.
information/ConfigItem: Instantiated 1 UserGroup.
information/ConfigItem: Instantiated 2 Users.
information/ConfigItem: Instantiated 32 Services.
information/ConfigItem: Instantiated 3 TimePeriods.
information/ConfigItem: Instantiated 3 ServiceGroups.
information/ConfigItem: Instantiated 1 ScheduledDowntime.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

# icinga2 object list --type Endpoint
Object 'SERVER03.DOMAIN.LOCAL' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 21:1-21:42
  * __name = "SERVER03.DOMAIN.LOCAL"
  * host = "192.168.1.11"
    % = modified in '/etc/icinga2/zones.conf', lines 22:3-22:23
  * log_duration = 86400
  * name = "SERVER03.DOMAIN.LOCAL"
  * package = "_etc"
  * port = "5665"
  * templates = [ "SERVER03.DOMAIN.LOCAL" ]
    % = modified in '/etc/icinga2/zones.conf', lines 21:1-21:42
  * type = "Endpoint"
  * zone = ""

Object 'SERVER05.DOMAIN.LOCAL' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 13:1-13:42
  * __name = "SERVER05.DOMAIN.LOCAL"
  * host = "192.168.1.13"
    % = modified in '/etc/icinga2/zones.conf', lines 14:3-14:23
  * log_duration = 86400
  * name = "SERVER05.DOMAIN.LOCAL"
  * package = "_etc"
  * port = "5665"
  * templates = [ "SERVER05.DOMAIN.LOCAL" ]
    % = modified in '/etc/icinga2/zones.conf', lines 13:1-13:42
  * type = "Endpoint"
  * zone = ""
<
Object 'server07.DOMAIN.LOCAL' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 29:1-29:42
  * __name = "server07.DOMAIN.LOCAL"
  * host = "192.168.1.9"
    % = modified in '/etc/icinga2/zones.conf', lines 30:3-30:22
  * log_duration = 86400
  * name = "server07.DOMAIN.LOCAL"
  * package = "_etc"
  * port = "5665"
  * templates = [ "server07.DOMAIN.LOCAL" ]
    % = modified in '/etc/icinga2/zones.conf', lines 29:1-29:42
  * type = "Endpoint"
  * zone = ""

# icinga2 object list --type Zone
Object 'SERVER03.DOMAIN.LOCAL' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 24:1-24:38
  * __name = "SERVER03.DOMAIN.LOCAL"
  * endpoints = [ "SERVER03.DOMAIN.LOCAL" ]
    % = modified in '/etc/icinga2/zones.conf', lines 25:3-25:44
  * global = false
  * name = "SERVER03.DOMAIN.LOCAL"
  * package = "_etc"
  * parent = "Icinga2"
    % = modified in '/etc/icinga2/zones.conf', lines 26:3-26:19
  * templates = [ "SERVER03.DOMAIN.LOCAL" ]
    % = modified in '/etc/icinga2/zones.conf', lines 24:1-24:38
  * type = "Zone"
  * zone = ""

Object 'SERVER05.DOMAIN.LOCAL' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 16:1-16:38
  * __name = "SERVER05.DOMAIN.LOCAL"
  * endpoints = [ "SERVER05.DOMAIN.LOCAL" ]
    % = modified in '/etc/icinga2/zones.conf', lines 17:3-17:44
  * global = false
  * name = "SERVER05.DOMAIN.LOCAL"
  * package = "_etc"
  * parent = "Icinga2"
    % = modified in '/etc/icinga2/zones.conf', lines 18:3-18:19
  * templates = [ "SERVER05.DOMAIN.LOCAL" ]
    % = modified in '/etc/icinga2/zones.conf', lines 16:1-16:38
  * type = "Zone"
  * zone = ""

Object 'server07.DOMAIN.LOCAL' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 32:1-32:38
  * __name = "server07.DOMAIN.LOCAL"
  * endpoints = [ "server07.DOMAIN.LOCAL" ]
    % = modified in '/etc/icinga2/zones.conf', lines 33:3-33:44
  * global = false
  * name = "server07.DOMAIN.LOCAL"
  * package = "_etc"
  * parent = "Icinga2"
    % = modified in '/etc/icinga2/zones.conf', lines 34:3-34:19
  * templates = [ "server07.DOMAIN.LOCAL" ]
    % = modified in '/etc/icinga2/zones.conf', lines 32:1-32:38
  * type = "Zone"
  * zone = ""

See also https://monitoring-portal.org/index.php?thread/41288-disk-windows-not-working-in-icingaweb2/&postID=252467#post252467.
One related bug I could find is this one: #3159

The text was updated successfully, but these errors were encountered:

Crunsher · 2017-07-14T14:35:07Z

The plugin output looks broken, after the percentage there should be a | to mark the start of the perfdata string. I'll have to check that

mcktr · 2017-07-16T10:06:28Z

@Crunsher are you sure that the output is broken? To me the output looks ok. Also I tried to reproduce this issue but I did not make it.

Output of the check_disk.exe directly from CLI

Icinga Web 2 output

Icinga 2 Master:

icinga2 - The Icinga 2 network monitoring daemon (version: r2.6.3-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 3.16.0-4-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.9.2
  Build host: smithers

Icinga 2 Windows Agent:

icinga2.exe - The Icinga 2 network monitoring daemon (version: v2.6.3-2-g9d26a9c)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: C:\Program Files\ICINGA2\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory: C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var
  Package data directory: C:\Program Files\ICINGA2\\share\icinga2
  State path: C:\ProgramData\icinga2\var/lib/icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var/lib/icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var/cache/icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var/cache/icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run/icinga2/icinga2.pid

System information:
  Platform: Windows
  Platform version: 8
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 18.0.40629.0
  Build host: ICINGA-BUILD-WI

Stefar77 · 2017-07-16T12:30:03Z

I'm using nscp-local-disk for remote agent disk status and didn't look any further because that worked fine.

I tried using check_disk.exe but the threshold seems to be only for the total free space on the machine or something weird like that, if you have a 100T partition and your 50GB C:\ is full you will get no alert or for whatever reason it decides to give CRITICAL on everything.
I remember it just didn't work as expected and I either got all alerts or no alerts from it.

edit:
I was used to putting in values the other way around (Nagios/NSClient++), it does work better when I use the right values for thresholds.

Going to test this plugin on hosts where all disks can use the same thresholds but it seems to work fine here.

DISK OK - free space:Total 94378MB (62%): C:\ 10104 MB (20%): D:\ 3852 MB (38%): E:\ 10175 MB (99%): F:\ 10058 MB (98%): G:\ 49364 MB (96%): H:\ 10825 MB (53%):

I think your next action is set logging a level higher and try and fetch a log message from the agent pushing the output and check if it gets mangled before it enters the API or after.
** also check the result in the API, maybe it's only the GUI not showing **
curl -k -s -u api_user:api_pass -H 'Accept: application/json' -X GET 'https://localhost:5665/v1/objects/services/{host_name}\!{service_name}?attrs=last_check_result'

..."output":"DISK OK - free space:Total 75289MB (74%); C:\ 25619 MB (50%); D:\ 49670 MB (97%); ","performance_data":["C:\=25619MB;5084.8000000000002;2542.4000000000001;0;50848","D:\=49670MB;5119.7000000000007;2559.8500000000004;0;51197"],...

ghost · 2017-07-17T12:53:51Z

I'm not sure if it's bug or a configuration issue. I'm pretty sure it used to work with these same versions of Icinga2 and Icingaweb2, so I'd say it's a configuration issue.

Crunsher · 2017-07-18T08:00:05Z

@mcktr I failed to parse the output correctly myself 🤕

I'm going to close this for now, since it seems not reproducible/a temporary config error. If you run into this issue again, open a new ticket and reference this one.

ghost · 2017-07-18T11:40:26Z

On this Windows server there are only one native drive: C:, S: and B: are network drives, maybe that has something to do with it, because on another Windows server where there are more than one native drive, the output is correct and lists all the native drives.
Now I'm not sure, whether the listed drives were not native drive all along and the network shares never worked in the first place.

I just deinstalled Icinga2, deleted C:\ProgramData\icinga2 and reinstalled the Windows client again, but still the same error. So, I'm not sure if this is still because of the Windows client' faulty config. I'm also not sure if this is the Icinga2 server' faulty config then either because I installed the it on a completely new VM, but since S: and B: are network shares, this could be the reason that Icinga2 process can't access them.

Ok. So, if I wanted to add the S: and B: shares then, I could paste the output of > .\check_disks.exe (see that PowerShell image in my initial post) into a txt file (a PowerShell script run by Administrator and independently of Icinga2) and read that by Icinga2' PowerShell object CheckCommand. That's a workaround but would work (I tested it already: since the output string is the same and it comes from check_disks.exe, Icingaweb2 can parse and display it correctly).

Closing this ticket is ok.

mcktr · 2017-07-18T16:27:39Z

I doubt that the check_disk.exe can handle network drives. AFAIK the network drives are mapped within the user context, so if you map B: as network drive with user1, the user2 has not automatically access to it. The Icinga 2 windows service runs with the default settings under the NetworkService Account and if you log in to your windows machine with the local administrator account and map a network drive the NetworkService account can't see this mapped network drive cause it is not in his context.

Also I don't know if the function GetDiskFreeSpaceEx that is used within the check plugin supports network drives.

An other solution to yours is to use the Linux check disk_smb but this requires that your master or a Linux satellite has access to the SMB share.

ghost · 2017-07-19T07:09:23Z

I changed Icinga2 service to run as Local System but still same.

AFAIK check_disk.exe uses GetDiskFreeSpaceEx and calling check_disk.exe directly from PowerShell (see first post) shows all drives including network shares (B: and S: are network shares). So I think it might be a rights issue.

Crunsher · 2017-07-19T07:29:48Z

Network drives should™️ work, if you run it with a user that has the appropriate permissions. Windows permissions being an issue is a known problem (ie check_update)

ghost · 2017-07-19T07:33:12Z

Ok. Fortunately check_update.exe works when Icinga2 is changed to Local System, but unfortunately check_disk.exe still does not. Running > .\check_disk.exe as Administrator prints all the drives though (see the PowerShell image in first post), so there's hope. services.msc offers only two possibilities, NetworkService and Local System. Maybe Administrator would work, if only that would be there.

Crunsher · 2017-07-19T07:38:12Z

Network drives are mounted per user afaik, so you'd need to have them mounted for the user icinga2 runs as to see them. Not sure if the system user even sees them...

mcktr · 2017-07-19T07:40:32Z

I googled a bit and found a thread.

Persistant drive mappings are only restored during an interactive login, which the service does not use. I believe the only way to get a service to use a network drive is for that service to map the drive itself or alternatively for it to us a UNC path instead of a mapped drive."

Possible solution, but it's a dirty one :-/

Do like this with the psexec utility:
Open an elevated cmd.exe prompt (Run as administrator). Then, do psexec -i -s cmd.exe. You are now "nt authority\system" :) Now, all you have to do is net use z: \servername\sharedfolder /persistent:yes
Keep in mind if you need to delete the mapped drive you will have to do it in the same way (instead doing net use z: /del)."

https://serverfault.com/questions/4623/windows-can-i-map-a-network-drive-for-a-service-account

ghost · 2017-07-19T08:24:17Z

Thanks for your suggestions.

Stefar77 · 2017-07-19T16:48:02Z

@passboltUser Why not just put a check on the fileserver for the drives?

Having it on client(s) could make it check the same volume multiple times, your fileserver is running out of space and you will get 30 warnings from 30 different hosts...

if (type == DRIVE_FIXED || type == DRIVE_REMOTE) { [It was intended to work on network drives but it should be removed]

Also you can poll your shares via;

# ./check_disk_smb -H {address} -s {share} -u {domain}\\{user} -p {password}
WARNING: Only 3.22G (10%) free on \\HOSTNAME\SHARE | 'SHARE'=28746186752B;27377296998.4;30598155468.8;0;32208584704

Footur · 2018-02-20T08:59:36Z

I solved this problem with the custom variable disk_win_show_used.

apply Service "Icinga - Disk" {
    import "Master Template"

    command_endpoint = host_name
    check_command = "disk-windows"

    vars.disk_win_show_used = "true"

    assign where host.vars.os == "Windows"
}

Crunsher · 2018-02-20T09:11:45Z

That can't be, show_used is only used in the computation and has nothing to do with how the disks are enumerated 😕

ghost mentioned this issue Jul 14, 2017

[dev.icinga.com #9664] Wrong value handle form check_disk to Icinga2 #3159

Closed

Crunsher self-assigned this Jul 14, 2017

Crunsher added bug Something isn't working Plugins area/windows Windows agent and plugins labels Jul 14, 2017

Crunsher changed the title ~~disk-windows not working in icingaweb2~~ disk-windows incorrect plugin output Jul 14, 2017

Crunsher closed this as completed Jul 18, 2017

ghost mentioned this issue Jul 18, 2017

disk-windows not working in icingaweb2 Icinga/icingaweb2#2889

Closed

dnsmichi unassigned Crunsher Sep 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

disk-windows incorrect plugin output #5414

disk-windows incorrect plugin output #5414

ghost commented Jul 14, 2017 •

edited by ghost

Crunsher commented Jul 14, 2017

mcktr commented Jul 16, 2017

Stefar77 commented Jul 16, 2017 •

edited

ghost commented Jul 17, 2017

Crunsher commented Jul 18, 2017

ghost commented Jul 18, 2017 •

edited by ghost

mcktr commented Jul 18, 2017

ghost commented Jul 19, 2017

Crunsher commented Jul 19, 2017

ghost commented Jul 19, 2017 •

edited by ghost

Crunsher commented Jul 19, 2017

mcktr commented Jul 19, 2017 •

edited

ghost commented Jul 19, 2017

Stefar77 commented Jul 19, 2017

Footur commented Feb 20, 2018

Crunsher commented Feb 20, 2018

disk-windows incorrect plugin output #5414

disk-windows incorrect plugin output #5414

Comments

ghost commented Jul 14, 2017 • edited by ghost

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Your Environment

Crunsher commented Jul 14, 2017

mcktr commented Jul 16, 2017

Stefar77 commented Jul 16, 2017 • edited

ghost commented Jul 17, 2017

Crunsher commented Jul 18, 2017

ghost commented Jul 18, 2017 • edited by ghost

mcktr commented Jul 18, 2017

ghost commented Jul 19, 2017

Crunsher commented Jul 19, 2017

ghost commented Jul 19, 2017 • edited by ghost

Crunsher commented Jul 19, 2017

mcktr commented Jul 19, 2017 • edited

ghost commented Jul 19, 2017

Stefar77 commented Jul 19, 2017

Footur commented Feb 20, 2018

Crunsher commented Feb 20, 2018

ghost commented Jul 14, 2017 •

edited by ghost

Stefar77 commented Jul 16, 2017 •

edited

ghost commented Jul 18, 2017 •

edited by ghost

ghost commented Jul 19, 2017 •

edited by ghost

mcktr commented Jul 19, 2017 •

edited