Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TopBeat 1.2.3.] Can't access PID, even with correct rights. #1897

Closed
csuka opened this issue Jun 22, 2016 · 23 comments

Comments

Projects
None yet
6 participants
@csuka
Copy link

commented Jun 22, 2016

Original thread here: https://discuss.elastic.co/t/server-2012-r2-some-metrics-not-send/51403

  • Version: TopBeat 1.2.3
  • Operating System: Windows Server 2012 R2 Standard (VMware host)
  • Steps to Reproduce:
  • Installed TopBeat via Powershell with Administrator Rights (right click, run as admin).
  • In processes, it is seen that TopBeat is owned by SYSTEM. Topbeat is also executed as SYSTEM.
  • The error in topbeat.log with debug level:

2016-05-31T11:00:13+02:00 DBG Skip process pid=0: error getting process state for pid=0: OpenProcess fails with The parameter is incorrect.
2016-05-31T11:00:13+02:00 DBG Skip process pid=4: error getting process state for pid=4: OpenProcess fails with Access is denied.
2016-05-31T11:00:13+02:00 DBG Windows is interactive: false

For convience, the full image:
lpv5ave

@tsg tsg added bug Topbeat labels Jun 27, 2016

@asednev

This comment has been minimized.

Copy link

commented Oct 5, 2016

This issue also exists in Topbeat 1.3.1

@tsg

This comment has been minimized.

Copy link
Collaborator

commented Oct 5, 2016

It would be useful to know what processes are not captured. The low PIDs numbers make me think they are system processes.

@asednev To clarify, you get the error for some of the processes but most of the processes are captured?

@asednev

This comment has been minimized.

Copy link

commented Oct 5, 2016

In my case none of the processes were captured.

@tsg

This comment has been minimized.

Copy link
Collaborator

commented Oct 5, 2016

What Windows version do you have? You start the Powershell session with "Run as Administrator", right?

@asednev

This comment has been minimized.

Copy link

commented Oct 5, 2016

Windows Server 2012 R2 version 6.3 (build 9600)

btw, I discovered that older version Topbeat 1.2.3 works well on the same VM.

@asednev

This comment has been minimized.

Copy link

commented Oct 5, 2016

Topbeat 1.3.1 I ran as windows service running under "Local System" account and also from command prompt "Run As Administrator". Neither worked.

Topbeat 1.2.3 mysteriously works.

@ruflin ruflin added the :Windows label Oct 5, 2016

@andrewkroh

This comment has been minimized.

Copy link
Member

commented Oct 12, 2016

To help debug the issue, could you use SysInternals - Process Explorer to examine the permissions of the running process.

I suspect the interesting privilege might be the SeDebugPrivilege.

topbeat-1 3 1_as_service

topbeat-1 3 1_as_user

This security policy could be related. https://technet.microsoft.com/en-us/library/cc976527.aspx?f=255&MSPPError=-2147217396

@andrewkroh

This comment has been minimized.

Copy link
Member

commented Oct 12, 2016

There is one place in gosigar where it requests PROCESS_ALL_ACCESS permission but the operation that it performs, GetProcessMemoryInfo, requires only PROCESS_QUERY_INFORMATION and PROCESS_VM_READ. gosigar should only request the permissions that it requires. Not sure if this would address the problem at hand, but in any case it's something that should be changed.

In other parts of gosigar it requests PROCESS_QUERY_INFORMATION, but in Windows Vista and newer there is a more limited privilege we can request called PROCESS_QUERY_LIMITED_INFORMATION. This can be used for GetProcessTimes, GetProcessImageFileName, and GetExitCodeProcess. It cannot be used for OpenProcessToken.

@andrewkroh andrewkroh self-assigned this Oct 12, 2016

@andrewkroh

This comment has been minimized.

Copy link
Member

commented Oct 13, 2016

I haven't figured out what OS settings are causing the issue for these users. Some feedback is required (see my earlier request for SysInternals data).

I have made the changes mentioned in my previous comment. See elastic/gosigar#50. We can put together a build of Topbeat/Metricbeat with these changes and see if the issue persists.

@monicasarbu

This comment has been minimized.

Copy link
Contributor

commented Oct 13, 2016

Closing for now. @csuka Feel free to re-open the issue if the error persists.

@andrewkroh

This comment has been minimized.

Copy link
Member

commented Oct 17, 2016

@csuka @asednev Would you be able to test the snapshot build of Metricbeat containing my changes on a machine that was having the "Access is denied" problem for all processes. BTW Metricbeat is replacing Topbeat in 5.0.

Access is denied is expected for some system processes, specifically the csrss.exe processes. And "incorrect parameter" is expected for PID 0. From the OpenProcess docs:

If the specified process is the System Process (0x00000000), the function fails and the last error code is ERROR_INVALID_PARAMETER. If the specified process is the Idle process or one of the CSRSS processes, this function fails and the last error code is ERROR_ACCESS_DENIED because their access restrictions prevent user-level code from opening them.

@csuka

This comment has been minimized.

Copy link
Author

commented Oct 18, 2016

@andrewkroh Destroyed the VM on which it was failing.
Will check if I can run it somewhere else, will take a while.

@asednev

This comment has been minimized.

Copy link

commented Oct 26, 2016

@andrewkroh, I tried metricbeat-6.0.0-alpha1-SNAPSHOT-windows-x86_64 and it still threw same errors:

2016/10/26 16:44:33.521779 helper.go:276: ERR Error getting process details pid=91076: error getting process mem for pid=91076: OpenProcess
fails with Access is denied.

2016/10/26 16:44:33.527793 helper.go:276: ERR Error getting process details pid=702776: error getting process mem for pid=702776: OpenProces
s fails with Access is denied.

I see system metrics logged to Elasticsearch but none of the process metrics.

@andrewkroh

This comment has been minimized.

Copy link
Member

commented Oct 26, 2016

@asednev Thanks for testing. Can you check in the process explorer what the names of those processes are? Are you seeing any events in Elasticsearch where metricset.name:process?

@asednev

This comment has been minimized.

Copy link

commented Oct 26, 2016

I don't see any events with metricset.name:process in Elasticsearch. Only cpu, memory, filesystem, and network.

Actually the list of PID in the output was rather long. I only copied 2 as an example. I went through the list of PID, these are the processes that I'm looking to monitor - our application services.

@andrewkroh

This comment has been minimized.

Copy link
Member

commented Oct 26, 2016

Sounds like the changes I made didn't help. In the metricbeat download you used, can you tell me what the commit has is inside the .build_hash.txt file. Just want to confirm that the version has my changes.

Are you aware of any local or group security policies that were tightened on this machine? The reason I ask is that it would be nice to be able to reproduce this locally so I don't have to take up your valuable time.

Could you run Sysinternals - ProcExp (like I did in #1897 (comment)) and check the privileges of Metricbeat?

There is one other change that I think could help. It basically involves escalating the Metricbeat processes access token to include the SeDebugPrivilege, like what is described in https://stackoverflow.com/questions/2932461/windows-vista-win7-privilege-problem-sedebugprivilege-openprocess.

@asednev

This comment has been minimized.

Copy link

commented Oct 26, 2016

I compared the list of privileges between machine where Metricbeat can get list of processes and machine where Metricbeat can't get the list of processes. The list of privilages is the same. Also, I will reiterate that Topbeat 1.2.3 can get process-level metrics on both of these machines.

image
image
image

@asednev

This comment has been minimized.

Copy link

commented Oct 26, 2016

Also, it's worth mentioning that machine where Metricbeat can't get process-level metrics is production machine that was hardened by InfoSec team. I'm not entirely sure what exact steps go into hardening.

@andrewkroh

This comment has been minimized.

Copy link
Member

commented Oct 26, 2016

@asednev I added code locally to metricbeat that tries to add SeDebugPrivilege to the process' access token. It's just test code at the moment and should print "AdjustTokenPrivileges success" to stdout at start. This version also has a bit more info in the error messages that will help identify the call that failed due to "Access is denied".

https://s3.amazonaws.com/beats-files/metricbeat.exe

If you are willing to try this binary and report the output this would be very helpful. Thanks

@asednev

This comment has been minimized.

Copy link

commented Oct 26, 2016

Voila, that fixed the problem. SeDebugPrivilege is Enabled in ProcessExplorer and I can see process-level metrics in Elasticsearch.

@andrewkroh

This comment has been minimized.

Copy link
Member

commented Oct 26, 2016

Awesome, thanks for the help. I will work on cleaning up my code and put out a PR to address the problem. I'll post here once the changes are available in the regular snapshot build.

@asednev

This comment has been minimized.

Copy link

commented Oct 26, 2016

Thank you @andrewkroh. I'm glad you were able to come out of this with a fix.

andrewkroh added a commit to andrewkroh/gosigar that referenced this issue Oct 27, 2016

Update gosigar windows code to use gosigar/sys/windows package
- Fixes elastic#53 (ProcStatus.PPID value is wrong on Windows). The code was incorrectly using `CreateToolhelp32Snapshot` + `Process32First`. This could have been fixed, but it would have required iterating over each process until finding the PPID of process we were interested it. Instead the code has been changed to use `NtQueryInformationProcess` to get the PPID given a process token.
- Fixes elastic#6 (Get the cpu usage per core in Windows). I used `NtQuerySystemInformation` to collect the timing information on a per CPU basis.
- Adds OS version checks to the functions that make certain WMI calls. The `Win32_Process` data is only available on Vista and newer. On XP and Win2003, these methods will return `ErrNotImplemented`. This will help address elastic/beats#1704.
- Implements `Uptime.Get` for Windows.
- Implements `Swap.Get` for Windows based on page file metrics.
- Removes cgo usage for Windows.
- Adds support to `github.com/gosigar/sys/windows` for querying and enabling privileges in a process token. This will help in addressing elastic/beats#1897.

andrewkroh added a commit to andrewkroh/gosigar that referenced this issue Oct 27, 2016

Update gosigar windows code to use gosigar/sys/windows package
- Fixes elastic#53 (ProcStatus.PPID value is wrong on Windows). The code was incorrectly using `CreateToolhelp32Snapshot` + `Process32First`. This could have been fixed, but it would have required iterating over each process until finding the PPID of process we were interested it. Instead the code has been changed to use `NtQueryInformationProcess` to get the PPID given a process token.
- Fixes elastic#6 (Get the cpu usage per core in Windows). I used `NtQuerySystemInformation` to collect the timing information on a per CPU basis.
- Adds OS version checks to the functions that make certain WMI calls. The `Win32_Process` data is only available on Vista and newer. On XP and Win2003, these methods will return `ErrNotImplemented`. This will help address elastic/beats#1704.
- Implements `Uptime.Get` for Windows.
- Implements `Swap.Get` for Windows based on page file metrics.
- Removes cgo usage for Windows.
- Adds support to `github.com/gosigar/sys/windows` for querying and enabling privileges in a process token. This will help in addressing elastic/beats#1897.

andrewkroh added a commit to andrewkroh/gosigar that referenced this issue Oct 27, 2016

Update gosigar windows code to use gosigar/sys/windows package
- Fixes elastic#53 (ProcStatus.PPID value is wrong on Windows). The code was incorrectly using `CreateToolhelp32Snapshot` + `Process32First`. This could have been fixed, but it would have required iterating over each process until finding the PPID of process we were interested it. Instead the code has been changed to use `NtQueryInformationProcess` to get the PPID given a process token.
- Fixes elastic#6 (Get the cpu usage per core in Windows). I used `NtQuerySystemInformation` to collect the timing information on a per CPU basis.
- Adds OS version checks to the functions that make certain WMI calls. The `Win32_Process` data is only available on Vista and newer. On XP and Win2003, these methods will return `ErrNotImplemented`. This will help address elastic/beats#1704.
- Implements `Uptime.Get` for Windows.
- Implements `Swap.Get` for Windows based on page file metrics.
- Removes cgo usage for Windows.
- Adds support to `github.com/gosigar/sys/windows` for querying and enabling privileges in a process token. This will help in addressing elastic/beats#1897.

ruflin added a commit to elastic/gosigar that referenced this issue Oct 28, 2016

Update gosigar windows code to use gosigar/sys/windows package (#55)
- Fixes #53 (ProcStatus.PPID value is wrong on Windows). The code was incorrectly using `CreateToolhelp32Snapshot` + `Process32First`. This could have been fixed, but it would have required iterating over each process until finding the PPID of process we were interested it. Instead the code has been changed to use `NtQueryInformationProcess` to get the PPID given a process token.
- Fixes #6 (Get the cpu usage per core in Windows). I used `NtQuerySystemInformation` to collect the timing information on a per CPU basis.
- Adds OS version checks to the functions that make certain WMI calls. The `Win32_Process` data is only available on Vista and newer. On XP and Win2003, these methods will return `ErrNotImplemented`. This will help address elastic/beats#1704.
- Implements `Uptime.Get` for Windows.
- Implements `Swap.Get` for Windows based on page file metrics.
- Removes cgo usage for Windows.
- Adds support to `github.com/gosigar/sys/windows` for querying and enabling privileges in a process token. This will help in addressing elastic/beats#1897.

@andrewkroh andrewkroh reopened this Oct 29, 2016

andrewkroh added a commit to andrewkroh/beats that referenced this issue Oct 29, 2016

Metricbeat system module updates for Windows
- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.

andrewkroh added a commit to andrewkroh/beats that referenced this issue Oct 29, 2016

Metricbeat system module updates for Windows
- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.

andrewkroh added a commit to andrewkroh/beats that referenced this issue Oct 29, 2016

Metricbeat system module updates for Windows
- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.

andrewkroh added a commit to andrewkroh/beats that referenced this issue Oct 29, 2016

Metricbeat system module updates for Windows
- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes elastic#2885 (diskio metricset fails on XP and 2003).

andrewkroh added a commit to andrewkroh/beats that referenced this issue Oct 29, 2016

Metricbeat system module updates for Windows
- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes elastic#2885 (diskio metricset fails on XP and 2003).

andrewkroh added a commit to andrewkroh/beats that referenced this issue Oct 31, 2016

Metricbeat system module updates for Windows
- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes elastic#2885 (diskio metricset fails on XP and 2003).

andrewkroh added a commit to andrewkroh/beats that referenced this issue Oct 31, 2016

Metricbeat system module updates for Windows
- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes elastic#2885 (diskio metricset fails on XP and 2003).
- Enabled fsstats by default in Metricbeat config.

andrewkroh added a commit to andrewkroh/beats that referenced this issue Oct 31, 2016

Metricbeat system module updates for Windows
- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes elastic#2885 (diskio metricset fails on XP and 2003).
- Enabled fsstats by default in Metricbeat config.

@ruflin ruflin closed this in #2884 Oct 31, 2016

ruflin added a commit that referenced this issue Oct 31, 2016

Metricbeat system module updates for Windows
- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes #2860 (PPID is zero on Windows).
- Fixes #1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes #1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes #2885 (diskio metricset fails on XP and 2003).
- Enabled fsstats by default in Metricbeat config.
@andrewkroh

This comment has been minimized.

Copy link
Member

commented Oct 31, 2016

The snapshot build now has the fix for this if you'd like to test it out. https://beats-nightlies.s3.amazonaws.com/index.html?prefix=winlogbeat/

If you look in the log output there will be some lines containing "SeDebugPrivilege" that provide some feedback. In your case, @asednev, you should see "SeDebugPrivilege is now enabled.".

andrewkroh added a commit to andrewkroh/beats that referenced this issue Nov 1, 2016

Metricbeat system module updates for Windows
- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes elastic#2885 (diskio metricset fails on XP and 2003).
- Enabled fsstats by default in Metricbeat config.

(cherry picked from commit eb88c2c)

ruflin added a commit that referenced this issue Nov 3, 2016

Metricbeat system module updates for Windows
- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes #2860 (PPID is zero on Windows).
- Fixes #1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes #1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes #2885 (diskio metricset fails on XP and 2003).
- Enabled fsstats by default in Metricbeat config.

(cherry picked from commit eb88c2c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.