Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process.cmdline() wrong results, parameters not correctly splitted #1179

Closed
bitranox opened this issue Nov 21, 2017 · 38 comments
Closed

Process.cmdline() wrong results, parameters not correctly splitted #1179

bitranox opened this issue Nov 21, 2017 · 38 comments

Comments

@bitranox
Copy link

@bitranox bitranox commented Nov 21, 2017

psutil version 5.4.1
python 3.6.1
Linux Kernel 4.13.0.16
inside an LXC container (if that matters)

proc=psutil.Process(xxx)   # some PID Number
proc.cmdline()    # does not give correctly splitted results 

for instance:

# WRONG RESULT
['/opt/python3/bin/python3 /opt/rotek-apps/bin/rpyc_server/rpyc_server.py '
 '--[app_startup]b_daemon=False']

# or sometimes WRONG RESULT 
['/opt/python3/bin/python3 /opt/rotek-apps/bin/rpyc_server/rpyc_server.py --[app_startup]b_daemon=False']

# CORRECT RESULT 
['/opt/python3/bin/python3', '/opt/rotek-apps/bin/rpyc_server/rpyc_server.py'
 '--[app_startup]b_daemon=False']

so I needed to concentate and shlex to split correctly.
Randomly wrong results, maybe because some of those processes were started via systemd service ?
No idea ...

yours sincerely
Robert

@bitranox bitranox changed the title Process.cmdline() wrong results, string instead of lists Process.cmdline() wrong results, parameters not correctly splitted Nov 21, 2017
@giampaolo
Copy link
Owner

@giampaolo giampaolo commented Nov 21, 2017

The cmdline is calculated by reading /proc/pid/cmdline file. Please paste the output of this cmd:

python -c "f = open('/proc/PROCESS_PID_HERE/cmdline', 'rt'); print(repr(f.read()))"
@bitranox
Copy link
Author

@bitranox bitranox commented Nov 21, 2017

ok, the result of the line above from commandline was :

python -c "f = open('/proc/4053/cmdline', 'rt'); print(repr(f.read()))" 

'/opt/python3/bin/python3 /opt/rotek-apps/bin/rpyc_server/rpyc_server.py --[app_startup]b_daemon=False\x00'

proc.cmdline() result was :

>>> proc=psutil.Process(4053)
>>> proc.cmdline()
5: ['/opt/python3/bin/python3 /opt/rotek-apps/bin/rpyc_server/rpyc_server.py '
 '--[app_startup]b_daemon=False']

but when I put the command directly in Dreampie, it also fails : 

>>> x = open('/proc/4053/cmdline', 'rt')
>>> repr(x.read())
2: ("'/opt/python3/bin/python3 /opt/rotek-apps/bin/rpyc_server/rpyc_server.py "
 "--[app_startup]b_daemon=False\\x00'")
>>> 

(please also note the blank after .... _server.py ')
WTF ?
FYI I just compiled Python 3.6.3 on my machine - same results

@bitranox
Copy link
Author

@bitranox bitranox commented Nov 24, 2017

I finally decided to work around, since I found no solution.
I found a nice splitter that is multi-platform and faster and more correct (supports pipe) then shlex - see :
https://stackoverflow.com/questions/33560364/python-windows-parsing-command-lines-with-shlex

maybe its worth looking into it ...

def cmdline_split(s, platform='this'):
    """Multi-platform variant of shlex.split() for command-line splitting.
    For use with subprocess, for argv injection etc. Using fast REGEX.

    platform: 'this' = auto from current platform;
              1 = POSIX; 
              0 = Windows/CMD
              (other values reserved)
    """
    if platform == 'this':
        platform = (sys.platform != 'win32')
    if platform == 1:
        RE_CMD_LEX = r'''"((?:\\["\\]|[^"])*)"|'([^']*)'|(\\.)|(&&?|\|\|?|\d?\>|[<])|([^\s'"\\&|<>]+)|(\s+)|(.)'''
    elif platform == 0:
        RE_CMD_LEX = r'''"((?:""|\\["\\]|[^"])*)"?()|(\\\\(?=\\*")|\\")|(&&?|\|\|?|\d?>|[<])|([^\s"&|<>]+)|(\s+)|(.)'''
    else:
        raise AssertionError('unkown platform %r' % platform)

    args = []
    accu = None   # collects pieces of one arg
    for qs, qss, esc, pipe, word, white, fail in re.findall(RE_CMD_LEX, s):
        if word:
            pass   # most frequent
        elif esc:
            word = esc[1]
        elif white or pipe:
            if accu is not None:
                args.append(accu)
            if pipe:
                args.append(pipe)
            accu = None
            continue
        elif fail:
            raise ValueError("invalid or incomplete shell string")
        elif qs:
            word = qs.replace('\\"', '"').replace('\\\\', '\\')
            if platform == 0:
                word = word.replace('""', '"')
        else:
            word = qss   # may be even empty; must be last

        accu = (accu or '') + word

    if accu is not None:
        args.append(accu)

    return args
@nicolargo
Copy link
Contributor

@nicolargo nicolargo commented Nov 26, 2017

Also reproduced on my side and related to the Glances issue (nicolargo/glances#1192):

4430 is not an Atom/Electron process ==> cmdline() return a list of strings
10053 is an Atom process ==> cmdline() return a list of string (only one string)

In [1]: import psutil
In [4]: p1 = psutil.Process(4430)
In [5]: p2 = psutil.Process(10053)
In [6]: p1.cmdline()
Out[6]: ['/opt/atom/atom --type=renderer --enable-experimental-web-platform-features --no-sandbox --primordial-pipe-token=03BB0BF476DDDA5EB53B9905B85C2709 --lang=en-US --app-path=/opt/atom/resources/app.asar --node-integration=true --disable-blink-features=Auxclick --hidden-page --enable-pinch --num-raster-threads=2 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553 --disable-accelerated-video-decode --service-request-channel-token=03BB0BF476DDDA5EB53B9905B85C2709 --renderer-client-id=4 --v8-natives-passed-by-fd --v8-snapshot-passed-by-fd']
In [7]: p2.cmdline()
Out[7]: ['gpg', '--encrypt', 'test.txt']
@giampaolo
Copy link
Owner

@giampaolo giampaolo commented Nov 28, 2017

So, this is the problem. From man proc:

/proc/[pid]/cmdline
This read-only file holds the complete command line for the process, unless the
process is a zombie.  In the latter case, there is nothing in this file: that 
is, a read on this file will return 0 characters.  The command-line arguments 
appear in this file as a  set of  strings  separated  by  null  bytes  ('\0'), 
with a further null byte after the last string.

The reason why you guys see a single string is because for some reason those processes use space (" ") as a separator instead of "\0". So technically it's the process/pid which is not compliant, not psutil.

With that said, I'm not sure what psutil should do in this case. Maybe it can split by space if the cmdline string has no "\0" in it but I'm a bit afraid, so before doing anything we should do some research.

@giampaolo
Copy link
Owner

@giampaolo giampaolo commented Nov 28, 2017

https://stackoverflow.com/a/1586001
It looks like there can be spaces when setproctitle() or similars are used.

@giampaolo
Copy link
Owner

@giampaolo giampaolo commented Nov 28, 2017

import os
pids = [x for x in os.listdir('/proc') if x.isdigit()]
for pid in pids:
    try:
        with open("/proc/%s/cmdline" % pid) as f:
            data = f.read()
    except EnvironmentError:
        pass
    else:
        if data and '\x00' not in data:
            print(repr(data))
            print("")

This shows the processes "misbehaving". I have different chrome processes like this and a bunch of others:

'avahi-daemon: running [N501VW.local]'

'avahi-daemon: chroot helper'

'/opt/google/chrome/chrome --type=renderer --field-trial-handle=15946332424714596201,340052762019669592,131072 --service-pipe-token=C43F846A2F41DF62B72AB2F9C58118AE --lang=en-US --enable-crash-reporter=458614bd-9830-42b7-87f7-6590a530bbdd, --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553;5,0,3553;5,1,3553;5,2,3553;5,3,3553;5,4,3553;5,5,3553;5,6,3553;5,7,3553;5,8,3553;5,9,3553;5,10,3553;5,11,3553;5,12,3553;5,13,3553;5,14,3553;5,15,3553;5,16,3553;5,17,3553;6,0,3553;6,1,3553;6,2,3553;6,3,3553;6,4,3553;6,5,3553;6,6,3553;6,7,3553;6,8,3553;6,9,3553;6,10,3553;6,11,3553;6,12,3553;6,13,3553;6,14,3553;6,15,3553;6,16,3553;6,17,3553 --disable-accelerated-video-decode --enable-gpu-async-worker-context --service-request-channel-token=C43F846A2F41DF62B72AB2F9C58118AE --renderer-client-id=4824 --shared-files=v8_natives_data:100,v8_snapshot_data:101'

'/opt/google/chrome/chrome --type=renderer --field-trial-handle=15946332424714596201,340052762019669592,131072 --service-pipe-token=965834BB41A5A5585B28F0FA3F9193C3 --lang=en-US --enable-crash-reporter=458614bd-9830-42b7-87f7-6590a530bbdd, --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553;5,0,3553;5,1,3553;5,2,3553;5,3,3553;5,4,3553;5,5,3553;5,6,3553;5,7,3553;5,8,3553;5,9,3553;5,10,3553;5,11,3553;5,12,3553;5,13,3553;5,14,3553;5,15,3553;5,16,3553;5,17,3553;6,0,3553;6,1,3553;6,2,3553;6,3,3553;6,4,3553;6,5,3553;6,6,3553;6,7,3553;6,8,3553;6,9,3553;6,10,3553;6,11,3553;6,12,3553;6,13,3553;6,14,3553;6,15,3553;6,16,3553;6,17,3553 --disable-accelerated-video-decode --enable-gpu-async-worker-context --service-request-channel-token=965834BB41A5A5585B28F0FA3F9193C3 --renderer-client-id=4846 --shared-files=v8_natives_data:100,v8_snapshot_data:101'

'/opt/google/chrome/chrome --type=renderer --field-trial-handle=15946332424714596201,340052762019669592,131072 --service-pipe-token=C5F882F77CEE9065287600DF09AFAE1B --lang=en-US --enable-crash-reporter=458614bd-9830-42b7-87f7-6590a530bbdd, --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553;5,0,3553;5,1,3553;5,2,3553;5,3,3553;5,4,3553;5,5,3553;5,6,3553;5,7,3553;5,8,3553;5,9,3553;5,10,3553;5,11,3553;5,12,3553;5,13,3553;5,14,3553;5,15,3553;5,16,3553;5,17,3553;6,0,3553;6,1,3553;6,2,3553;6,3,3553;6,4,3553;6,5,3553;6,6,3553;6,7,3553;6,8,3553;6,9,3553;6,10,3553;6,11,3553;6,12,3553;6,13,3553;6,14,3553;6,15,3553;6,16,3553;6,17,3553 --disable-accelerated-video-decode --enable-gpu-async-worker-context --service-request-channel-token=C5F882F77CEE9065287600DF09AFAE1B --renderer-client-id=4872 --shared-files=v8_natives_data:100,v8_snapshot_data:101'

'/opt/google/chrome/chrome --type=ppapi --field-trial-handle=15946332424714596201,340052762019669592,131072 --ppapi-flash-args --lang=en-US --enable-crash-reporter=458614bd-9830-42b7-87f7-6590a530bbdd, --service-request-channel-token=2A8EB062495DF054DED3138D0B2138F4'

'/opt/google/chrome/chrome --type=renderer --field-trial-handle=15946332424714596201,340052762019669592,131072 --service-pipe-token=752210288667214C22CCECAEBCC2040F --lang=en-US --enable-crash-reporter=458614bd-9830-42b7-87f7-6590a530bbdd, --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553;5,0,3553;5,1,3553;5,2,3553;5,3,3553;5,4,3553;5,5,3553;5,6,3553;5,7,3553;5,8,3553;5,9,3553;5,10,3553;5,11,3553;5,12,3553;5,13,3553;5,14,3553;5,15,3553;5,16,3553;5,17,3553;6,0,3553;6,1,3553;6,2,3553;6,3,3553;6,4,3553;6,5,3553;6,6,3553;6,7,3553;6,8,3553;6,9,3553;6,10,3553;6,11,3553;6,12,3553;6,13,3553;6,14,3553;6,15,3553;6,16,3553;6,17,3553 --disable-accelerated-video-decode --enable-gpu-async-worker-context --service-request-channel-token=752210288667214C22CCECAEBCC2040F --renderer-client-id=4994 --shared-files=v8_natives_data:100,v8_snapshot_data:101'

'/opt/google/chrome/chrome --type=renderer --field-trial-handle=15946332424714596201,340052762019669592,131072 --service-pipe-token=25B98A3A7B3FA5BDFCBE6B1E77A22162 --lang=en-US --enable-crash-reporter=458614bd-9830-42b7-87f7-6590a530bbdd, --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553;5,0,3553;5,1,3553;5,2,3553;5,3,3553;5,4,3553;5,5,3553;5,6,3553;5,7,3553;5,8,3553;5,9,3553;5,10,3553;5,11,3553;5,12,3553;5,13,3553;5,14,3553;5,15,3553;5,16,3553;5,17,3553;6,0,3553;6,1,3553;6,2,3553;6,3,3553;6,4,3553;6,5,3553;6,6,3553;6,7,3553;6,8,3553;6,9,3553;6,10,3553;6,11,3553;6,12,3553;6,13,3553;6,14,3553;6,15,3553;6,16,3553;6,17,3553 --disable-accelerated-video-decode --enable-gpu-async-worker-context --service-request-channel-token=25B98A3A7B3FA5BDFCBE6B1E77A22162 --renderer-client-id=5005 --shared-files=v8_natives_data:100,v8_snapshot_data:101'

'/opt/google/chrome/chrome --type=renderer --field-trial-handle=15946332424714596201,340052762019669592,131072 --service-pipe-token=B71630091F9A394AC5C6BB1F8DF47A27 --lang=en-US --enable-crash-reporter=458614bd-9830-42b7-87f7-6590a530bbdd, --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553;5,0,3553;5,1,3553;5,2,3553;5,3,3553;5,4,3553;5,5,3553;5,6,3553;5,7,3553;5,8,3553;5,9,3553;5,10,3553;5,11,3553;5,12,3553;5,13,3553;5,14,3553;5,15,3553;5,16,3553;5,17,3553;6,0,3553;6,1,3553;6,2,3553;6,3,3553;6,4,3553;6,5,3553;6,6,3553;6,7,3553;6,8,3553;6,9,3553;6,10,3553;6,11,3553;6,12,3553;6,13,3553;6,14,3553;6,15,3553;6,16,3553;6,17,3553 --disable-accelerated-video-decode --enable-gpu-async-worker-context --service-request-channel-token=B71630091F9A394AC5C6BB1F8DF47A27 --renderer-client-id=5014 --shared-files=v8_natives_data:100,v8_snapshot_data:101'

'/opt/google/chrome/chrome --type=renderer --field-trial-handle=15946332424714596201,340052762019669592,131072 --service-pipe-token=C78DFB86D028F752C5665C9AE198A824 --lang=en-US --enable-crash-reporter=458614bd-9830-42b7-87f7-6590a530bbdd, --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553;5,0,3553;5,1,3553;5,2,3553;5,3,3553;5,4,3553;5,5,3553;5,6,3553;5,7,3553;5,8,3553;5,9,3553;5,10,3553;5,11,3553;5,12,3553;5,13,3553;5,14,3553;5,15,3553;5,16,3553;5,17,3553;6,0,3553;6,1,3553;6,2,3553;6,3,3553;6,4,3553;6,5,3553;6,6,3553;6,7,3553;6,8,3553;6,9,3553;6,10,3553;6,11,3553;6,12,3553;6,13,3553;6,14,3553;6,15,3553;6,16,3553;6,17,3553 --disable-accelerated-video-decode --enable-gpu-async-worker-context --service-request-channel-token=C78DFB86D028F752C5665C9AE198A824 --renderer-client-id=5028 --shared-files=v8_natives_data:100,v8_snapshot_data:101'

'/opt/google/chrome/chrome --type=renderer --field-trial-handle=15946332424714596201,340052762019669592,131072 --service-pipe-token=22CE9B91A8B06624072A928FE5D343CD --lang=en-US --enable-crash-reporter=458614bd-9830-42b7-87f7-6590a530bbdd, --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553;5,0,3553;5,1,3553;5,2,3553;5,3,3553;5,4,3553;5,5,3553;5,6,3553;5,7,3553;5,8,3553;5,9,3553;5,10,3553;5,11,3553;5,12,3553;5,13,3553;5,14,3553;5,15,3553;5,16,3553;5,17,3553;6,0,3553;6,1,3553;6,2,3553;6,3,3553;6,4,3553;6,5,3553;6,6,3553;6,7,3553;6,8,3553;6,9,3553;6,10,3553;6,11,3553;6,12,3553;6,13,3553;6,14,3553;6,15,3553;6,16,3553;6,17,3553 --disable-accelerated-video-decode --enable-gpu-async-worker-context --service-request-channel-token=22CE9B91A8B06624072A928FE5D343CD --renderer-client-id=5035 --shared-files=v8_natives_data:100,v8_snapshot_data:101'

'/opt/google/chrome/chrome --type=renderer --field-trial-handle=15946332424714596201,340052762019669592,131072 --service-pipe-token=89410E1783E802F174E265F00026CE0B --lang=en-US --enable-crash-reporter=458614bd-9830-42b7-87f7-6590a530bbdd, --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553;5,0,3553;5,1,3553;5,2,3553;5,3,3553;5,4,3553;5,5,3553;5,6,3553;5,7,3553;5,8,3553;5,9,3553;5,10,3553;5,11,3553;5,12,3553;5,13,3553;5,14,3553;5,15,3553;5,16,3553;5,17,3553;6,0,3553;6,1,3553;6,2,3553;6,3,3553;6,4,3553;6,5,3553;6,6,3553;6,7,3553;6,8,3553;6,9,3553;6,10,3553;6,11,3553;6,12,3553;6,13,3553;6,14,3553;6,15,3553;6,16,3553;6,17,3553 --disable-accelerated-video-decode --enable-gpu-async-worker-context --service-request-channel-token=89410E1783E802F174E265F00026CE0B --renderer-client-id=5050 --shared-files=v8_natives_data:100,v8_snapshot_data:101'

'/opt/google/chrome/chrome --type=renderer --field-trial-handle=15946332424714596201,340052762019669592,131072 --service-pipe-token=6DD76D3A7C5405561374E98A3806EA06 --lang=en-US --enable-crash-reporter=458614bd-9830-42b7-87f7-6590a530bbdd, --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553;5,0,3553;5,1,3553;5,2,3553;5,3,3553;5,4,3553;5,5,3553;5,6,3553;5,7,3553;5,8,3553;5,9,3553;5,10,3553;5,11,3553;5,12,3553;5,13,3553;5,14,3553;5,15,3553;5,16,3553;5,17,3553;6,0,3553;6,1,3553;6,2,3553;6,3,3553;6,4,3553;6,5,3553;6,6,3553;6,7,3553;6,8,3553;6,9,3553;6,10,3553;6,11,3553;6,12,3553;6,13,3553;6,14,3553;6,15,3553;6,16,3553;6,17,3553 --disable-accelerated-video-decode --enable-gpu-async-worker-context --service-request-channel-token=6DD76D3A7C5405561374E98A3806EA06 --renderer-client-id=5066 --shared-files=v8_natives_data:100,v8_snapshot_data:101'

....
@giampaolo giampaolo closed this in 7c6b6c2 Nov 28, 2017
@giampaolo
Copy link
Owner

@giampaolo giampaolo commented Nov 28, 2017

OK this is fixed.

nlevitt added a commit to nlevitt/psutil that referenced this issue Apr 9, 2019
* 'pslisten' of github.com:nlevitt/psutil: (922 commits)
  Update INSTALL.rst
  Pass python_requires argument to setuptools (giampaolo#1208)
  giampaolo#1152: fix doc to mention CLI command necessary to enable disk_io_counters() on win
  pre release
  pre release
  pre release
  pre-release
  fix giampaolo#1201: document that timeout kwarg is expressed in seconds
  Add mount points to disk_partitions() in Windows (giampaolo#775) (giampaolo#1192)
  add test for cpu_affinity
  what a stupid bug! (giampaolo#1190)
  update doc
  pre release
  pre-release; also get rid of PSUTIL_DEBUG doc instructions (it's kinda useless for the user after all)
  Use FutureWarning instead of DeprecationWarning (giampaolo#1188)
  fix test
  refactor environ() test
  Fix OSX pid 0 bug (giampaolo#1187)
  change assert in test
  refactor Process.__repr__
  Faster Process.children(recursive=True) (giampaolo#1186)
  Speedup Process.children()  (giampaolo#1185)
  update doc
  update HISTORY
  fix giampaolo#1179 / linux / cmdline: handle processes erroneously overwriting /proc/pid/cmdline by using spaces instead of null bytes as args separator
  set x bit to test_aix.py
  fix giampaolo#1181: raise AD if task_for_pid() fails with 5 and errno == ENOENT
  fix posix failure
  Arguments for NoSuchProcess and AccessDenied for the C ext (giampaolo#1180)
  fix travis failure https://travis-ci.org/giampaolo/psutil/jobs/306424509
  be smarter in searching python exe
  do not test platf specific modules on wheelhouse
  try to fix travis failure
  fix travis failures
  try to use PYTHON_EXE instead of sys.executable
  giampaolo#1177: give credits to @wiggin15
  OSX: implement sensors_battery (giampaolo#1177)
  improve error msg for old windows systems giampaolo#811
  add debug messages
  do not mention apt-get as method of installation as it's not recommended
  syntax highlight in doc files
  syntax highlight in doc files
  fix doc indentation
  1173 debug mode (giampaolo#1176)
  code style
  update MANIFEST
  giampaolo#1174: use TimeoutExpired in wait_pid()
  sort imports by name
  Move exceptions to separate file (giampaolo#1174)
  appveyor: enable python warnings when running tests
  refactor winmake.py
  use a C global variable to figure out whether we're in testing mode
  fix unicode err
  define a setup() function which is called on import by all C modules
  move PyUnicode compt fun definition up in the file
  rename C func
  re-enable test on appveyor; remove unused C code
  refactor PSUTIL_TESTING C APIs
  inspect PSUTIL_TESTING env var from C again
  giampaolo#1152: (DeviceIOControl), skip disk on ERROR_INVALID_FUNCTION and ERROR_NOT_SUPPORTED
  giampaolo#1152 / win / disk_io_counters(): DeviceIOControl errors were ignored; che return value and retry call on ERROR_INSUFFICIENT_BUFFER
  upgrade dist cmds
  change make cmds
  disable IPv6 tests if IPv6 is not supported
  travis / OSX: run py 3.6 instead of 3.4
  fix giampaolo#1169: (Linux) users() hostname returns username instead
  update README, bump up version
  get rid of PSUTIL_TESTING env var: it must be necessarily set from cmdline, hence 'python -m psutil.tests' won't work out of the box
  try to set PSUTIL_TESTING env var from python before failing
  skip cpu_freq tests if not available (giampaolo#1170)
  update doc
  pre-release
  giampaolo#1053: drop python 3.3 support
  try to fix appveyor failure; also refactor generate_manifest.py
  giampaolo#1167 give CREDITS to @matray
  Including non-unicast packets in packet count calculation (giampaolo#1167)
  fix giampaolo#1166 (doc mistake)
  provide a 'make help' command
  ifconfig.py humanize bytes
  try to limit false positives on appveyor/windows
  reap_children() in a finally block in order to limit false positives
  unicode tests: use different name for test dir
  fix failure on osx/travis
  update Makefile
  fix test
  giampaolo#1164 give CREDITS to @wiggin15
  AIX: implement num_ctx_switches (giampaolo#1164)
  use new PYTHON_EXE
  improve logic to determine python exe location
  add DEVNOTES file; move TODO.aix into _psutil_aix.c
  Fix test_emulate_energy_full_not_avail (giampaolo#1163)
  update README
  try to limit occasional appveyor failure
  Remove trove classifiers for untested and unsupported platforms (giampaolo#1162)
  Fix giampaolo#1154: remove 'threads' method on older AIX (giampaolo#1156)
  give CREDITS to @adpag for giampaolo#1159, giampaolo#1160 and giampaolo#1161
  Fix test asserts due to leftover test subprocesses (giampaolo#1161)
  Fix network tests for newer ifconfig versions. (giampaolo#1160)
  Fix pre-commit hook for python 3.x. (giampaolo#1159)
  revert last commit
  ...
@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 12, 2019

This doesn't seem to be completely fixed. In some circumstances the cmdline file can end with a null byte but still be separated by spaces. In this case cmdline() does not return the parameters correctly split.

Example:

❯ hexdump -C /proc/206724/cmdline
00000000  2f 75 73 72 2f 6c 69 62  2f 71 74 2f 6c 69 62 65  |/usr/lib/qt/libe|
00000010  78 65 63 2f 51 74 57 65  62 45 6e 67 69 6e 65 50  |xec/QtWebEngineP|
00000020  72 6f 63 65 73 73 20 2d  2d 74 79 70 65 3d 7a 79  |rocess --type=zy|
00000030  67 6f 74 65 20 2d 2d 77  65 62 65 6e 67 69 6e 65  |gote --webengine|
00000040  2d 73 63 68 65 6d 65 73  3d 71 75 74 65 3a 6c 4c  |-schemes=qute:lL|
00000050  3b 71 72 63 3a 73 4c 56  20 2d 2d 6c 61 6e 67 3d  |;qrc:sLV --lang=|
00000060  65 6e 2d 47 42 00                                 |en-GB.|
00000066
@giampaolo
Copy link
Owner

@giampaolo giampaolo commented Nov 13, 2019

psutil already bends itself to accomodate for processes manually re-setting the cmdline without following the rules. This looks like yet another corner case. We can't cover all of them. =)

@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 14, 2019

I get where you're coming from, but how else is a program relying on cmdline being accurate going to know if it has been modified?

Edit:
Ok having said that I can't think of how you'd do it from your end either lol

@bitranox
Copy link
Author

@bitranox bitranox commented Nov 14, 2019

Jonny, You can try following:

pip install git+https://github.com/bitranox/lib_shell.git

import lib_shell
import psutil

process = psutil.Process(...)
l_commands = process.cmdline()
s_command = ' '.join(l_commands)
l_correct_splitted_commands = lib_shell.shlex_split_multi_platform(s_command)

...

let me know
Robert
@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 14, 2019

@bitranox No, unfortunately this does not solve the problem because an argument could have spaces in it, but once you join the arguments together, there are no quotes.

E.g.

+>>> l_commands = ['zathura', '--', '/home/user/Documents/some random document.pdf']
+>>> s_command = ' '.join(l
+>>> s_command = ' '.join(l_commands)
+>>> lib_shell.shlex_split_multi_platform(s_command)
['zathura', '--', '/home/user/Documents/some', 'random', 'document.pdf']

The result of running this command would be that zathura would try to open 3 (non-existent) files instead of 1 real file.

@bitranox
Copy link
Author

@bitranox bitranox commented Nov 14, 2019

@JonnyHaystack
ok, I dont know the whole information, and how exactly the commandline for zathura looks like,
but You just might add the quotes before sending the string to shlexer. (of course not for elements in the list which start with '-' or '--', because thats obviously a parameter.
I am sure You can solve it that way ...
You might check how /proc/.../cmdline looks like ... maybe the quotes are there ?

Robert

@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 14, 2019

@bitranox But then that does nothing. It would only work if the arguments were correctly in an array to begin with.

If you had ['zathura -- "/home/user/Documents/some random document.pdf"'] then it would become ['"zathura -- "/home/user/Documents/some random document.pdf"'], in which case the quotes are invalid.

@bitranox
Copy link
Author

@bitranox bitranox commented Nov 14, 2019

I dont get Your meaning.

>>> lib_shell.shlex_split_multi_platform('zathura -- "/home/user/Documents/some random document.pdf"')
['zathura', '--', '/home/user/Documents/some random document.pdf']

everything good ?

@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 14, 2019

But you can't cover both cases...

If you write something that works for a "bad" cmdline like:

['zathura -- "/home/user/Documents/some random document.pdf"']

It will not work for a good cmdline like:

['zathura', '--', '/home/user/Documents/some random document.pdf']
@bitranox
Copy link
Author

@bitranox bitranox commented Nov 14, 2019

You might check how /proc/.../cmdline looks like ... maybe the quotes are there ?
In that case use only shlexer ?

on my system, /proc/.../cmdline parts are seperated by "0" :

geany ".\test test.txt"

00000000  67 65 61 6e 79 00 2e 2f  74 65 73 74 20 74 65 73  |geany../test tes|
00000010  74 2e 74 78 74 00                                                    |t.txt.|

@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 14, 2019

I'm not sure what you're trying to say. If the program modifies its own cmdline, generally it will include quotes where necessary. But you there is no way of telling if the program has done this, and there is no way to reliably interpret the cmdline without knowing if it has been modified into one string.

@bitranox
Copy link
Author

@bitranox bitranox commented Nov 15, 2019

@JonnyHaystack , I made a new version for You - please try :

# fresh install
pip install git+https://github.com/bitranox/lib_shell.git

# or if You installed it already : 
pip install --upgrade --upgrade-strategy eager git+https://github.com/bitranox/lib_shell.git

import psutil
import lib_shell

# via psutil process
process = psutil.Process(...)
l_commandline = lib_shell.get_l_commandline_from_psutil_process(process)

# via pid
pid = process.pid
l_commandline = lib_shell.get_l_commandline_from_pid(pid)

...

I tested it under Linux, OSx and Windows - let me know if You find some cornercases were it does not work. 
(include hexdump of /proc/.../cmdline in case, and how originally the command was issued)
Robert
@bitranox
Copy link
Author

@bitranox bitranox commented Nov 15, 2019

@giampaolo
does not look like a corner case - it looks like on linux, if there are parameters with a blank (usually filenames) the psutil.cmdline is not splitted correctly.
simply splitting /proc/.../cmdline at x00 should do the job, if I had not missed something out ?
Robert

@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 15, 2019

psutil does split it by the null byte, \x00. If the cmdline does not end with a null byte, it splits by the space character instead. Not sure why it splits by the space character.. imo it would be much better to use shlex.split() there.

But anyway my problem is that a lot of the processes that modify their own cmdline still end with the null byte, despite not being separated by null bytes. I can't think of a reliable way to distinguish between a legitimate cmdline that just has one argument and a modified cmdline that has multiple arguments stuffed into argv[0].

@bitranox
Copy link
Author

@bitranox bitranox commented Nov 15, 2019

But anyway my problem is that a lot of the processes that modify their own cmdline still end with the null byte, despite not being separated by null bytes.

Do You have an example for me ? Did You really check the root process or just some subprocess of the program ?
Did You check my solution ?

@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 15, 2019

These are processes obtained from window IDs using xprop. See JonnyHaystack/i3-resurrect#35 (comment) for a real life example.

@bitranox
Copy link
Author

@bitranox bitranox commented Nov 16, 2019

@JonnyHaystack
You are absolutely right - there are some cases on linux were the string is separated with blank instead of "\x00" - for instance the postgrey service

I investigated a bit, and found some of my own scripts having a commandline like that - if they are started via a systemd startscript. (I defenitely did not change /proc/<pid>/cmdline myself).
So it might have something to to with systemd.

I adapted my script, it should now give back the correct values in both cases, under Windows and Linux. BTW, the lexer I use (with regexp) is about 10x faster than the the original python shlex, if it matters to You at all.

Please test it !

  • if it gives wrong values, please tell me the OS and the Program , the returned vs expected values, so I can test against it.
# fresh install
pip install git+https://github.com/bitranox/lib_shell.git

# or if You installed it already : 
pip install --upgrade --upgrade-strategy eager git+https://github.com/bitranox/lib_shell.git
import psutil
import lib_shell

# via psutil process
process = psutil.Process(...)
l_commandline = lib_shell.get_l_commandline_from_psutil_process(process)

# via pid
pid = process.pid
l_commandline = lib_shell.get_l_commandline_from_pid(pid)
...

please let me know
Robert
@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 16, 2019

Python 3.8.0 (default, Oct 23 2019, 18:51:26) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
+>>> import psutil
+>>> import lib_shell
+>>> process = psutil.Process(3255330)
+>>> process.cmdline()
['/usr/lib/electron6/electron --no-sandbox --unity-launch --no-sandbox /usr/lib/code/code.js']
+>>> l_commandline = lib_shell.get_l_commandline_from_psutil_process(process)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'lib_shell' has no attribute 'get_l_commandline_from_psutil_process'
@bitranox
Copy link
Author

@bitranox bitranox commented Nov 16, 2019

ah, yes, sorry, forgot to put it in __init__.py ...
it should work now

use:

pip install --upgrade --upgrade-strategy eager git+https://github.com/bitranox/lib_shell.git

Robert

@bitranox
Copy link
Author

@bitranox bitranox commented Nov 18, 2019

@JonnyHaystack
please let me know if You find some cases where it is not working ...
Robert

@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 18, 2019

I have thought of doing what you are doing, but it doesn't cover the case where the path to the executable contains spaces. See the problem:

+>>> import psutil
+>>> import lib_shell
+>>> process = psutil.Process(3582982)
+>>> process.cmdline()
['/usr/lib/electron6/electron --no-sandbox --unity-launch --no-sandbox /usr/lib/code/code.js']
+>>> l_commandline = lib_shell.get_l_commandline_from_psutil_process(process)
+>>> l_commandline
['/usr/lib/electron6/electron', '--no-sandbox', '--unity-launch', '--no-sandbox', '/usr/lib/code/code.js']
+>>> process = psutil.Process(847052)
+>>> process.cmdline()
['zathura', '--', '/home/haystack/Documents/Lectures/Notes/COMP33711/The Agile Methodologies.pdf']
+>>> l_commandline = lib_shell.get_l_commandline_from_psutil_process(process)
+>>> l_commandline
['zathura', '--', '/home/haystack/Documents/Lectures/Notes/COMP33711/The Agile Methodologies.pdf']
+>>> process = psutil.Process(3600595)
+>>> process.cmdline()
['/home/haystack/Downloads/Test Test/bash']
+>>> l_commandline = lib_shell.get_l_commandline_from_psutil_process(process)
+>>> l_commandline
['/home/haystack/Downloads/Test', 'Test/bash']

In the last case, the path to the executable is /home/haystack/Downloads/Test Test/bash, but your code is assuming that just because there is a space in it, it should be split, which in this case is incorrect.

This is why I say I can't think of a reliable way to distinguish between a valid or modified cmdline.

@bitranox
Copy link
Author

@bitranox bitranox commented Nov 18, 2019

@JonnyHaystack
oh yes, You challenge me ;-)
I checked and I think there is a way to cover all cases:
we just check if the executable exists, and of course we will have to take also relative paths in the commandline into consideration, because the actual directory might have been changed.

Before "resurrecting" we need also to change the current directory, because also parameters might contain filenames which are relative to the current active directory
we can get the active directory of the process with psutil.Process(...).cdw
from Your Example :

s_cmdline = '/home/haystack/Downloads/Test Test/bash'
# check if '/home/haystack/Downloads/Test' exists ? --> NO
# check if '/home/haystack/Downloads/Test Test/bash' exists? --> YES
# so this is the command - lets quote it and put it into shlex : 
shlex('"/home/haystack/Downloads/Test Test/bash"')
['/home/haystack/Downloads/Test Test/bash'] # correct result

will work on it and let You know ...

Robert

@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 18, 2019

Oh yeah good idea, I didn't think of just checking if it is an existing executable. That ought to cover all cases nicely! I'm assuming either absolute paths or something in the user's PATH for now at least, so it should be fine with just a simple check.

I do already set the cwd when restoring processes. You also have to make sure to set the PWD environment variable if you want the working directory to be set properly.

EDIT:
Actually hang on, can't we just solve all this by using psutil.Process.exe()?
EDIT2:
Well maybe not all, but at least that makes checking for the executable easier when the cmdline uses a relative path.

@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 19, 2019

Okay I got it working to the point where it will only fail if cmdline has one argument, which is a relative path to the executable and contains spaces. That's plenty good enough for me.

The way I do it is like so:

cmdline = process.cmdline()
exe = process.exe()

if len(cmdline) == 1 and shutil.which(cmdline[0]) is None:
    cmdline = shlex.split(cmdline[0])
if exe is not None:
    cmdline[0] = exe
@JonnyHaystack
Copy link

@JonnyHaystack JonnyHaystack commented Nov 19, 2019

I've found out that there are some cases where the cmdline ends with multiple null characters at the end, but psutil assumes there can only be one, so it just removes the last character before splitting. The result of this is that you have a load of empty string arguments that shouldn't be there.

I also don't think this is the correct thing to do if the cmdline has all been crammed into one string. Why not use shlex.split() in that case? If you split by the space character and there are arguments with spaces, it would break the cmdline to the point where it's impossible to reconstruct and one would be forced to not use psutil in that case.

@bitranox
Copy link
Author

@bitranox bitranox commented Nov 19, 2019

@JonnyHaystack
You are right - my version takes that into consideration, I read the cmdline string from proc myself, not through psutil.
I am just tinkering around with the tests at the moment, but it should work.
In case of any errors, please provide hexdump + the original cmdline in order to put it into my tests ....
let me know.
Robert

@giampaolo
Copy link
Owner

@giampaolo giampaolo commented Nov 20, 2019

I've found out that there are some cases where the cmdline ends with multiple null characters at the end, but psutil assumes there can only be one, so it just removes the last character before splitting. The result of this is that you have a load of empty string arguments that shouldn't be there.

That is expected and there is also a test case for it:

cmdline = [funky_path, "-c",
"import time; [time.sleep(0.01) for x in range(3000)];"
"arg1", "arg2", "", "arg3", ""]

As for this issue:
#1179 (comment)
...I guess you can manage it yourself like:

cmdline = proc.cmdline()
if len(cmdline) == 1 and ' ' in cmdline[0]:
    cmdline = cmdline[0].split(' ')

How many processes like this do you have? 1? Do you know anything about it (app name or parent name)? It would be good to get a sense of how common this is in practice, and whether it is worth to include a fix in psutil or not.

To b clear, what I imagine this misbehaving app is doing is:

  • use setproctitle() to alter the process cmdline
  • use spaces to divide the cmdline arguments
  • add '\x00' at the end
@bitranox
Copy link
Author

@bitranox bitranox commented Nov 20, 2019

Hello Paolo,
good to see You again - hope You are safely back from China.

I have a python program myself, that shows 0x20 instead of 0x00 as a separator between the command and the parameters - first I thought it is because started with systemd daemon, but that was not the case.
In my case, it seems to be the rpyc package, that forks the process (for rpyc forking server) and somehow messes up with the parameters. I did not find the root cause yet, and it is not that important to me to spend that much time on it. I glanced over the code, but could not find anything unusual - it seems to happen when the process is forked, but was not able to create a minimal example with os.fork (my minimal examples always worked correctly)
It was definitely NOT setproctitle, and definitely NOT tinkering with sys.argv !
However, I have seen myself an additional \x00 at the end of some commandlines.
And I have seen commandlines terminated with \x00 but separated with \x20 :

00000000  2f 6f 70 74 2f 70 79 74  68 6f 6e 33 2f 62 69 6e  |/opt/python3/bin|
00000010  2f 70 79 74 68 6f 6e 33  20 2f 6d 65 64 69 61 2f  |/python3 /media/|
00000020  73 72 76 2d 6d 61 69 6e  2d 73 6f 66 74 64 65 76  |srv-main-softdev|
00000030  2f 72 6f 74 65 6b 2d 61  70 70 73 2f 62 69 6e 2f  |/rotek-apps/bin/|
00000040  72 70 79 63 5f 73 65 72  76 65 72 2f 72 70 79 63  |rpyc_server/rpyc|
00000050  5f 73 65 72 76 65 72 2e  70 79 20 2d 2d 5b 63 6f  |_server.py --[co|
00000060  6e 66 5f 61 70 70 5f 73  74 61 72 74 75 70 5d 64  |nf_app_startup]d|
00000070  61 65 6d 6f 6e 3d 46 61  6c 73 65 00              |aemon=False.|
0000007c

this was started with systemd service with root rights. If I start it from the console with "sudo" I see following processes :

7948 pts/0    S+     0:00 sudo /opt/python3/bin/python3 /opt/rotek-apps/bin/rpyc_server/rpyc_server.py --[conf_app_startup]daemon=False
 7949 pts/0    Sl+    0:00 /opt/python3/bin/python3 /opt/rotek-apps/bin/rpyc_server/rpyc_server.py --[conf_app_startup]daemon=False

pid 7948 is the process I started
pid 7949 is the forked process obviously:

# hexdump -C /proc/7948/cmdline - correctly separated with 00
00000000  73 75 64 6f 00 2f 6f 70  74 2f 70 79 74 68 6f 6e  |sudo./opt/python|
00000010  33 2f 62 69 6e 2f 70 79  74 68 6f 6e 33 00 2f 6f  |3/bin/python3./o|
00000020  70 74 2f 72 6f 74 65 6b  2d 61 70 70 73 2f 62 69  |pt/rotek-apps/bi|
00000030  6e 2f 72 70 79 63 5f 73  65 72 76 65 72 2f 72 70  |n/rpyc_server/rp|
00000040  79 63 5f 73 65 72 76 65  72 2e 70 79 00 2d 2d 5b  |yc_server.py.--[|
00000050  63 6f 6e 66 5f 61 70 70  5f 73 74 61 72 74 75 70  |conf_app_startup|
00000060  5d 64 61 65 6d 6f 6e 3d  46 61 6c 73 65 00        |]daemon=False.|
# hexdump -C /proc/7949/cmdline - separated with 20
00000000  2f 6f 70 74 2f 70 79 74  68 6f 6e 33 2f 62 69 6e  |/opt/python3/bin|
00000010  2f 70 79 74 68 6f 6e 33  20 2f 6f 70 74 2f 72 6f  |/python3 /opt/ro|
00000020  74 65 6b 2d 61 70 70 73  2f 62 69 6e 2f 72 70 79  |tek-apps/bin/rpy|
00000030  63 5f 73 65 72 76 65 72  2f 72 70 79 63 5f 73 65  |c_server/rpyc_se|
00000040  72 76 65 72 2e 70 79 20  2d 2d 5b 63 6f 6e 66 5f  |rver.py --[conf_|
00000050  61 70 70 5f 73 74 61 72  74 75 70 5d 64 61 65 6d  |app_startup]daem|
00000060  6f 6e 3d 46 61 6c 73 65  00                       |on=False.|

If You look at the processlist of Your linux box, You will find for sure some of that misbehaving processes (for instance google chrome subprocesses, they are all separated with 0x20)

I think there can not be a 100% working solution, especially if You have parameters with blank in it - because the shell throws away the quoting, and it ends up in a cmdline separated with blanks - so no way to make it 100% correctly working in every case.

Anyway, I think You should patch psutil.Process().cmdline a bit -

lets assume the original command was :
"./test test/test test.sh" --parameter=testparm1 --parameter2="test parm2"
and the cmdline ends up 0x20 separated (reasons unclear):
./test test/test test.sh --parameter=testparm1 --parameter2=test parm2

- split /x00
- throw away empty elements of the list (if any)
- if the list is one element only, check if 0x20 is in the string - if not --> return cmdline
- if 0x20 is in the string, check all combinations for the executable : 

cmdstring='./test test/test test.sh --parameter=testparm1 --parameter2=test parm2'
does './test test/test test.sh --parameter=testparm1  --parameter2=test parm2' exist ? -> NO
does './test test/test test.sh --parameter=testparm1  --parameter2=test' exist ? -> NO
does './test test/test test.sh --parameter=testparm1 ' exist ? -> NO
does './test test/test test.sh' exist ? -> yes, s_executable = './test test/test test.sh'
(of course You must check relative to the current directory of the process if the path is relative)
s_parameter = cmdstring.split[s_executable][1]
cmdline = '"' +  s_executable + '"' + s_parameter   # of course we need correct quoting here  
l_cmdline = shlex(cmdline)
return l_cmdline

well - this works in MOST of the cases - except if there is a blank in one of the parameters
You might check for details, but it is refractored constantly ...

pip install git+https://github.com/bitranox/lib_shell.git

it would be really good to find out the root cause, but I did not succeed - maybe You are more lucky.
yours sincerely
Robert

EDIT1: interesting - on some cases it can be really setproctitle - but not at rpyc, that does not use python setproctitle at all.
https://unix.stackexchange.com/questions/432419/unexpected-non-null-encoding-of-proc-pid-cmdline

giampaolo added a commit that referenced this issue Nov 21, 2019
…ends with null byte but args are separated by spaces
@giampaolo
Copy link
Owner

@giampaolo giampaolo commented Nov 21, 2019

Hello Paolo, good to see You again - hope You are safely back from China.

Hey there. No, I am still Chinese for a little while. =)
So, AFAIU you experience the same problem (end = "\x00", args separator = "\x20" / space). Well, I guess it makes sense to fix this thing in psutil after all. I did it in edb20f6. Can some of you guys confirm it worked?

@bitranox
Copy link
Author

@bitranox bitranox commented Nov 21, 2019

Paolo,
the test string should be extended :

# OLD: fake_file = io.StringIO(u('foo\x20bar\x00'))
# add additional \x00
# check if executable foo\ bar exists, to detect that "parameter1" and "para meter2"
# are parameters, and not part of the filename 
# (of course, in that case the file "foo\ bar" needs to exist
# create_file(''foo\x20bar')
# TEST1 : u('foo\x20bar\x20parameter1\x20para\x20meter2\x00\x00'))
# we will receive cmdline == ['foo\ bar', 'parameter1', 'para', 'meter2']
# but we cant do anything about it. If there are blanks in the parameters, we cant do anything about it
#
# TEST2 : u('foo\x20bar\x20parameter1\x20"para\x20meter2"\x00\x00'))
# we will receive cmdline == ['foo\ bar', 'parameter1', 'para meter2']
# it would be good to test also "fancy" and utf-8 characters in paths, parameters, etc.

Dont know if You considered that we can only find out which part is the execuable (absolute or relative path) or symlink, by actually looking if the file exists. all the rest are parameters for the executable.

BTW - this happens only on Linux, not in Windows - there I use just the lexer.

Well - that is the best we can do, it still will fail sometimes, but thats the best we can come up with.

zhu ni tian tian kuai le
zai jian
Robert

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants