Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSX: memory_maps() segfaults or raise EINVAL #1291

Closed
wrishel opened this issue Jun 10, 2018 · 22 comments · Fixed by #1436
Closed

OSX: memory_maps() segfaults or raise EINVAL #1291

wrishel opened this issue Jun 10, 2018 · 22 comments · Fixed by #1436

Comments

@wrishel
Copy link

wrishel commented Jun 10, 2018

I get Exception: OSError(22, 'Invalid argument') on OSX 10.13.4 with Python 2.7

Adding this to the loop makes it go away:

for proc in psutil.process_iter():
        if proc.pid == os.getpid(): continue   
@giampaolo
Copy link
Owner

giampaolo commented Jun 10, 2018 via email

@wrishel
Copy link
Author

wrishel commented Jun 10, 2018

Giampaolo, here is the full traceback. I'm embarrassed that I didn't think to do this in the first place.

As always I appreciate how promptly you reply.

Wes

psutil-debug> proc_regionfilename() failed
Traceback (most recent call last):
  File "/Users/Wes/Dropbox/Programming/Python/etpruncnt2.1/lib/gen/queue_coordinator/kill_coordinator.py", line 39, in <module>
    coords = find_all_coordinators()
  File "/Users/Wes/Dropbox/Programming/Python/etpruncnt2.1/lib/gen/queue_coordinator/kill_coordinator.py", line 22, in find_all_coordinators
    pinfo = proc.as_dict()
  File "/Users/Wes/.virtualenvs/ppyy/lib/python2.7/site-packages/psutil/__init__.py", line 527, in as_dict
    ret = meth()
  File "/Users/Wes/.virtualenvs/ppyy/lib/python2.7/site-packages/psutil/__init__.py", line 1112, in memory_maps
    it = self._proc.memory_maps()
  File "/Users/Wes/.virtualenvs/ppyy/lib/python2.7/site-packages/psutil/_psosx.py", line 330, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/Wes/.virtualenvs/ppyy/lib/python2.7/site-packages/psutil/_psosx.py", line 571, in memory_maps
    return cext.proc_memory_maps(self.pid)
OSError: [Errno 22] Invalid argument

@giampaolo
Copy link
Owner

Duplicate of #1209.

giampaolo added a commit that referenced this issue Jun 14, 2018
giampaolo added a commit that referenced this issue Jun 14, 2018
(OSX) wrapper around task_for_pid()
fix #1181, fix #1209, fix #1291
@wiggin15
Copy link
Collaborator

@giampaolo I'm still getting this with the latest code. #1209 seems to deal with zombie processes and task_for_pid but this issue seems to be about the running process and proc_regionfilename:

~ python -c "import psutil; print psutil.Process().memory_maps()"
psutil-debug> proc_regionfilename() failed
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/arnony/git/external/psutil/psutil/__init__.py", line 1111, in memory_maps
    it = self._proc.memory_maps()
  File "/Users/arnony/git/external/psutil/psutil/_psosx.py", line 476, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/arnony/git/external/psutil/psutil/_psosx.py", line 717, in memory_maps
    return cext.proc_memory_maps(self.pid)
OSError: [Errno 22] Invalid argument

Please consider reopening this issue.

This fails multiple tests when running make test and I'm even getting a segfault at test_memory_maps. There are some things about the implementation of memory_maps on macOS that don't look right, for example:

  • if info.is_submap is True, we increase the depth but also increase the address later in the loop, instead of querying the same address with the increased depth
  • We don't really handle the case where vm_region_recurse_64 returns != KERN_SUCCESS. We only print a debug log and continue.
  • There is an if that checks ((sizeof(buf)) <= 0 - but sizeof(buf) is constant.

I wrote a small program that goes over the memory mapping to see where the problem is here. I'm also getting "errno 22" on the last regions.

@giampaolo
Copy link
Owner

Mmm segfault is not good. I'm not familiar with that code:
cf18ae528
@whitlockjc Jeremy, any idea?

@giampaolo giampaolo reopened this Jul 29, 2018
@giampaolo
Copy link
Owner

What's even worse this time is that travis nor I can reproduce it. :-\

@giampaolo giampaolo changed the title proc.as_dict() fails when retrieving your own process OSX: memory_maps() segfaults or raise EINVAL Feb 26, 2019
@giampaolo
Copy link
Owner

@wiggin15 I think I may have fixed this as of #1436. Could you give it a try?

@wiggin15
Copy link
Collaborator

I'm still getting the same thing: OSError: [Errno 22] Invalid argument when running with python and segmentation fault when running through make test, and also when running with ipython...

Please see program I wrote here - it seems that proc_regionfilename starts failing at some point during "regular" traversal with this isolated code and I can't tell why. The vmmap tool successfully traverses the memory mappings of the process, but it's probably not using proc_regionfilename for all the regions. See for example:

Output from test program above program (snipped to where it starts failing with "errno 22"):

000000011c004000-000000011c038000 (depth=0) /usr/lib/dyld (errno=0 ret=13)
000000011c038000-000000011c061000 (depth=0) /usr/lib/dyld (errno=0 ret=13)
00007fcc0ac00000-00007fcc0ad00000 (depth=0) /usr/lib/dyld (errno=22 ret=13)
00007fcc0b000000-00007fcc0b800000 (depth=0) /usr/lib/dyld (errno=22 ret=13)

And snipped output from vmmap of the same process for these addresses:

__DATA                 000000011bfff000-000000011c004000 [   20K    20K    20K     0K] rw-/rwx SM=COW          /usr/lib/dyld
__DATA                 000000011c004000-000000011c038000 [  208K    32K    32K     0K] rw-/rwx SM=PRV          /usr/lib/dyld
MALLOC_TINY            00007fcc0ac00000-00007fcc0ad00000 [ 1024K    20K    20K     0K] rw-/rwx SM=PRV          DefaultMallocZone_0x10e359000
MALLOC_SMALL           00007fcc0b000000-00007fcc0b800000 [ 8192K    12K    12K     0K] rw-/rwx SM=PRV          DefaultMallocZone_0x10e359000

@giampaolo
Copy link
Owner

I see. Yes, I tried your program and I can't get errno=22 (EINVAL) on OSX 10.11.6 .

@giampaolo
Copy link
Owner

@wiggin15 I've found this:
https://searchcode.com/file/133107236/libr/debug/p/native/maps/darwin.c
...which calls proc_regionfilename on a certain condition:

		if (info.max_protection!=0 && !contiguous) {
                        ...
			int ret = proc_regionfilename (dbg->pid, address,
				module_name, sizeof (module_name));

On my machine info.max_protection is always 0 so proc_regionfilename is always called. Maybe on yours is not? This is the info structure:

struct vm_region_submap_info_64 {
        vm_prot_t               protection;     /* present access protection */
        vm_prot_t               max_protection; /* max avail through vm_prot */
        vm_inherit_t            inheritance;/* behavior of map/obj on fork */
        memory_object_offset_t  offset;         /* offset into object/map */
        unsigned int            user_tag;       /* user tag on map entry */
        unsigned int            pages_resident; /* only valid for objects */
        unsigned int            pages_shared_now_private; /* only for objects */
        unsigned int            pages_swapped_out; /* only for objects */
        unsigned int            pages_dirtied;   /* only for objects */
        unsigned int            ref_count;       /* obj/map mappers, etc */
        unsigned short          shadow_depth;   /* only for obj */
        unsigned char           external_pager;  /* only for obj */
        unsigned char           share_mode;     /* see enumeration */
        boolean_t               is_submap;      /* submap vs obj */
        vm_behavior_t           behavior;       /* access behavior hint */
        vm32_object_id_t        object_id;      /* obj/map name, not a handle */
        unsigned short          user_wired_count;
        unsigned int            pages_reusable;
};

Basically what I am suggesting is trying to reverse engineer this thing in order to understand under what conditions we should avoid calling proc_regionfilename.

@wiggin15
Copy link
Collaborator

I'm afraid that doesn't work either - info.max_protection is not 0 for any region in the process.
I'm also trying to figure out how to fix this but no luck so far... I tried to run "dtruss" to see if I can peek into what vmmap calls and what's causing the segfault, but there are some security limitations on this command so it didn't help.

This doesn't reproduce on our continuous-integration mac machine, which runs macOS 10.12.1, but it does reproduce on more development mac stations in my team, running different types of mac machines and macOS 10.14.3.

@giampaolo
Copy link
Owner

I see. Thanks for looking into this. This reminds me of open_files() on Windows (but worse). Also memory_maps on OSX only works for the current process and the children spawned by it, even as root. For anything else we get AccessDenied, which AFAICT is a first in psutil. If we can't fix it then I think I am gonna remove memory_maps on OSX in the next major version. I am not thrilled at the idea, but getting a segfault just for calling as_dict() is not acceptable.

@wiggin15 wiggin15 reopened this Feb 27, 2019
@giampaolo
Copy link
Owner

giampaolo commented Feb 27, 2019

We have 2 issues here: the segfault and the compatibility breakage. I think a reasonable way to approach this is the following:

  • remove the C code and raise a deprecation warning + access denied for all processes right now in 5.6.0 (next release)
  • remove the API in 6.0.0

@giampaolo
Copy link
Owner

Done in b31f3bf. Closing this out.

@giampaolo
Copy link
Owner

On a second thought also raising AccessDenied for the current process is backward incompatible. Returning an empty list is too. It looks like the only way forward is to remove this API in 5.6.0 (which is a major version BTW).

@giampaolo
Copy link
Owner

OK, memory_maps() on OSX is gone. While removing it I realized the unit-test on OSX was disabled, meaning this has been broken for a very long time. Considering the unsolvable [1], long standing and critical segfault, I consider the breakage of backward compatibility an acceptable cost. A similar case happened to open_files() on Windows which can return an incomplete result but doesn't segfault.

[1] vmmap source code is closed source, proc_regionfilename is undocumented (perhaps even broken), no solution was found

nehaljwani added a commit to regro-cf-autotick-bot/pycryptodomex-feedstock that referenced this issue Mar 24, 2019
nehaljwani added a commit to regro-cf-autotick-bot/pycryptodomex-feedstock that referenced this issue Mar 24, 2019
Call to .memory_maps() fails on Azure pipelines with:

    Traceback (most recent call last):
        ...
        return cext.proc_memory_maps(self.pid)
    OSError: [Errno 22] Invalid argument

Also, support for memory_maps() has been removed for macOS starting
psutil-v5.6.x xref: giampaolo/psutil#1291
nehaljwani added a commit to regro-cf-autotick-bot/pycryptodomex-feedstock that referenced this issue Mar 24, 2019
Call to .memory_maps() fails on Azure pipelines with:

    Traceback (most recent call last):
        ...
        return cext.proc_memory_maps(self.pid)
    OSError: [Errno 22] Invalid argument

Also, support for memory_maps() has been removed for macOS starting
psutil-v5.6.x xref: giampaolo/psutil#1291
nlevitt added a commit to nlevitt/psutil that referenced this issue Apr 9, 2019
* origin/master: (150 commits)
  Linux / CPU freq, fixes giampaolo#1481
  improve pmap.py script
  reuse ps.py script in psutil.test()
  move get_terminal_size() in _compat.py
  improve ps.py script
  improve ps.py script
  move bytes2human() into psutil._common and reused it from scripts dir
  fix windows failure re. py 2 vs. 3
  fix linux tests
  fix giampaolo#1474: fix formatting of psutil.tests() which mimicks 'ps aux' output
  give CREDITS for giampaolo#1480
  remove outdated tests
  Fix read access violation in psutil.cpu_count(logical=False) (giampaolo#1480)
  update doc
  update doc
  [Win] Process IO priority constants + high priority (giampaolo#1479 / giampaolo#1476)
  don't fail if there are not prev failed tests
  fix giampaolo#1478: add make command to re-run tests failed on last run
  [Win] return value is not properly handled for undocumented NT* Windows APIs. (giampaolo#1477)
  fix error on py 2.7 where OSError doesn't always have winerror attribute
  update HISTORY for giampaolo#1475
  properly check OSError.winerror
  refactor ionice() on Linux
  refactor ionice() on Linux
  ionice test refactoring
  giampaolo#1404: fix regression not returning CPUs > 9
  give CREDITS to Daniel Beer for giampaolo#1471
  Fix spurious exception when iterating processes on Solaris (giampaolo#1471)
  give CREDITS for giampaolo#1470
  Fix corner case when /etc/mtab doesn't exist and procfs=/proc (giampaolo#1470)
  update DEVNOTES
  Typo fixed (giampaolo#1469)
  giampaolo#1458: implement colors on Windows
  update HISTORY / CREDITS, fix some C warnings
  Make uptime type consitent to fix boot time error. (giampaolo#1225)
  fix giampaolo#1463: cpu_distribution.py script is broken
  update issue template
  issue giampaolo#1404 / linux / phys CPUs count
  Big docfix (giampaolo#1464)
  Big docfix (giampaolo#1464)
  Make tests invariant to LANG setting (giampaolo#1462)
  test runner: show errors on KeyboardInterrupt
  test runner refactoring (avoid code duplication)
  Coloured tests (giampaolo#1459)
  pre-release
  [Windows] calculate USS memory by using NtQueryVirtualMemory (giampaolo#1453)
  fix version highlighting in docs/index (giampaolo#1455)
  fix version highlighting in README (giampaolo#1454)
  run win specific tests twice as fast
  test refactoring
  test: avoid failing at import time
  mention how to run tests in INSTALL guide
  giampaolo#1448: fix Wine support due to missing rtlIpv6AddressToStringA
  update HISTORY
  Fix giampaolo#1329: [AIX] disable some functions based on availability in libperfstat (giampaolo#1349)
  bump up version, fix some doc issues
  fix ResourceWarning
  pre-release
  fix giampaolo#1447: we weren't use @wrap_exceptions around oneshot() (doh\!)
  update doc + change git hook location
  update doc
  make pre-release checks/install src dist in a venv
  add new make command to check tar.gz sanity
  move doc; rephrase it a bit
  add issue templates for 'bug' and 'enhancement' types
  remove issue template commited by accident
  Update issue templates
  giampaolo#1291: (BACKWARD-INCOMPATIBLE) remove memory_maps() on OSX
  Restore Win-7 support on GIT master (5.5.1 was OK) (giampaolo#1446)
  try to fix ntext.h
  restore previous def
  fix compiler warning
  fix compiler warning
  fix compiler warning
  fix compiler warning
  take defs from PH
  set proper  SYSTEM_PROCESS_INFORMATION struct from PH
  fix compilation warnings
  giampaolo#1398 / win / cmdline: call NtQueryInformationProcess twice, the first time to get the right buf size (ProcessHacker does this)
  update doc
  better print formatting for print scripts
  fix giampaolo#1442: use python3 as Makefile default
  appveyor: run print scripts after tests
  highlight top 6 slowest calls
  add printerr() and exit() to shared utils module
  add arg parser for ad script
  introduce a new scriptsutils.py private module shared by all internal utils + refactor print_access_speed.py script
  giampaolo#1291 / OSX: mark memory_maps() as deprecated and make it alwats raise AccessDenied
  OSX memory_maps() - add error handling
  add script for to benchmark API calls
  move access_denied script
  _assert_alive() refactor (linux)
  refactor
  fix NetBSD: Process.connections() may return incomplete results if using oneshot() giampaolo#1439
  add win tests related to send_signal(CTRL_C_EVENT) giampaolo#1227
  fix win test
  fix win tests
  fix backslash warnings
  refactor README a bit
  #fix 1438: do not return any parent() for PID 0 + update doc
  ...
@bwoodsend
Copy link

I'm able to get around this by subprocess calling lsof -p $pid instead of vmmap, and parsing the output. I'm currently implementing it for another project put I'd rather add this implementation to psutil. Would you allow me to do this?

Below is the info yielded by parsing lsof of Python's PID.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python3.7 1380 runner cwd DIR 1,4 352 12893113052 /Users/runner/work/pyinstaller-hook-factory/pyinstaller-hook-factory
python3.7 1380 runner txt REG 1,4 4192140 12893124030 /usr/local/miniconda/bin/python3.7
python3.7 1380 runner txt REG 1,4 19980 12893124065 /usr/local/miniconda/lib/python3.7/lib-dynload/_heapq.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 51860 12893124099 /usr/local/miniconda/lib/python3.7/lib-dynload/math.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 106992 12893124060 /usr/local/miniconda/lib/python3.7/lib-dynload/_datetime.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 16436 12893124074 /usr/local/miniconda/lib/python3.7/lib-dynload/_posixsubprocess.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 33728 12893124106 /usr/local/miniconda/lib/python3.7/lib-dynload/select.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 92988 12893124082 /usr/local/miniconda/lib/python3.7/lib-dynload/_socket.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 37476 12893124111 /usr/local/miniconda/lib/python3.7/lib-dynload/zlib.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 100996 12891728561 /usr/local/miniconda/lib/libz.1.2.11.dylib
python3.7 1380 runner txt REG 1,4 95464 12893124046 /usr/local/miniconda/lib/python3.7/lib-dynload/_bz2.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 40320 12893124068 /usr/local/miniconda/lib/python3.7/lib-dynload/_lzma.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 154396 12891728467 /usr/local/miniconda/lib/liblzma.5.dylib
python3.7 1380 runner txt REG 1,4 16204 12893124098 /usr/local/miniconda/lib/python3.7/lib-dynload/grp.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 46612 12893132255 /usr/local/miniconda/lib/python3.7/site-packages/psutil/_psutil_osx.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 15524 12893132256 /usr/local/miniconda/lib/python3.7/site-packages/psutil/_psutil_posix.cpython-37m-darwin.so
python3.7 1380 runner txt REG 1,4 1568368 1152921500312899604 /usr/lib/dyld
python3.7 1380 runner 0 PIPE 0xceb898bcad398973 16384
python3.7 1380 runner 1 PIPE 0xe88391d08adf368f 65536 ->0xff6e2faf57a81150
python3.7 1380 runner 2 PIPE 0x590b43d7c0ca7945 16384 ->0x37d3f8e5c729229e
python3.7 1380 runner 3 PIPE 0x8f906b3253d4a8b6 16384 ->0x7fc1f421784607ca

Online you may see speed concerns regarding lsof but they don't apply to lsof -p for a specific process.

@giampaolo
Copy link
Owner

Hello. I'd rather not, sorry. In general I prefer to refrain from parsing subprocess output and maintain it internally. I've been there and I know it's painful. Plus memory_maps() it's sort of low-levelish and underused.

@bwoodsend
Copy link

Plus memory_maps() it's sort of low-levelish and underused.

I use it. It's the closest I've found to a cross platform way to test shared library dependencies at run time. And I've already written the parser and it's not too ugly 😛.

@P403n1x87
Copy link

I have an example of how to iterate over the memory maps in Austin. It certainly isn't documentation, but perhaps can be of help?

https://github.com/P403n1x87/austin/blob/master/src/mac/py_proc.h

It would be nice to have memory_maps back into psutil. I don't have easy access to macOS so I can't really try to do this myself at the moment, unfortunately.

@Maxwell175
Copy link

it would be great to be able to rely on this function across all platforms.

@mzpqnxow
Copy link

I'm in a situation where I need to get a list of memory mappings- specifically, I need the file-backed mappings (e.g. the various dynamic libraries and other resources, like Java jar files- I need the full path to each on a per-process basis)

I currently do this on Linux with psutil.Process.memory_maps() but don't have this field on MacOS (darwin 20.6.0 xnu-7195.141.14) and

I understand that shelling out to lsof is not an ideal solution, but that is the only way to get this information right now as far as I'm aware (excluding the few snippets of C code here, and of course in the lsof source itself- though the MacOS memory mapping bit might be closed, I suspect it isn't)

It's probably clear, but this is the kind of data I need:

com.apple 77215             _driverkit  txt       REG                1,5     292928 1152921500311993973 /System/DriverKit/usr/lib/system/libsystem_pthread.dylib
com.apple 77215             _driverkit  txt       REG                1,5     157488 1152921500311993975 /System/DriverKit/usr/lib/system/libsystem_trace.dylib
com.apple 77215             _driverkit  txt       REG                1,5     461936 1152921500311993948 /System/DriverKit/usr/lib/libc++abi.dylib
com.apple 77216             _driverkit  txt       REG                1,5     401680 1152921500311993946 /System/DriverKit/usr/lib/libc++.dylib
com.apple 77216             _driverkit  txt       REG                1,5     140272 1152921500311993944 /System/DriverKit/usr/lib/libSystem.dylib
com.apple 77216             _driverkit  txt       REG                1,5     194336 1152921500311993951 /System/DriverKit/usr/lib/system/libcompiler_rt.dylib
com.apple 77216             _driverkit  txt       REG                1,5    1490784 1152921500311993953 /System/DriverKit/usr/lib/system/libcorecrypto.dylib
com.apple 77216             _driverkit  txt       REG                1,5    1009216 1152921500311993955 /System/DriverKit/usr/lib/system/libdispatch.dylib
com.apple 77216             _driverkit  txt       REG                1,5     383824 1152921500311993957 /System/DriverKit/usr/lib/system/libdyld.dylib
com.apple 77216             _driverkit  txt       REG                1,5     120608 1152921500311993959 /System/DriverKit/usr/lib/system/libmacho.dylib
com.apple 77216             _driverkit  txt       REG                1,5     138864 1152921500311993961 /System/DriverKit/usr/lib/system/libsystem_blocks.dylib
com.apple 77216             _driverkit  txt       REG                1,5    1292064 1152921500311993963 /System/DriverKit/usr/lib/system/libsystem_c.dylib

I was hoping that p.open_files() would do what I needed, but no such luck- it is woefully incomplete and only shows descriptors/files that were opened as plain file descriptors, not any that were implicitly opened and mapped by the MACH loader

I'm bummed because I switched away from a custom Python solution that married ps and lsof data together, hoping that psutil would save all my problems- but I'm a bit stuck now and may have to go back to it

Anyway, just a long-winded way to say "+1", I guess ;)

Thanks for the work you all do on this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants