Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prototype for Cygwin support module (not ready to merge) #998

Closed
wants to merge 86 commits into from

Commits on Mar 14, 2017

  1. Copy the full SHA
    6f5aa56 View commit details
    Browse the repository at this point in the history
  2. Initial addition of a cygwin platform module.

    Some common utilitiy functions were moved out of _pslinux into _common.
    
    Most of cygwin's Process implementation is directly copied from the
    linux Proces, minus a few very Linux-specific parts.  I hope to
    eliminate the code duplication later with further refactoring.
    
    Most tests fail due to some key Process methods not having
    implementations yet (mostly because their implementations for Linux do
    not work on Cygwin).
    embray committed Mar 14, 2017
    Copy the full SHA
    79a579f View commit details
    Browse the repository at this point in the history
  3. First pass at implementing Process.create_time for Cygwin

    I left this unimplemented originally just because I wanted to take the
    time to understand what I was doing before continuing.  Turns out it can
    be implemented almost the same as on Linux, except that the process
    start time is in the 22nd field of /proc/<pid>/stat instead of the 21st.
    I can't find enough documentation for Cygwin's /proc filesystem, but I
    confirmed this in the source.  Other than that the semantics are the
    same as on Linux.
    
    So much of this was copied agained from the Linux module, but I will
    refactor later.
    embray committed Mar 14, 2017
    Copy the full SHA
    8150510 View commit details
    Browse the repository at this point in the history
  4. Copy the full SHA
    1f765b1 View commit details
    Browse the repository at this point in the history
  5. Added initial virtual_memory() and swap_memory() implementations.

    These are much simpler than on Linux, and effectively the same as the
    existing Windows implementations of these functions, though easier to
    implement on Cygwin via /proc/meminfo
    embray committed Mar 14, 2017
    Copy the full SHA
    6f9ee0d View commit details
    Browse the repository at this point in the history
  6. Add previously missing pfullmem namedtuple, which some of the tests

    expect to find.
    
    Currently it is identical to pmem; will return later to seeing what
    additional memory info we can provide on Cygwin
    embray committed Mar 14, 2017
    Copy the full SHA
    893ebe0 View commit details
    Browse the repository at this point in the history
  7. Fix broken Process.uids and Process.gids on Cygwin

    Later this should be able to share its implementation with the Linux
    module but we'll come back around to that.
    embray committed Mar 14, 2017
    Copy the full SHA
    9753c8c View commit details
    Browse the repository at this point in the history
  8. Add cpu_count_logical(), cpu_count_physical(), and cpu_stats() for

    Cygwin
    
    Copied almost verbatim from the Linux module.
    embray committed Mar 14, 2017
    Copy the full SHA
    d652487 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2017

  1. Compile at least the arch/windows/process_info.c module into the

    cygwin module so that the functions in it can be reused there.
    
    This required some tweaks to ntextapi.h, since a few of the
    types defined there are already included in the winimpl.h included
    with Cygwin.
    
    This includes are own implementation of PyErr_SetFromWindowsErr which
    is not defined in Python for Cygwin.
    
    TODO: Because this PyErr_SetFromWindowsErr is lifted, only slightly
    modified, from the Python source it should include a copyright notice
    to the effect.  This implementation probably does not work quite right
    on Python 2 either--it should return an actual WindowsError there.
    embray committed Mar 15, 2017
    Copy the full SHA
    8fb812a View commit details
    Browse the repository at this point in the history
  2. Add proc_cpu_affinity_get/set. This is copied exactly from the Windows

    module for now.
    
    Will refactor later.
    embray committed Mar 15, 2017
    Copy the full SHA
    e7f548f View commit details
    Browse the repository at this point in the history
  3. Copy the full SHA
    6c1f0b4 View commit details
    Browse the repository at this point in the history
  4. Initial disk_partitions() implementation for cygwin module.

    Copied exactly from the Linux module and seems to work; will refactor
    later.
    embray committed Mar 15, 2017
    Copy the full SHA
    cfca2e3 View commit details
    Browse the repository at this point in the history
  5. Initial implementation of disk_usage() for cygwin.

    Using the posix implementation directly seems to just work.
    embray committed Mar 15, 2017
    Copy the full SHA
    2e482bc View commit details
    Browse the repository at this point in the history
  6. Initial implementation of net_if_addrs() for the cygwin module.

    The only obvious issue currently is that rather than the human-friendly interface
    names returned by this function in the normal Windows implementation, the
    interface names are instead some UUID looking things for each interface (the same
    as appear in the registry in various places to represent the interface).  This is
    fine for now, but not ideal.  Should be able to map these to friendlier names.
    embray committed Mar 15, 2017
    Copy the full SHA
    3c8fa07 View commit details
    Browse the repository at this point in the history
  7. Copy the full SHA
    99d628f View commit details
    Browse the repository at this point in the history
  8. Move the py2_strencode utility from the windows module into common,

    renaming it to 'encode', for symmetry with the 'decode' utility.
    
    This utility may also be useful in the Cygwin module, and is otherwise
    not Windows-specific, having more to do with Py2/3 compatibility.
    embray committed Mar 15, 2017
    Copy the full SHA
    9bce4b9 View commit details
    Browse the repository at this point in the history
  9. Very rough first pass at implementing net_if_stats for Cygwin

    This is mostly copied from the Windows module with a few small tweaks, but compiling this
    is a bit of a pain and somewhat fragile.  Will have to come back to this for a closer look
    later, but for now I just want to get all of the API implemented as best I can.
    embray committed Mar 15, 2017
    Copy the full SHA
    e456738 View commit details
    Browse the repository at this point in the history
  10. Initial implementation of net_io_counters for the cygwin module.

    As with net_if_stats, this is a bit of a mess right now to compile, and copies from the Windows module, but will clean up later.
    embray committed Mar 15, 2017
    Copy the full SHA
    e330394 View commit details
    Browse the repository at this point in the history
  11. Initial implementation of the net_connections() function for cygwin

    Once again, mostly copied from the Windows module, with much refactoring needed.
    embray committed Mar 15, 2017
    Copy the full SHA
    4386116 View commit details
    Browse the repository at this point in the history
  12. Added a simple wrapper for cygwin_conv_path() from Cygwin's (very lim…

    …ited) API.
    
    This will allow methods that return paths to return Cygwin paths instead of native Windows paths.
    I have yet to go through and determine all the places this should be used, but there are probably
    at least a few.
    embray committed Mar 15, 2017
    Copy the full SHA
    f196006 View commit details
    Browse the repository at this point in the history
  13. Add conversion from Cygwin PID to native Windows PID.

    Whenever calling a C extension method that calls out to the Windows API make sure to use
    the Windows PID.  This should fix a lot of issues where the wrong PID was being used.
    embray committed Mar 15, 2017
    Copy the full SHA
    3613ad8 View commit details
    Browse the repository at this point in the history
  14. Initial implementation of Process.memory_maps for the cygwin module.

    Most of this is copied from the windows module with a few minor tweaks, necessitating later
    refactoring.
    
    This is also the first method to make proper use of the new winpath_to_cygpath function.
    embray committed Mar 15, 2017
    Copy the full SHA
    b04e40e View commit details
    Browse the repository at this point in the history
  15. Fix cygwin's Process.cpu_affinity_get/set by copying the Python-level…

    … implementations from the windows module (rather than the linux module) but using the winpid.
    embray committed Mar 15, 2017
    Copy the full SHA
    87ce780 View commit details
    Browse the repository at this point in the history
  16. Initial implementation of disk_io_counters for Cygwin; copied almost …

    …verbatim from the windows module.
    embray committed Mar 15, 2017
    Copy the full SHA
    6f72eda View commit details
    Browse the repository at this point in the history
  17. Initial implementation of the psutil.users() function for the cygwin …

    …module.
    
    This is copied mostly from the Linux module, though it needs to use the utmpx APIs.  This seems to 'work'
    insofar as the API is available on cygwin, but it's not clear to me that the information returned is fully
    correct or useful.  Will return to this if needed.
    embray committed Mar 15, 2017
    Copy the full SHA
    de0e4fe View commit details
    Browse the repository at this point in the history
  18. Copy the full SHA
    c6a6c71 View commit details
    Browse the repository at this point in the history
  19. Added initial support for Process.environ, again copied from the wind…

    …ows module
    
    (The linux module's method of looking in /proc doesn't work on cygwin's more limited /proc namespace)
    embray committed Mar 15, 2017
    Copy the full SHA
    b685c57 View commit details
    Browse the repository at this point in the history
  20. Initial cygwin support for Process.io_counters and Process.num_ctx_sw…

    …itches, mostly
    
    supported by the proc_info function copied from the windows module.
    embray committed Mar 15, 2017
    Copy the full SHA
    6b7fe0f View commit details
    Browse the repository at this point in the history
  21. Add cygwin interface for Process.num_threads

    (This is just a thin wrapper around proc_info())
    embray committed Mar 15, 2017
    Copy the full SHA
    00fa137 View commit details
    Browse the repository at this point in the history
  22. Copy the full SHA
    c3afcaf View commit details
    Browse the repository at this point in the history
  23. Initial cygwin support for Process.threads

    Copy/pasted from windows module, so this carries the usual admonition about refactoring.
    embray committed Mar 15, 2017
    Copy the full SHA
    93469a5 View commit details
    Browse the repository at this point in the history
  24. The child process search that occurs if ppid_map is not available sho…

    …uld fail gracefully on AccessDenied errors.
    
    If we get an AccessDenied trying to get the ppid of a process, it's *probably* not a child process of the process in
    question anyways.
    embray committed Mar 15, 2017
    Copy the full SHA
    818a9d2 View commit details
    Browse the repository at this point in the history
  25. Use the Process.memory_info implementation from the windows module

    instead of going through Cygwin's /proc which turns out to be a bit buggy and
    for this purpose.
    
    This allows us to expose more information anyways such as USS in memory_info_full,
    just unfortunate that the Cygwin interface can't be relied on here.
    embray committed Mar 15, 2017
    Copy the full SHA
    f1302cb View commit details
    Browse the repository at this point in the history
  26. Turns out there does not (for now) appear to be any straightforward way

    to get the process environment on Cygwin.  So we remove the
    functionality for now.
    
    See https://cygwin.com/ml/cygwin/2016-11/msg00205.html
    embray committed Mar 15, 2017
    Copy the full SHA
    18e0e14 View commit details
    Browse the repository at this point in the history
  27. After researching the subject extensively, I've determined it's not p…

    …ossible to enumerate UNIX sockets on Cygwin
    
    Cygwin emulates UNIX sockets as a INET socket.  Although it does list them in /proc/<pid>/fd as some 'socket:[N]' like on Linux, the 'N' here is only of internal meaning to Cygwin (this is true on Linux as well), but there is nothing resembling /proc/net/unix or anything else that allows one to map that number N to an actual socket file (such a file *does* exist, and in theory we could find it be enumerating ALL files on the system until it is found, but this is not practical).  I can't find any other obvious way to do that.  The only practical solution would be if Cygwin had an equivalent to /proc/net/unix.
    embray committed Mar 15, 2017
    Copy the full SHA
    e8ae762 View commit details
    Browse the repository at this point in the history
  28. Add Cygwin version info constants--these are not used anywhere yet bu…

    …t may prove useful later
    embray committed Mar 15, 2017
    Copy the full SHA
    d04cb55 View commit details
    Browse the repository at this point in the history
  29. allow process_iter to skip processes on AccessDenied, which can happe…

    …n when trying to query processes for ppid that one does not have access to (TODO: it may be possible to get the ppid by some other means in this case though)
    embray committed Mar 15, 2017
    Copy the full SHA
    2b521f8 View commit details
    Browse the repository at this point in the history
  30. Copy the full SHA
    c56e69f View commit details
    Browse the repository at this point in the history
  31. Copy the full SHA
    8eb5392 View commit details
    Browse the repository at this point in the history
  32. Copy the full SHA
    d8f0551 View commit details
    Browse the repository at this point in the history
  33. Copy the full SHA
    833b132 View commit details
    Browse the repository at this point in the history
  34. Copy the full SHA
    328eb4e View commit details
    Browse the repository at this point in the history
  35. Adds a new ps() interface for the posix tests that better clarifies the

    platform-specific differences between how ps is called.
    
    Also implements significant support for Cygwin, where the built-in ps
    lacks many of the features of ps on other systems.
    
    This fixes a number of test errors on Cygwin (it also exposes a few new
    failures but those will be addressed separately).
    embray committed Mar 15, 2017
    Copy the full SHA
    ac2e269 View commit details
    Browse the repository at this point in the history
  36. Cygwin does not necessarily create utmp entries when you start a logi…

    …n terminal, so
    
    'who' can return empty--this fix allows the test to work even when there are no utmp entries.
    embray committed Mar 15, 2017
    Copy the full SHA
    fff8b2b View commit details
    Browse the repository at this point in the history
  37. Fix a corner case that can occur when a getmntent() entry's mount point

    does not actually exist.
    
    This happened to be affecting my test runs.
    embray committed Mar 15, 2017
    Copy the full SHA
    3cc14ac View commit details
    Browse the repository at this point in the history
  38. Turns out the process creation time from /proc/<pid>/stat cannot be

    relied on in Cygwin, due to a rounding error there which can cause it to
    return fluctuating results over time.
    
    This is really bad because process creation time is part of the
    'identity' psutil util uses for a process when comparing it for equality
    to another process.
    
    This once again copies code from the windows module into the cygwin
    module, which I intend to refactor later.  I also plan to address the
    upstream bug in Cygwin, but until that's fixed this workaround is
    needed.
    embray committed Mar 15, 2017
    Copy the full SHA
    03f3b76 View commit details
    Browse the repository at this point in the history
  39. A TODO note for later

    embray committed Mar 15, 2017
    Copy the full SHA
    099de6c View commit details
    Browse the repository at this point in the history
  40. It seems this test is having the same problem on Cygwin as on plain

    Windows with processes not being truly destroyed quickly enough.
    embray committed Mar 15, 2017
    Copy the full SHA
    3d629ef View commit details
    Browse the repository at this point in the history
  41. Make one small Cygwin-specific ifdef in psutil_net_if_addrs to suppor…

    …t getting the
    
    'friendly name' as opposed to the fixed UUID-based name of an interface.  This is more consistent
    with the other network APIs on Windows/Cygwin which return the interface friendly names
    embray committed Mar 15, 2017
    Copy the full SHA
    3b8ff83 View commit details
    Browse the repository at this point in the history
  42. Change net_if_stats() to return the same network interfaces as net_if…

    …_addrs()--kind of a hack for now.
    embray committed Mar 15, 2017
    Copy the full SHA
    5e9d940 View commit details
    Browse the repository at this point in the history
  43. Make this test pass more reliably by allowing the ps output to includ…

    …e one extra process for ps itself.
    
    (I'm not sure how this test worked on Linux since it normally includes ps itself too...)
    embray committed Mar 15, 2017
    Copy the full SHA
    1616f2a View commit details
    Browse the repository at this point in the history
  44. As written, this test could not work when usernames contain spaces (w…

    …hich is not uncommon on Cygwin)
    embray committed Mar 15, 2017
    Copy the full SHA
    70beaca View commit details
    Browse the repository at this point in the history
  45. Prevent this test from failing on Cygwin on processes running at a hi…

    …gher privilege than the test runner
    embray committed Mar 15, 2017
    Copy the full SHA
    f83f21b View commit details
    Browse the repository at this point in the history
  46. Copy the full SHA
    6886cf8 View commit details
    Browse the repository at this point in the history
  47. Copy the full SHA
    ca5b620 View commit details
    Browse the repository at this point in the history
  48. Skip this test on Cygwin, as there is no reliable way currently to ma…

    …p from net_connections() results to file descriptors
    embray committed Mar 15, 2017
    Copy the full SHA
    6e7c59e View commit details
    Browse the repository at this point in the history
  49. Improved handling for a buggy (?) state that can occur on Cygwin wher…

    …e a process is in the zombie state but its /proc/<pid>/stat file can no longer be read.
    
    See https://www.cygwin.com/ml/cygwin/2017-02/msg00187.html
    embray committed Mar 15, 2017
    Copy the full SHA
    31649ff View commit details
    Browse the repository at this point in the history
  50. Copy the full SHA
    d3d50cb View commit details
    Browse the repository at this point in the history
  51. Return fewer interfaces from net_io_counters on Cygwin, for parity wi…

    …th the functionality in the Windows interface.
    
    This allows the test_nic_names test from the Windows tests to be reused
    and to pass on Cygwin.
    embray committed Mar 15, 2017
    Copy the full SHA
    d6e1cfa View commit details
    Browse the repository at this point in the history
  52. More fixes for this test to work on Cygwin.

    I'm worried it still might be a bit flakey though.
    embray committed Mar 15, 2017
    Copy the full SHA
    eede620 View commit details
    Browse the repository at this point in the history
  53. Copy the full SHA
    105ab20 View commit details
    Browse the repository at this point in the history
  54. Copy the full SHA
    d213e0c View commit details
    Browse the repository at this point in the history
  55. Copy the full SHA
    6c24010 View commit details
    Browse the repository at this point in the history
  56. Copy the full SHA
    0d09339 View commit details
    Browse the repository at this point in the history
  57. Copy the full SHA
    b694138 View commit details
    Browse the repository at this point in the history
  58. Add support for sensors_battery on Cygwin

    (Copied directly from the Windows module for now)
    embray committed Mar 15, 2017
    Copy the full SHA
    f164492 View commit details
    Browse the repository at this point in the history
  59. Copy the full SHA
    f5862fb View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2017

  1. Copy the full SHA
    fd982d2 View commit details
    Browse the repository at this point in the history
  2. Skip these tests on AppVeyor (under Cygwin only)

    They are failing persistently due to fork errors.  No matter how I rebase the affected DLLs it seems
    to have problems on these. I have given up trying to find out why for now.
    embray committed Mar 23, 2017
    Copy the full SHA
    2091d1e View commit details
    Browse the repository at this point in the history
  3. Support /dev/consN on Cygwin

    embray committed Mar 23, 2017
    Copy the full SHA
    ae73082 View commit details
    Browse the repository at this point in the history
  4. Copy the full SHA
    ad8c6c8 View commit details
    Browse the repository at this point in the history
  5. Copy the full SHA
    03f6be8 View commit details
    Browse the repository at this point in the history
  6. Copy the full SHA
    839e186 View commit details
    Browse the repository at this point in the history
  7. Copy the full SHA
    a8d99fa View commit details
    Browse the repository at this point in the history
  8. More fork errors on Cygwin, now immediately upon running setup.py bui…

    …ld (this did not happen before).
    
    Try a rebaseall immediately before setup.py build.
    embray committed Mar 23, 2017
    Copy the full SHA
    2eeb7c4 View commit details
    Browse the repository at this point in the history
  9. Copy the full SHA
    8f30340 View commit details
    Browse the repository at this point in the history
  10. Strip .exe explicitly

    (previously this would mistakenly convert something like 'python2.7' to just 'python2'--if the latter
    is a symlink this can break things)
    embray committed Mar 23, 2017
    Copy the full SHA
    ae339b0 View commit details
    Browse the repository at this point in the history
  11. Retry get_test_subprocess

    It seems that random fork errors can occur here on Cygwin, but not *always*. Retrying usually works.
    embray committed Mar 23, 2017
    Copy the full SHA
    af327d3 View commit details
    Browse the repository at this point in the history
  12. Disable testing Python 3 on Cygwin in AppVeyor for now until I can spend

    more time figuring out what the problem is.
    embray committed Mar 23, 2017
    Copy the full SHA
    ee8be22 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2017

  1. Copy the full SHA
    20113a2 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2017

  1. Better way to get the major/minor version of Windows, without relying…

    … on running a subprocess
    
    or cmd.exe.
    
    Doesn't return the build number unfortunately, but we're not using that right now anyways.
    embray committed Apr 26, 2017
    Copy the full SHA
    a33c43c View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2017

  1. Copy the full SHA
    13969d5 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2017

  1. Copy the full SHA
    5df6009 View commit details
    Browse the repository at this point in the history
  2. A much cleaner, better functioning, and more Cygwin-ish approach to e…

    …rror
    
    handling from Python wrappers for Windows APIs.
    
    Instead of attempting to reimplement PyErr_SetFromWindowsError, on Cygwin we just
    replace those calls with a macro wrapping PyErr_SetFromErrno which uses the
    cygwin_internal API to map Windows error codes to POSIX error codes just in the same
    way Cygwin would.
    
    This appears to work correctly (for example trying to get info on a non-existent
    process returns errno ESRCH (No such process).
    embray committed Nov 6, 2017
    Copy the full SHA
    5c24482 View commit details
    Browse the repository at this point in the history
  3. As noted in the comment, cygpid_to_winpid and winpid_to_cygpid can have

    their pure Python implementations replaced with more efficient (and less
    racy) C implementations.
    embray committed Nov 6, 2017
    Copy the full SHA
    a455308 View commit details
    Browse the repository at this point in the history
  4. First move of a utility common to both the windows and cygwin modules.

    Although I considered adding a new module for this purpose, for now it
    suffices to use the _common module for this (albeit a bit hackishly).
    
    In this case the C wrapper for this function makes sense to put in
    arch/windows.
    embray committed Nov 6, 2017
    Copy the full SHA
    339514d View commit details
    Browse the repository at this point in the history