support AIX

[wiggin15]

I see it's in the TODO file, filed under "lower priority", but I wanted a ticket to be open because we actually encountered this:

root@aixio003 (AIX 7.1.0.0) ➜  ~  easy_install psutil                                                                                                                                                                                                         15-03-08 17:46
Searching for psutil
Reading http://pypi.python.org/simple/psutil/
Best match: psutil 2.2.1
Downloading https://pypi.python.org/packages/source/p/psutil/psutil-2.2.1.tar.gz#md5=1a2b58cd9e3a53528bb6148f0c4d5244
Processing psutil-2.2.1.tar.gz
Writing /tmp/easy_install-GaoOfH/psutil-2.2.1/setup.cfg
Running psutil-2.2.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-GaoOfH/psutil-2.2.1/egg-dist-tmp-9X2hV0
error: Setup script exited with platform aix7 is not supported

[giampaolo]

Yes, AIX is not supported. Considering that this is a niche platform and that I don't have access to any AIX box it's very unlikely this is gonna get pushed forward, unless somebody steps in and decides to contribute a patch.

[wiggin15]

I started working on this here: Infinidat@2d037eb
There are still some missing functions, but if I open a pull request will you consider merging?

[mffrench]

will be interested on that too... :)

[giampaolo]

@wiggin15 what's the status of that branch of yours? What APIs are not implemented?

[wiggin15]

We're using it here at my company. I think it is mostly stable. There are 10 API functions not implemented (5 of them are methods of the Process class) but I added documentation and disabled them for AIX in the latest commit. See https://github.com/Infinidat/psutil/commits/aix

(Please note that the AIX branch is also rebased on top of the branch in pull request #610 which is also not merged yet)

[giampaolo]

I'm sorry but IMO 10 unsupported APIs means this is not acceptable/ready for inclusion yet. It would mean over complicating the existent code and tests with ifs, unittest.skipIf(AIX) etc., and make the doc less readable.
Please note that it is also OK to parse the output of CLI tools if the functionality is too hard to implement in C. We currently do this to provide swap_memory stats on Solaris, if I'm not mistaken.

[giampaolo]

@wiggin15: looking back at this. I wonder if it's possible to install AIX on a laptop / virtualbox so that I can try to take a look at this myself. https://github.com/Infinidat/psutil/blob/aix/psutil/_psutil_aix.c looks similar to the Solaris implementation, which is encouraging.

[wiggin15]

I'm afraid AIX runs only on PowerPC architecture, and virtualization technologies like VirtualBox/VMWare etc. can't run it (IBM have their own virtualization solution called VIOS, but it requires a PowerPC server). I don't know how I can help with that.

[wiggin15]

I updated the branch and it's now down to only two missing methods.
I'm not sure it's ready for inclusion yet, though. Some tests are failing (most because AIX behaves differently than the tests expect but not only), and it should be run and tested on other environments than what I'm running - AIX environments are sometimes very different.

[giampaolo]

What are the 2 missing methods? Could you paste a test result?

[wiggin15]

I committed this file with the missing methods and a list of failing tests:
https://github.com/Infinidat/psutil/blob/aix/psutil/TODO.aix
Here's a full output of a test run from now:
https://gist.github.com/wiggin15/fa7469f5638bb52acc8e

[giampaolo]

This is very good progress. Kudos to you!
Some comments.

    def test_children(self):
        p = psutil.Process()
>       self.assertEqual(p.children(), [])
E       AssertionError: Lists differ: [<psutil.Process(pid=19267672,... != []

This is weird as there shouldn't be any subprocess (parent of os.getpid()) at the beginning of the test, meaning reap_children() function, which is run before any uniitest, didn't work as expected.
Also the PID is very high (19267672) which suggests that Process.ppid() implementation might be the issue (Process.children() relies entirely on it).
The fact that reap_children (which also relies on children()) didn't work looks like a confirmation of that fact. Are we sure this return Py_BuildValue("KKKdiiiK", (unsigned long long) info.pr_ppid, ... is correct in terms of types? Is pr_ppid an unsigned long long? Try to search its definition under /usr/include.

TestProcess.test_cmdline long args are cut from cmdline in /proc/pid/psinfo and getargs (flaky)

Does ps show the full cmdline? If there's no way to obtain the full cmdline I would add a // XXX cmdline is truncated to 15 chars comment in the C source code and update doc in accordance.

E AttributeError: 'Process' object has no attribute 'memory_maps'

If there's no way to implement memory_maps() I suggest to just remove support for it, skip the test as in @unittest.skipIf(AIX, "not supported on AIX") and update the doc.

E AttributeError: 'Process' object has no attribute 'num_ctx_switches'

Same here: remove support, skip test and update doc.

    def test_pid_0(self):
         ...
>       self.assertTrue(p.name())
E       AssertionError: '' is not true

Does ps show a name? If so you can enforce that name in pure python as in if self.pid == 0: return "name" otherwise it is fine to skip the test.

def test_prog_w_funky_name(self):

You can simply skip this test.

[wiggin15]

Thanks for the feedback.

1. I also noticed that reap_children isn't working and that this causes most of the "flaky" test failures. I started to debug and noticed that the children aren't stopped because they return False from is_running even though they are running - because of this check in is_running:
   return self == Process(self.pid)
   the new process instance ident doesn't match the one we have. One of them (I don't remember which) has 0 in the creation time (which is read from procfs). I don't know how that can be. I double checked the procfs struct definitions, they're ok. The pids are correct (most pids on the system are very high - 6-8 digits). I didn't investigate further. AIX is one weird system.
2. ps doesn't show the full command line (and procfs and getargs don't either). It's not exactly that the command line is truncated. Instead, very long arguments (like python code we pass after "-c") are "skipped". I can document this and skip this test.
3. pid 0 has a name in ps ("swapper") but it doesn't show in procfs and other API functions I tried. You're right, maybe I can just check this in the pure python code and return that name there.
4. test_prog_w_funky_name - I can skip this test. This needs to be skipped in Solaris too, by the way. It doesn't work there either.
5. I can skip the 3 tests that fail because of the two missing methods too. I already updated the docs about what's missing and more relevant notes.

[giampaolo]

I started to debug and noticed that the children aren't stopped because they return False from
is_running even though they are running - because of this check in is_running:
return self == Process(self.pid)
the new process instance ident doesn't match the one we have.

So you may want to look at how you determine process creation time because that is what is used to identify a process uniquely over time. It may be that creation time has a C overflow / typing issue similarly to ppid.

One of them (I don't remember which) has 0 in the creation time (which is read from procfs).

Maybe it's PID 0? In that case I recommend to return the boot time (instead of 0) in pure python as in if self.pid == 0: return boot_time()

The pids are correct (most pids on the system are very high - 6-8 digits). I didn't investigate further.
AIX is one weird system.

I see. OK.

test_prog_w_funky_name - I can skip this test. This needs to be skipped in Solaris too, by the way. It
doesn't work there either.

In this case skip the tests on both platforms.

[dmurphy18]

@wiggin15 wondering if there is any further work for support of AIX or has the work been dropped ?