Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow specifying /proc and other needed folders #558

Closed
wojons opened this issue Dec 15, 2014 · 13 comments

Comments

@wojons
Copy link

@wojons wojons commented Dec 15, 2014

The use case of this is running monitoring tools or scripts built around psutil that run inside docker continents. With docker the admin can specify that the /proc of the host machine is read only and mounted in a location on the sub continent the issue for this is that /proc room the host maybe /proc_host and then you need to read things from there for monitoring. An issue that comes up is CoreOS and a few other docker OS dont have package managers so there is no way to really install tools there its expected that the user installs the tools in a docker container.

@giampaolo

This comment has been minimized.

Copy link
Owner

@giampaolo giampaolo commented Dec 15, 2014

Mmm... it's not clear to me why anyone would want to change the default /proc location as I suppose a lot of different system tools makes the assumption that /proc lives in "/proc", not only psutil. Is this something you do explicitly or is it Docker which does it?

@wojons

This comment has been minimized.

Copy link
Author

@wojons wojons commented Dec 15, 2014

Yes I agree in 99.99% of the time /proc is /proc. So CoreOS which is a OS made for docker it does not have a package manager so there is no way to install new software on the server it assumes that everything you need to do can run from another docker container. And if that docker continer needs access to /proc for getting monitoring data then you mount that /proc from the host OS to some other location on the container.

@giampaolo

This comment has been minimized.

Copy link
Owner

@giampaolo giampaolo commented Dec 15, 2014

Yes. I would say that another possible approach here in case /proc is /proc_host or whatever is to just create a symlink to /proc. This has the downside that it requires root privileges though. Is that something you can do by default with CoreOS? What is the default /proc location in CoreOS? I'm wondering whether that can be determined at runtime.

@wojons

This comment has been minimized.

Copy link
Author

@wojons wojons commented Dec 15, 2014

Not sure if you used docker before or openvz. but each container is acting like its own operating system but shares the same kernel as the host but uses cgroups to limit the scope of processes that the container can see. I can mount the /proc easily to /proc_host but its hard to get the system to work correctly if i mount /proc of the host over the /proc of the container.

@giampaolo giampaolo changed the title Allow speifying /proc and other needed folders Allow specifying /proc and other needed folders Aug 7, 2015
@ggueret

This comment has been minimized.

Copy link

@ggueret ggueret commented Oct 30, 2015

+1 with the following use case : mount the host rootfs in a docker container and get some metrics from psutil, like cAdvisor "--volume=/:/rootfs:ro".

@giampaolo

This comment has been minimized.

Copy link
Owner

@giampaolo giampaolo commented Oct 30, 2015

OK, I'm convinced there's an use case which justifies this change.
The problem is how to expose this feature in terms of API.
The Linux implementation lives in the psutil._pslinux namespace:
https://github.com/giampaolo/psutil/blob/master/psutil/_pslinux.py
...which psutil imports as "_psplatform":

from . import _pslinux as _psplatform

The first thing which pops into my mind is to define a PROCFS_PATH constant in _pslinux.py which the user can change like this:

import psutil
psutil._psplatform.PROCFS_PATH

I'm not too happy about this though because we are accessing a private module. I can't think of any other solution though (also renaming the module doesn't fully convince me).
Thoughts? @jloden @josiahcarlson

@josiahcarlson

This comment has been minimized.

Copy link

@josiahcarlson josiahcarlson commented Oct 31, 2015

I hate to say it, but what about a module property in psutil? Dirty, but then you aren't hiding it in a private module. Make it do the right thing on all platforms (like an exception on platforms where it doesn't exist).

Or don't even make it a property, make it a constant, and make psutil._psplatform.* check psutil.PROCFS_PATH whenever it is looking for the value, and it can just be a global like any other. You can import psutil in _psplatform functions that need it, or reference it directly from sys.modules based on __name__. Stuff the module property in _psplatform.PROCFS_PATH, which checks the attribute in psutil.

GIampaolo, you always have fun programming problems :)

P.S. At least one of those suggestions is a joke, can you guess which one(s)? Trick or Treat ;)

@giampaolo

This comment has been minimized.

Copy link
Owner

@giampaolo giampaolo commented Oct 31, 2015

Thanks for chiming in Josiah. =)

I hate to say it, but what about a module property in psutil?

I knew you'd say that. =)

You can import psutil in _psplatform functions that need it.

Not possible. This would create a circular import because it's psutil (aka psutil/__init__.py) which imports _pslinux.

reference it directly from sys.modules based on __name__

This is more interesting. I have tried to import psutil; psutil.PROCFS_PATH = 'foo' then print sys.modules['psutil'].PROCFS_PATH from _pslinux.py but it doesn't get updated (it prints the old value). Is this what you were suggesting? Or was this the joke? =)

Another possitility would be to provide a psutil.set_procfs_path() function in psutil namespace which changes _psplatform.PROCFS_PATH value. Perhaps this is the cleanest solution, even though I hate to provide a brand new function just for this. :-(

@josiahcarlson

This comment has been minimized.

Copy link

@josiahcarlson josiahcarlson commented Oct 31, 2015

Funnily enough, the module property actually answers the problem pretty well, and wasn't the joke.

If sys.modules['psutil'].PROCFS_PATH doesn't get updated when you update psutil.PROCFS_PATH, then you really need to check the __name__ and __file__ values in sys.modules['psutil'] and _pslinux.py, because I bet they are referring to different packages.

When I pull down the git repo, do a quick make, add the attribute to __init__.py, and add a simple function to _pslinux.py:

def do_stuff():
    import sys
    return sys.modules[__name__.rpartition('.')[0]].PROCFS_PATH

I get the following behavior:

>>> import psutil
>>> psutil.PROCFS_PATH
'/proc'
>>> from psutil import _pslinux
>>> _pslinux.do_stuff()
'/proc'
>>> psutil.PROCFS_PATH = '/other/path'
>>> _pslinux.do_stuff()
'/other/path'

So that one should work, if you are using the right __name__ in your references.

You can import psutil in _psplatform functions that need it.

Not possible. This would create a circular import because it's psutil (aka psutil/init.py) which imports _pslinux.

Importing psutil inside a function (like I do with sys above in the do_stuff() function) is one way to avoid exceptions caused by circular imports without injecting stuff into __builtins__ or using sys.modules as I did above in do_stuff(). The sys.modules reference may be the cleanest without resorting to a module property.

@giampaolo

This comment has been minimized.

Copy link
Owner

@giampaolo giampaolo commented Oct 31, 2015

If sys.modules['psutil'].PROCFS_PATH doesn't get updated when you update psutil.PROCFS_PATH,
then you really need to check the name and file values in sys.modules['psutil'] and
_pslinux.py, because I bet they are referring to different packages.

This was the issue indeed. Cool! I will stick with psutil.PROCFS_PATH as to me it seems the cleanest API from the user standpoint. Thanks Josiah. I will not tell you I owe you another beer because with all the beers I owe you'd probably need a liver transplant. :D

giampaolo added a commit that referenced this issue Oct 31, 2015
@josiahcarlson

This comment has been minimized.

Copy link

@josiahcarlson josiahcarlson commented Oct 31, 2015

Always happy to help. Save the beer money for your travel, and we'll run into each other again somewhere between LA and Turin :D

@ggueret

This comment has been minimized.

Copy link

@ggueret ggueret commented Nov 26, 2015

Works well, thanks a lot : )

@giampaolo

This comment has been minimized.

Copy link
Owner

@giampaolo giampaolo commented Nov 26, 2015

;-)

sethp-jive added a commit to sethp-jive/psutil that referenced this issue Dec 3, 2015
The very excellent work that went in under giampaolo#558
gave us a psutil that can be used on systems with a moved /proc.

However, on some systems (e.g. test harnesses, containers) may not
have any files mounted at /proc at all. In these cases, psutil
fails at import time with an IOError trying to access /proc/stat.

This patch removes the import-time dependency on a live procfs
mounted at /proc, and validates the behavior of some of the
details we were hoping to read at import time (i.e. cpu % stats).
sethp-jive added a commit to sethp-jive/psutil that referenced this issue Dec 3, 2015
The very excellent work that went in under giampaolo#558
gave us a psutil that can be used on systems with a moved /proc.

However, on some systems (e.g. test harnesses, containers) may not
have any files mounted at /proc at all. In these cases, psutil
fails at import time with an IOError trying to access /proc/stat.

This patch removes the import-time dependency on a live procfs
mounted at /proc, and validates the behavior of some of the
details we were hoping to read at import time (i.e. cpu % stats).
sethp-jive added a commit to sethp-jive/psutil that referenced this issue Dec 3, 2015
The very excellent work that went in under giampaolo#558
gave us a psutil that can be used on systems with a moved /proc.

However, on some systems (e.g. test harnesses, containers) may not
have any files mounted at /proc at all. In these cases, psutil
fails at import time with an IOError trying to access /proc/stat.

This patch removes the import-time dependency on a live procfs
mounted at /proc, and validates the behavior of some of the
details we were hoping to read at import time (i.e. cpu % stats).
sethp-jive added a commit to sethp-jive/psutil that referenced this issue Dec 3, 2015
The very excellent work that went in under giampaolo#558
gave us a psutil that can be used on systems with a moved /proc.

However, on some systems (e.g. test harnesses, containers) may not
have any files mounted at /proc at all. In these cases, psutil
fails at import time with an IOError trying to access /proc/stat.

This patch removes the import-time dependency on a live procfs
mounted at /proc, and validates the behavior of some of the
details we were hoping to read at import time (i.e. cpu % stats).
sethp-jive added a commit to sethp-jive/psutil that referenced this issue Dec 3, 2015
The very excellent work that went in under giampaolo#558
gave us a psutil that can be used on systems with a moved /proc.

However, on some systems (e.g. test harnesses, containers) may not
have any files mounted at /proc at all. In these cases, psutil
fails at import time with an IOError trying to access /proc/stat.

This patch removes the import-time dependency on a live procfs
mounted at /proc, and validates the behavior of some of the
details we were hoping to read at import time (i.e. cpu % stats).
@giampaolo giampaolo closed this Feb 3, 2016
@giampaolo giampaolo referenced this issue Sep 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.