Skip to content

Loading…

Support environments without a home dir or writable file system #1824

Merged
merged 17 commits into from

5 participants

@mgiuca-google

This patch makes it possible to import and use matplotlib on a system where the user has no $HOME directory, and furthermore, on which there is no writable file system. It does this by:

  • Falling back to creating a temporary config directory instead of raising RuntimeError if the user has no $HOME directory.
  • Falling back to not writing to the config dir if it is not possible to create a temporary directory. (In this case, all caching mechanisms, including texmanager, font_manager and finance, are disabled.)

Partial fix for Issue #1823.

@pelson pelson commented on an outdated diff
lib/matplotlib/__init__.py
@@ -215,13 +215,8 @@ def _is_writable_dir(p):
try: p + '' # test is string like
except TypeError: return False
try:
- t = tempfile.TemporaryFile(dir=p)
- try:
- t.write(ascii('1'))
- finally:
- t.close()
+ return os.access(p, os.W_OK)
except OSError: return False
@pelson Matplotlib Developers member
pelson added a note

Would you mind separating this line into 2.

@mdboom Matplotlib Developers member
mdboom added a note

I believe there is a corner case in which os.access will not do the right thing, hence the convoluted approach you see here... just navigated the git history to no avail. At the very least, this should use realpath of p to resolve any symbolic links.

@pelson: Done.

@mdboom: Yeah the code was written in 2005 and wasn't really changed since. I assume the corner case is what the Python manual for os.access hints at: "Note: I/O operations may fail even when access() indicates that they would succeed, particularly for operations on network filesystems which may have permissions semantics beyond the usual POSIX permission-bit model."

I don't know what this means in practice. Is there a concrete example of access() returning True for a directory but you can't create files in it? Creating and deleting a temporary file seems like a messy thing to do if there is a function that tells you directly whether a directory is writable. But I am happy to put it back. The App Engine corner case is a failure in the other direction: TemporaryFile succeeds even though the file system is not writable (since TemporaryFile just creates a StringIO, ignoring the dir argument). So it should be fine to have both checks: os.access() to check if it is theoretically writable (which works for App Engine), followed up by TemporaryFile to test if it is writable in practice. Should I go ahead and do this?

At the very least, this should use realpath of p to resolve any symbolic links.

I don't think so. From man 2 access:

If pathname is a symbolic link, it is dereferenced.

I tested this on Linux and it seemed to work (created a non-writable directory and a writable symlink to the directory; os.access with W_OK on the symlink returns False).

@mdboom Matplotlib Developers member

Done. Also I added an extra check that it is actually a directory!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on the diff
lib/matplotlib/__init__.py
((6 lines not shown))
- :see: http://mail.python.org/pipermail/python-list/2005-February/263921.html
+ :see: http://mail.python.org/pipermail/python-list/2005-February/325395.html
@pelson Matplotlib Developers member
pelson added a note

:+1:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/__init__.py
((11 lines not shown))
try:
- path=os.path.expanduser("~")
+ path = os.path.expanduser("~")
except:
@pelson Matplotlib Developers member
pelson added a note

I'd rather remove this bare except. Would you mind adding the specific exception(s) types that might occur?

OK. Well I hope I didn't miss any... but the documentation for both os.path.expanduser and pwd do not mention any exceptions. I dug through the code and it doesn't look like any exceptions are possible on normal (non-App-Engine) platforms. (pwd.getpwnam can raise KeyError, but that is caught in expanduser.) So I think we could actually remove that try-except entirely, except on App Engine, the pwd module is sadly not present, so calling expanduser raises ImportError. I'm changing the code to explicitly catch ImportError with a comment that this is for App Engine. I hope this is okay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/__init__.py
((25 lines not shown))
else:
- raise RuntimeError('please define environment variable $HOME')
+ if os.path.isdir(path):
+ return path
+ for evar in ('HOME', 'USERPROFILE', 'TMP'):
+ try:
@pelson Matplotlib Developers member
pelson added a note

The try except seems superfluous to me, I'd change this to:

path = os.environ.get(evar)
if path is not None and os.path.isdir(path):
    return path

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on the diff
lib/matplotlib/__init__.py
((8 lines not shown))
- if os.path.exists(p):
- if not _is_writable_dir(p):
- return _create_tmp_config_dir()
- else:
- if not _is_writable_dir(h):
- return _create_tmp_config_dir()
- from matplotlib.cbook import mkdirs
- mkdirs(p)
+ if os.path.exists(p):
+ if not _is_writable_dir(p):
+ return _create_tmp_config_dir()
+ else:
+ if not _is_writable_dir(h):
+ return _create_tmp_config_dir()
+ from matplotlib.cbook import mkdirs
+ mkdirs(p)
@pelson Matplotlib Developers member
pelson added a note

There must be a subtlety that I'm missing here - I can't see the difference between cbook.mkdirs and os.makedirs. Any ideas?

@mdboom Matplotlib Developers member
mdboom added a note

The difference is that cbook.mkdirs doesn't complain if the directory already exists.

@WeatherGod Matplotlib Developers member

cbook.mkdirs() is even safer than os.makedirs(). Imagine two processes, one needing to make A/B, and the other needing to make A/C. There is a race condition in os.makedirs() where one of the two processes will fail to make their directory because the exception was thrown while dealing with the parent directory.

So why does it use os.makedirs above? (It's a little bit above the context shown in this patch, but the first block of get_configdir calls os.makedirs, whereas the second block calls cbook.mkdirs.) Can we change it so it consistently calls cbook.mkdirs?

@mdboom Matplotlib Developers member
mdboom added a note

Yes -- let's change this to cbook.mkdirs in both cases.

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/__init__.py
((9 lines not shown))
home = get_home()
- oldname = os.path.join( home, '.matplotlibrc')
- if os.path.exists(oldname):
- configdir = get_configdir()
- newname = os.path.join(configdir, 'matplotlibrc')
- print("""\
+ configdir = get_configdir()
+ if home:
+ oldname = os.path.join( home, '.matplotlibrc')
@pelson Matplotlib Developers member
pelson added a note

Please remove the space before home.

Done (and the rest).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/__init__.py
@@ -641,18 +649,31 @@ def matplotlib_fname():
WARNING: Old rc filename ".matplotlibrc" found in working dir
and and renamed to new default rc file name "matplotlibrc"
(no leading"dot"). """, file=sys.stderr)
- shutil.move('.matplotlibrc', 'matplotlibrc')
+ try:
+ shutil.move('.matplotlibrc', 'matplotlibrc')
+ except IOError as e:
+ print("WARNING: File could not be renamed: %s" % e, file=sys.stderr)
@pelson Matplotlib Developers member
pelson added a note

Could you turn this into a real warning.

Done (and the rest).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/__init__.py
((9 lines not shown))
home = get_home()
- oldname = os.path.join( home, '.matplotlibrc')
- if os.path.exists(oldname):
- configdir = get_configdir()
- newname = os.path.join(configdir, 'matplotlibrc')
- print("""\
+ configdir = get_configdir()
+ if home:
+ oldname = os.path.join( home, '.matplotlibrc')
+ if os.path.exists(oldname):
+ if configdir is not None:
+ newname = os.path.join(configdir, 'matplotlibrc')
+ print("""\
@pelson Matplotlib Developers member
pelson added a note

Again, would you mind turning this into a real warning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/__init__.py
((23 lines not shown))
WARNING: Old rc filename "%s" found and renamed to
new default rc file name "%s"."""%(oldname, newname), file=sys.stderr)
- shutil.move(oldname, newname)
+ try:
+ shutil.move(oldname, newname)
+ except IOError as e:
+ print("WARNING: File could not be renamed: %s" % e,
@pelson Matplotlib Developers member
pelson added a note

Ditto.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/__init__.py
((23 lines not shown))
WARNING: Old rc filename "%s" found and renamed to
new default rc file name "%s"."""%(oldname, newname), file=sys.stderr)
- shutil.move(oldname, newname)
+ try:
+ shutil.move(oldname, newname)
+ except IOError as e:
+ print("WARNING: File could not be renamed: %s" % e,
+ file=sys.stderr)
+ else:
+ print("""\
@pelson Matplotlib Developers member
pelson added a note

Ditto.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/__init__.py
((23 lines not shown))
WARNING: Old rc filename "%s" found and renamed to
new default rc file name "%s"."""%(oldname, newname), file=sys.stderr)
- shutil.move(oldname, newname)
+ try:
+ shutil.move(oldname, newname)
+ except IOError as e:
+ print("WARNING: File could not be renamed: %s" % e,
+ file=sys.stderr)
+ else:
+ print("""\
+WARNING: Could not rename old rc file "%s": a suitable configuration directory
+ could not be found.""" % oldname, file=sys.stderr)
fname = os.path.join( os.getcwd(), 'matplotlibrc')
@pelson Matplotlib Developers member
pelson added a note

Would you mind removing a newline before this line & getting rid of the space before os.getcwd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/__init__.py
@@ -665,8 +686,9 @@ def matplotlib_fname():
if os.path.exists(fname):
return fname
- fname = os.path.join(get_configdir(), 'matplotlibrc')
- if os.path.exists(fname): return fname
+ if configdir is not None:
+ fname = os.path.join(configdir, 'matplotlibrc')
+ if os.path.exists(fname): return fname
@pelson Matplotlib Developers member
pelson added a note

Could you remove this newline and turn the if statement into 2 lines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/finance.py
((5 lines not shown))
- cachename = os.path.join(cachedir, md5(url).hexdigest())
- if os.path.exists(cachename):
- fh = open(cachename)
- verbose.report('Using cachefile %s for %s'%(cachename, ticker))
+ # Cache the finance data if there is a writable cache directory.
+ if cachedir is not None:
+ if cachename is None:
+ cachename = os.path.join(cachedir, md5(url).hexdigest())
+ if os.path.exists(cachename):
+ fh = open(cachename)
+ verbose.report('Using cachefile %s for %s'%(cachename, ticker))
+ else:
+ mkdirs(cachedir)
+ urlfh = urlopen(url)
+
+ fh = open(cachename, 'wb')
@pelson Matplotlib Developers member
pelson added a note

Use the context manager form.

Fixed in PR #1834.

Note: I also caught and fixed my own bug here: if cachename is supplied, it should not skip caching if there is no cachedir.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/texmanager.py
((11 lines not shown))
if os.path.exists(oldcache):
- # FIXME raise proper warning
- print("""\
+ if texcache is not None:
+ # FIXME raise proper warning
+ print("""\
@pelson Matplotlib Developers member
pelson added a note

Please turn this into a warning.

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/texmanager.py
((18 lines not shown))
WARNING: found a TeX cache dir in the deprecated location "%s".
Moving it to the new default location "%s".""" % (oldcache, texcache),
- file=sys.stderr)
- shutil.move(oldcache, texcache)
- mkdirs(texcache)
+ file=sys.stderr)
+ try:
+ shutil.move(oldcache, texcache)
+ except IOError as e:
+ print("WARNING: File could not be renamed: %s" % e,
@pelson Matplotlib Developers member
pelson added a note

Ditto.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/texmanager.py
((18 lines not shown))
WARNING: found a TeX cache dir in the deprecated location "%s".
Moving it to the new default location "%s".""" % (oldcache, texcache),
- file=sys.stderr)
- shutil.move(oldcache, texcache)
- mkdirs(texcache)
+ file=sys.stderr)
+ try:
+ shutil.move(oldcache, texcache)
+ except IOError as e:
+ print("WARNING: File could not be renamed: %s" % e,
+ file=sys.stderr)
+ else:
+ print("""\
@pelson Matplotlib Developers member
pelson added a note

Ditto.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson pelson commented on an outdated diff
lib/matplotlib/texmanager.py
@@ -141,6 +157,11 @@ class TexManager:
def __init__(self):
+ if not self.texcache:
@pelson Matplotlib Developers member
pelson added a note

Lets be consistent. Could you turn this into if self.texcache is None.

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pelson
Matplotlib Developers member

Thanks @mgiuca-google - this looks good to me. Thanks for putting this work in.

I'm guessing you've tried this out on G.A.E.? That would be the best way to give me confidence that this is doing the right thing for those circumstances.

@pelson pelson was assigned
@pelson
Matplotlib Developers member

Also, would you mind adding details in the whats new section (making a sing and a dance that Google app engine should now work). If you're keen I'd also love to see a write-up at http://matplotlib.org/faq/howto_faq.html#matplotlib-in-a-web-application-server about how to go about setting up your App engine instance for mpl?

@mdboom mdboom commented on an outdated diff
lib/matplotlib/texmanager.py
@@ -141,6 +157,11 @@ class TexManager:
def __init__(self):
+ if not self.texcache:
+ raise RuntimeError(
+ ('Cannot create TexManager, as there is no cache directory '
+ 'available'))
@mdboom Matplotlib Developers member
mdboom added a note

This probably doesn't affect Google AppEngine (which I suspect doesn't have a TeX installation anyway), but it would be nice to continue to be able to get TeX snippets without them being cached in this case.

From my reading of the code, this whole module doesn't work if there is no writable directory. The so-called "texcache" is not really a cache at all, it's also a working directory, where intermediate files are generated and processed with dvipng. So even if we assume a system that does have a TeX installation and the ability to launch external commands, but does not have any writable file system, we won't be able to run these commands. (This isn't a simple matter of disabling caching.)

Correct me if I'm wrong about that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@WeatherGod WeatherGod commented on an outdated diff
lib/matplotlib/finance.py
@@ -178,22 +184,26 @@ def fetch_historical_yahoo(ticker, date1, date2, cachename=None,dividends=False)
d2[0], d2[1], d2[2], ticker, g)
- if cachename is None:
- cachename = os.path.join(cachedir, md5(url).hexdigest())
- if os.path.exists(cachename):
- fh = open(cachename)
- verbose.report('Using cachefile %s for %s'%(cachename, ticker))
+ # Cache the finance data if there is a writable cache directory.
+ if cachedir is not None:
+ if cachename is None:
+ cachename = os.path.join(cachedir, md5(url).hexdigest())
+ if os.path.exists(cachename):
+ fh = open(cachename)
+ verbose.report('Using cachefile %s for %s'%(cachename, ticker))
+ else:
+ mkdirs(cachedir)
@WeatherGod Matplotlib Developers member

Perhaps we need to import cbook.mkdirs() at the top. I think there is a possibility that this function could be not defined at this point.

cbook.mkdirs is imported at the top.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@mdboom
Matplotlib Developers member

For the font cache: ideally if .matplotlib is not writable, it should fall back to a font cache created at install time and installed in mpl-data/fonts -- that way the user would still get fast access to the fonts that ship with matplotlib. I don't know if we need that level of finesse in this first round of PR, but it would certainly deal with 90% of use cases and have them be as fast as ever.

@mgiuca-google

@pelson:

Thanks @mgiuca-google - this looks good to me. Thanks for putting this work in.

No problem, thanks for looking at these GAE-specific patches. This should make our (GAE Python team's) lives easier in the future.

I'm guessing you've tried this out on G.A.E.? That would be the best way to give me confidence that this is doing the right thing for those circumstances.

Yeah (well, the dev appserver). I basically wrote this patch by running it on the dev appserver and then fixing things until it worked. This, combined with my other two PRs (#1825 and #1826), allows Matplotlib to run on the dev appserver. There is a caveat, though: because of the recent change to Matplotlib to remove pytz and dateutil, and require them as external dependencies, I had to hack the devappserver code to whitelist those modules. I think that is something we'll have to fix when the time comes to upgrade to Matplotlib 1.3.0, not something that you can fix on your end.

Also, would you mind adding details in the whats new section (making a sing and a dance that Google app engine should now work). If you're keen I'd also love to see a write-up at http://matplotlib.org/faq/howto_faq.html#matplotlib-in-a-web-application-server about how to go about setting up your App engine instance for mpl?

Yeah, I'd be happy to write a short "matplotlib with Google App Engine" section there. The general advice in that section is pretty much all you need for App Engine (though obviously you can't use fig.savefig('test.png'); you need to write to stdout instead). I'll address that in a separate PR. Note that Matplotlib already runs on the production servers (since I have already applied similar patches to the above on our local version); this PR just allows it to run on the dev appserver.

@mdboom:

For the font cache: ideally if .matplotlib is not writable, it should fall back to a font cache created at install time and installed in mpl-data/fonts -- that way the user would still get fast access to the fonts that ship with matplotlib. I don't know if we need that level of finesse in this first round of PR, but it would certainly deal with 90% of use cases and have them be as fast as ever.

But as far as I can tell, the font cache is not actually caching the font files, it's caching the font list. I don't think that mpl-data/fonts contains a font list file, does it? (It doesn't on my system.)

@mdboom
Matplotlib Developers member

Yes -- the fontList.cache is just a directory of the font attributes to font paths. I was suggesting that we generate a font directory at build time of only the fonts that ship with matplotlib, and use that as a fallback if we can't write a font directory at import time.

@mgiuca-google

Yes -- the fontList.cache is just a directory of the font attributes to font paths. I was suggesting that we generate a font directory at build time of only the fonts that ship with matplotlib, and use that as a fallback if we can't write a font directory at import time.

OK. Well can we do that in another PR? This PR is not going to make things worse on this front (since all of these cases are going from "won't work at all" to "won't have access to the cache".

@mdboom
Matplotlib Developers member

Right -- I was suggesting that having a default cache could be another PR.

@mgiuca-google

OK. I've rebased the branch to the current master.

@dmcdougall
Matplotlib Developers member

@mgiuca-google Looks like you need another rebase. That might be my fault; I just firehosed a bunch of PRs.

@mgiuca-google

Rebased.

(It was my fault: I conflicted with my own change in another branch!)

mgiuca-google added some commits
@mgiuca-google mgiuca-google shutil.move: Guard against IOError and print warnings instead of cras…
…hing.
8083f25
@mgiuca-google mgiuca-google matplotlib: _is_writable_dir uses os.access instead of TemporaryFile.
This is a more direct approach and avoids strange behaviour on systems that
provide a fake implementation of TemporaryFile. In particular, on Google App
Engine, TemporaryFile is equivalent to StringIO, so this would have always
succeeded, despite the fact that the directory is not actually writable.

It is also not particularly nice to go creating and deleting random files in the
user's home directory.
a697a26
@mgiuca-google mgiuca-google Matplotlib now works when the user has no home directory.
matplotlib.get_home now returns None if there is no home directory, instead of
raising RuntimeError. Updated code in several places to handle this gracefully.

matplotlib.get_configdir now returns a temporary directory if there is no home
directory, instead of raising RuntimeError.
4d65400
@mgiuca-google mgiuca-google get_configdir returns None if tempfile.gettempdir() is not available.
Previously, it would raise NotImplementedError. This is necessary on restricted
platforms such as Google App Engine that do not provide gettempdir.
21921a3
@mgiuca-google mgiuca-google Deal with all cases where get_configdir might return None.
Each case is either handled gracefully, or raises a RuntimeError.
4987dcd
@mgiuca-google mgiuca-google font_manager: Gracefully handle the case of there being no config dir.
Instead of raising RuntimeError, now avoids reading and writing the font cache.
64c797b
@mgiuca-google mgiuca-google texmanager: Gracefully handle the case of there being no config dir u…
…pon import.

Instead of raising RuntimeError, now avoids creating the cache dir. This permits
texmanager to be imported in a restricted environment such as Google App Engine.

Actually constructing a TexManager object will fail, but it can be imported.
1dbd6de
@mgiuca-google mgiuca-google finance: Gracefully handle the case of there being no config dir.
Instead of raising RuntimeError, now avoids reading and writing the cache.
1adfc85
@mgiuca-google mgiuca-google Fix formatting and other misc code tweaks. 941efd4
@mgiuca-google mgiuca-google matplotlib.get_home: Removing catch-all except blocks. ca6cd19
@mgiuca-google mgiuca-google matplotlib, texmanager: Change WARNING prints into real warnings. cc8cd1b
@mgiuca-google mgiuca-google matplotlib, texmanager: Only print the rename message if it actually …
…succeeded.
f01ebe1
@mgiuca-google mgiuca-google finance: Fixed caching when cachename is supplied.
Previously, it would not cache if there was no cachedir, even if cachename was
supplied.
018ce26
@mgiuca-google mgiuca-google matplotlib: Use cbook.mkdirs instead of os.makedirs. 6a4f1e7
@mgiuca-google mgiuca-google matplotlib: Remove catch for OSError.
os.access does not raise this exception.
4f55a27
@mgiuca-google mgiuca-google matplotlib: _is_writable_dir tests with os.access and TemporaryFile.
This fails either if the OS tells us it isn't writable, or if a file cannot be
created there.
8335773
@mgiuca-google mgiuca-google matplotlib: _is_writable_dir checks that it is a directory. 81639a1
@mgiuca-google

Hey guys, we haven't had any traction on this issue for about a month. I thought I'd wait until the 1.2.1 release before pinging.

It looked like we were almost at a consensus. Are there any more issues? I have just rebased again against the latest HEAD, and there were no conflicts.

@pelson pelson merged commit 91b1d9f into matplotlib:master

1 check passed

Details default The Travis build passed
@pelson
Matplotlib Developers member

Done. Another excellent piece of work @mgiuca-google - thanks again!

@mgiuca-google mgiuca-google deleted the mgiuca-google:no-home-dir branch
@mgiuca-google

Thanks, Phil. That's all the patches I have lined up for now. It's been great working with you.

@pelson
Matplotlib Developers member

That's all the patches I have lined up for now. It's been great working with you.

Shame! Has this been a work-driven or a hobby-driven project? We've plenty of space for people willing to get their hands dirty improving the codebase :wink: .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 16, 2013
  1. @mgiuca-google
  2. @mgiuca-google

    matplotlib: _is_writable_dir uses os.access instead of TemporaryFile.

    mgiuca-google committed
    This is a more direct approach and avoids strange behaviour on systems that
    provide a fake implementation of TemporaryFile. In particular, on Google App
    Engine, TemporaryFile is equivalent to StringIO, so this would have always
    succeeded, despite the fact that the directory is not actually writable.
    
    It is also not particularly nice to go creating and deleting random files in the
    user's home directory.
  3. @mgiuca-google

    Matplotlib now works when the user has no home directory.

    mgiuca-google committed
    matplotlib.get_home now returns None if there is no home directory, instead of
    raising RuntimeError. Updated code in several places to handle this gracefully.
    
    matplotlib.get_configdir now returns a temporary directory if there is no home
    directory, instead of raising RuntimeError.
  4. @mgiuca-google

    get_configdir returns None if tempfile.gettempdir() is not available.

    mgiuca-google committed
    Previously, it would raise NotImplementedError. This is necessary on restricted
    platforms such as Google App Engine that do not provide gettempdir.
  5. @mgiuca-google

    Deal with all cases where get_configdir might return None.

    mgiuca-google committed
    Each case is either handled gracefully, or raises a RuntimeError.
  6. @mgiuca-google

    font_manager: Gracefully handle the case of there being no config dir.

    mgiuca-google committed
    Instead of raising RuntimeError, now avoids reading and writing the font cache.
  7. @mgiuca-google

    texmanager: Gracefully handle the case of there being no config dir u…

    mgiuca-google committed
    …pon import.
    
    Instead of raising RuntimeError, now avoids creating the cache dir. This permits
    texmanager to be imported in a restricted environment such as Google App Engine.
    
    Actually constructing a TexManager object will fail, but it can be imported.
  8. @mgiuca-google

    finance: Gracefully handle the case of there being no config dir.

    mgiuca-google committed
    Instead of raising RuntimeError, now avoids reading and writing the cache.
  9. @mgiuca-google
  10. @mgiuca-google
  11. @mgiuca-google
  12. @mgiuca-google
  13. @mgiuca-google

    finance: Fixed caching when cachename is supplied.

    mgiuca-google committed
    Previously, it would not cache if there was no cachedir, even if cachename was
    supplied.
  14. @mgiuca-google
  15. @mgiuca-google

    matplotlib: Remove catch for OSError.

    mgiuca-google committed
    os.access does not raise this exception.
  16. @mgiuca-google

    matplotlib: _is_writable_dir tests with os.access and TemporaryFile.

    mgiuca-google committed
    This fails either if the OS tells us it isn't writable, or if a file cannot be
    created there.
  17. @mgiuca-google
View
152 lib/matplotlib/__init__.py
@@ -213,16 +213,28 @@ def _is_writable_dir(p):
p is a string pointing to a putative writable dir -- return True p
is such a string, else False
"""
- try: p + '' # test is string like
- except TypeError: return False
+ try:
+ p + '' # test is string like
+ except TypeError:
+ return False
+
+ # Test whether the operating system thinks it's a writable directory.
+ # Note that this check is necessary on Google App Engine, because the
+ # subsequent check will succeed even though p may not be writable.
+ if not os.access(p, os.W_OK) or not os.path.isdir(p):
+ return False
+
+ # Also test that it is actually possible to write to a file here.
try:
t = tempfile.TemporaryFile(dir=p)
try:
t.write(ascii('1'))
finally:
t.close()
- except OSError: return False
- else: return True
+ except OSError:
+ return False
+
+ return True
class Verbose:
"""
@@ -475,38 +487,42 @@ def checkdep_usetex(s):
def _get_home():
"""Find user's home directory if possible.
- Otherwise raise error.
+ Otherwise, returns None.
- :see: http://mail.python.org/pipermail/python-list/2005-February/263921.html
+ :see: http://mail.python.org/pipermail/python-list/2005-February/325395.html
@pelson Matplotlib Developers member
pelson added a note

:+1:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
"""
- path=''
try:
- path=os.path.expanduser("~")
- except:
+ path = os.path.expanduser("~")
+ except ImportError:
+ # This happens on Google App Engine (pwd module is not present).
pass
- if not os.path.isdir(path):
- for evar in ('HOME', 'USERPROFILE', 'TMP'):
- try:
- path = os.environ[evar]
- if os.path.isdir(path):
- break
- except: pass
- if path:
- return path
else:
- raise RuntimeError('please define environment variable $HOME')
+ if os.path.isdir(path):
+ return path
+ for evar in ('HOME', 'USERPROFILE', 'TMP'):
+ path = os.environ.get(evar)
+ if path is not None and os.path.isdir(path):
+ return path
+ return None
def _create_tmp_config_dir():
"""
If the config directory can not be created, create a temporary
directory.
+
+ Returns None if a writable temporary directory could not be created.
"""
import getpass
import tempfile
- tempdir = os.path.join(
- tempfile.gettempdir(), 'matplotlib-%s' % getpass.getuser())
+ try:
+ tempdir = tempfile.gettempdir()
+ except NotImplementedError:
+ # Some restricted platforms (such as Google App Engine) do not provide
+ # gettempdir.
+ return None
+ tempdir = os.path.join(tempdir, 'matplotlib-%s' % getpass.getuser())
os.environ['MPLCONFIGDIR'] = tempdir
return tempdir
@@ -518,35 +534,42 @@ def _get_configdir():
"""
Return the string representing the configuration directory.
- Default is HOME/.matplotlib. You can override this with the
- MPLCONFIGDIR environment variable. If the default is not
- writable, and MPLCONFIGDIR is not set, then
- tempfile.gettempdir() is used to provide a directory in
- which a matplotlib subdirectory is created as the configuration
- directory.
+ The directory is chosen as follows:
+
+ 1. If the MPLCONFIGDIR environment variable is supplied, choose that. Else,
+ choose the '.matplotlib' subdirectory of the user's home directory (and
+ create it if necessary).
+ 2. If the chosen directory exists and is writable, use that as the
+ configuration directory.
+ 3. If possible, create a temporary directory, and use it as the
+ configuration directory.
+ 4. A writable directory could not be found or created; return None.
"""
configdir = os.environ.get('MPLCONFIGDIR')
if configdir is not None:
if not os.path.exists(configdir):
- os.makedirs(configdir)
+ mkdirs(configdir)
if not _is_writable_dir(configdir):
return _create_tmp_config_dir()
return configdir
h = get_home()
- p = os.path.join(get_home(), '.matplotlib')
+ if h is not None:
+ p = os.path.join(h, '.matplotlib')
- if os.path.exists(p):
- if not _is_writable_dir(p):
- return _create_tmp_config_dir()
- else:
- if not _is_writable_dir(h):
- return _create_tmp_config_dir()
- from matplotlib.cbook import mkdirs
- mkdirs(p)
+ if os.path.exists(p):
+ if not _is_writable_dir(p):
+ return _create_tmp_config_dir()
+ else:
+ if not _is_writable_dir(h):
+ return _create_tmp_config_dir()
+ from matplotlib.cbook import mkdirs
+ mkdirs(p)
@pelson Matplotlib Developers member
pelson added a note

There must be a subtlety that I'm missing here - I can't see the difference between cbook.mkdirs and os.makedirs. Any ideas?

@mdboom Matplotlib Developers member
mdboom added a note

The difference is that cbook.mkdirs doesn't complain if the directory already exists.

@WeatherGod Matplotlib Developers member

cbook.mkdirs() is even safer than os.makedirs(). Imagine two processes, one needing to make A/B, and the other needing to make A/C. There is a race condition in os.makedirs() where one of the two processes will fail to make their directory because the exception was thrown while dealing with the parent directory.

So why does it use os.makedirs above? (It's a little bit above the context shown in this patch, but the first block of get_configdir calls os.makedirs, whereas the second block calls cbook.mkdirs.) Can we change it so it consistently calls cbook.mkdirs?

@mdboom Matplotlib Developers member
mdboom added a note

Yes -- let's change this to cbook.mkdirs in both cases.

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
- return p
+ return p
+
+ return _create_tmp_config_dir()
get_configdir = verbose.wrap('CONFIGDIR=%s', _get_configdir, always=False)
@@ -636,27 +659,39 @@ def matplotlib_fname():
"""
- oldname = os.path.join( os.getcwd(), '.matplotlibrc')
+ oldname = os.path.join(os.getcwd(), '.matplotlibrc')
if os.path.exists(oldname):
- print("""\
-WARNING: Old rc filename ".matplotlibrc" found in working dir
- and and renamed to new default rc file name "matplotlibrc"
- (no leading"dot"). """, file=sys.stderr)
- shutil.move('.matplotlibrc', 'matplotlibrc')
+ try:
+ shutil.move('.matplotlibrc', 'matplotlibrc')
+ except IOError as e:
+ warnings.warn('File could not be renamed: %s' % e)
+ else:
+ warnings.warn("""\
+Old rc filename ".matplotlibrc" found in working dir and and renamed to new
+ default rc file name "matplotlibrc" (no leading ".").""")
home = get_home()
- oldname = os.path.join( home, '.matplotlibrc')
- if os.path.exists(oldname):
- configdir = get_configdir()
- newname = os.path.join(configdir, 'matplotlibrc')
- print("""\
-WARNING: Old rc filename "%s" found and renamed to
- new default rc file name "%s"."""%(oldname, newname), file=sys.stderr)
-
- shutil.move(oldname, newname)
-
+ configdir = get_configdir()
+ if home:
+ oldname = os.path.join(home, '.matplotlibrc')
+ if os.path.exists(oldname):
+ if configdir is not None:
+ newname = os.path.join(configdir, 'matplotlibrc')
+
+ try:
+ shutil.move(oldname, newname)
+ except IOError as e:
+ warnings.warn('File could not be renamed: %s' % e)
+ else:
+ warnings.warn("""\
+Old rc filename "%s" found and renamed to new default rc file name "%s"."""
+ % (oldname, newname))
+ else:
+ warnings.warn("""\
+Could not rename old rc file "%s": a suitable configuration directory could not
+ be found.""" % oldname)
- fname = os.path.join( os.getcwd(), 'matplotlibrc')
+ fname = os.path.join(os.getcwd(), 'matplotlibrc')
if os.path.exists(fname): return fname
if 'MATPLOTLIBRC' in os.environ:
@@ -666,9 +701,10 @@ def matplotlib_fname():
if os.path.exists(fname):
return fname
- fname = os.path.join(get_configdir(), 'matplotlibrc')
- if os.path.exists(fname): return fname
-
+ if configdir is not None:
+ fname = os.path.join(configdir, 'matplotlibrc')
+ if os.path.exists(fname):
+ return fname
path = get_data_path() # guaranteed to exist or raise
fname = os.path.join(path, 'matplotlibrc')
View
37 lib/matplotlib/finance.py
@@ -28,7 +28,13 @@
configdir = get_configdir()
-cachedir = os.path.join(configdir, 'finance.cache')
+# cachedir will be None if there is no writable directory.
+if configdir is not None:
+ cachedir = os.path.join(configdir, 'finance.cache')
+else:
+ # Should only happen in a restricted environment (such as Google App
+ # Engine). Deal with this gracefully by not caching finance data.
+ cachedir = None
stock_dt = np.dtype([('date', object),
@@ -178,20 +184,25 @@ def fetch_historical_yahoo(ticker, date1, date2, cachename=None,dividends=False)
d2[0], d2[1], d2[2], ticker, g)
- if cachename is None:
+ # Cache the finance data if cachename is supplied, or there is a writable
+ # cache directory.
+ if cachename is None and cachedir is not None:
cachename = os.path.join(cachedir, md5(url).hexdigest())
- if os.path.exists(cachename):
- fh = open(cachename)
- verbose.report('Using cachefile %s for %s'%(cachename, ticker))
+ if cachename is not None:
+ if os.path.exists(cachename):
+ fh = open(cachename)
+ verbose.report('Using cachefile %s for %s'%(cachename, ticker))
+ else:
+ mkdirs(os.path.abspath(os.path.dirname(cachename)))
+ with contextlib.closing(urlopen(url)) as urlfh:
+ with open(cachename, 'wb') as fh:
+ fh.write(urlfh.read())
+ verbose.report('Saved %s data to cache file %s'%(ticker, cachename))
+ fh = open(cachename, 'r')
+
+ return fh
else:
- mkdirs(os.path.abspath(os.path.dirname(cachename)))
- with contextlib.closing(urlopen(url)) as urlfh:
- with open(cachename, 'wb') as fh:
- fh.write(urlfh.read())
- verbose.report('Saved %s data to cache file %s'%(ticker, cachename))
- fh = open(cachename, 'r')
-
- return fh
+ return urlopen(url)
def quotes_historical_yahoo(ticker, date1, date2, asobject=False,
View
34 lib/matplotlib/font_manager.py
@@ -1311,28 +1311,38 @@ def findfont(prop, fontext='ttf'):
return result
else:
- if sys.version_info[0] >= 3:
- _fmcache = os.path.join(get_configdir(), 'fontList.py3k.cache')
+ configdir = get_configdir()
+ if configdir is not None:
+ if sys.version_info[0] >= 3:
+ _fmcache = os.path.join(configdir, 'fontList.py3k.cache')
+ else:
+ _fmcache = os.path.join(configdir, 'fontList.cache')
else:
- _fmcache = os.path.join(get_configdir(), 'fontList.cache')
+ # Should only happen in a restricted environment (such as Google App
+ # Engine). Deal with this gracefully by not caching fonts.
+ _fmcache = None
fontManager = None
def _rebuild():
global fontManager
fontManager = FontManager()
- pickle_dump(fontManager, _fmcache)
+ if _fmcache:
+ pickle_dump(fontManager, _fmcache)
verbose.report("generated new fontManager")
- try:
- fontManager = pickle_load(_fmcache)
- if (not hasattr(fontManager, '_version') or
- fontManager._version != FontManager.__version__):
+ if _fmcache:
+ try:
+ fontManager = pickle_load(_fmcache)
+ if (not hasattr(fontManager, '_version') or
+ fontManager._version != FontManager.__version__):
+ _rebuild()
+ else:
+ fontManager.default_size = None
+ verbose.report("Using fontManager instance from %s" % _fmcache)
+ except:
_rebuild()
- else:
- fontManager.default_size = None
- verbose.report("Using fontManager instance from %s" % _fmcache)
- except:
+ else:
_rebuild()
def findfont(prop, **kw):
View
5 lib/matplotlib/testing/compare.py
@@ -99,7 +99,10 @@ def compare_float( expected, actual, relTol = None, absTol = None ):
# parameters old and new to a list that can be passed to Popen to
# convert files with that extension to png format.
def get_cache_dir():
- cache_dir = os.path.join(_get_configdir(), 'test_cache')
+ configdir = _get_configdir()
+ if configdir is None:
+ raise RuntimeError('Could not find a suitable configuration directory')
+ cache_dir = os.path.join(configdir, 'test_cache')
if not os.path.exists(cache_dir):
try:
os.makedirs(cache_dir)
View
36 lib/matplotlib/texmanager.py
@@ -41,6 +41,7 @@
import os
import shutil
import sys
+import warnings
from hashlib import md5
@@ -94,16 +95,30 @@ class TexManager:
oldcache = os.path.join(oldpath, '.tex.cache')
configdir = mpl.get_configdir()
- texcache = os.path.join(configdir, 'tex.cache')
+ if configdir is not None:
+ texcache = os.path.join(configdir, 'tex.cache')
+ else:
+ # Should only happen in a restricted environment (such as Google App
+ # Engine). Deal with this gracefully by not creating a cache directory.
+ texcache = None
if os.path.exists(oldcache):
- # FIXME raise proper warning
- print("""\
-WARNING: found a TeX cache dir in the deprecated location "%s".
- Moving it to the new default location "%s".""" % (oldcache, texcache),
- file=sys.stderr)
- shutil.move(oldcache, texcache)
- mkdirs(texcache)
+ if texcache is not None:
+ try:
+ shutil.move(oldcache, texcache)
+ except IOError as e:
+ warnings.warn('File could not be renamed: %s' % e)
+ else:
+ warnings.warn("""\
+Found a TeX cache dir in the deprecated location "%s".
+ Moving it to the new default location "%s".""" % (oldcache, texcache))
+ else:
+ warnings.warn("""\
+Could not rename old TeX cache dir "%s": a suitable configuration
+ directory could not be found.""" % oldcache)
+
+ if texcache is not None:
+ mkdirs(texcache)
_dvipng_hack_alpha = None
#_dvipng_hack_alpha = dvipng_hack_alpha()
@@ -145,6 +160,11 @@ class TexManager:
def __init__(self):
+ if self.texcache is None:
+ raise RuntimeError(
+ ('Cannot create TexManager, as there is no cache directory '
+ 'available'))
+
mkdirs(self.texcache)
ff = rcParams['font.family'].lower()
if ff in self.font_families:
Something went wrong with that request. Please try again.