Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Too many open files" in test runs on Python 3.3 #3315

Closed
matthew-brett opened this issue Jul 27, 2014 · 2 comments · Fixed by #5295
Closed

"Too many open files" in test runs on Python 3.3 #3315

matthew-brett opened this issue Jul 27, 2014 · 2 comments · Fixed by #5295
Assignees
@mdboom
Labels
OS: Apple

Comments

@matthew-brett
Copy link
Contributor

The OSX testing juggernaut rolls on. For tests on Python 3.3.5, I am getting multiple (> 400) test errors related to "Too many open files":

======================================================================
ERROR: matplotlib.tests.test_mathtext.test_mathtext_cm_37.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/matplotlib/testing/decorators.py", line 51, in failer
    result = f(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/matplotlib/testing/decorators.py", line 183, in do_test
    figure.savefig(actual_fname, **self._savefig_kwarg)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/matplotlib/figure.py", line 1470, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/matplotlib/backend_bases.py", line 2192, in print_figure
    **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/matplotlib/backends/backend_pdf.py", line 2475, in print_pdf
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/matplotlib/backends/backend_pdf.py", line 525, in close
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/matplotlib/backends/backend_pdf.py", line 628, in writeFonts
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/matplotlib/backends/backend_pdf.py", line 798, in embedTTF
OSError: [Errno 24] Too many open files: '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/matplotlib/mpl-data/fonts/ttf/cmr10.ttf'

See:

I wasn't getting this failure for the multiprocess tests on previous Python 3.3 runs (which will have been testing a different commit on the 1.4.x branch):

https://s3.amazonaws.com/archive.travis-ci.org/jobs/30794971/log.txt

I assume this is because the multiprocess tests were each only using a smaller number of open files per process.

Any ideas why this should be happening for Python 3.3 specifically, not 2.7 or 3.4?
@matthew-brett matthew-brett mentioned this issue Jul 27, 2014
Closed
@tacaswell tacaswell added this to the unassigned milestone Aug 17, 2014
@tacaswell tacaswell added the OSX label Aug 17, 2014
@tacaswell
Copy link
Member

@matthew-brett Is there anything we need to do mpl side about this?

@matthew-brett
Copy link
Contributor Author

Sorry - I am not sure of the cause of the problem, or why it is fixed
in Python 3.4 - so I don't know whether there is a real mpl problem or
not :). Is there anything simple I can do to track down what is
happening?

mdboom added a commit to mdboom/matplotlib that referenced this issue Oct 21, 2015
@mdboom
Reduce number of font files opened
2b98a5f 
This should hopefully address the long-reported "Too many open files"
error message (Fix matplotlib#3315).

To reproduce: On a Mac or Windows box with starvation for file
handles (Linux has a much higher file handle limit by default), build
the docs, then immediately build again.  This will trigger the caching
bug.

The font cache in the mathtext renderer was broken.  It was caching a
font file once for every *combination* of font properties, including
things like size.  Therefore, in a complex math expression containing
many different sizes of the same font, the font file was opened once for
each of those sizes.

Font files are opened and kept open (rather than opened, read,
and closed) so that FreeType only needs to load the actual glyphs that
are used, rather than the entire font.  In an era of cheap memory and
fast disk, it probably doesn't matter for our current fonts, but once
 matplotlib#5214 is merged, we will have larger font files with many more glyphs
and this loading time will matter more.

The solution here is to do all font file loading in one place and to use
`lru_cache` (available since Python 3.2) to do the caching, and to use
only the file name and hinting parameters as a cache key.  For earlier
versions of Python, the functools32 backport package is required.  (Or
we can discuss whether we want to vendor it).
@mdboom mdboom mentioned this issue Oct 21, 2015
Merged
@mdboom mdboom self-assigned this Oct 21, 2015
mdboom added a commit to mdboom/matplotlib that referenced this issue Oct 26, 2015
@mdboom
Reduce number of font files opened
facbee5 
This should hopefully address the long-reported "Too many open files"
error message (Fix matplotlib#3315).

To reproduce: On a Mac or Windows box with starvation for file
handles (Linux has a much higher file handle limit by default), build
the docs, then immediately build again.  This will trigger the caching
bug.

The font cache in the mathtext renderer was broken.  It was caching a
font file once for every *combination* of font properties, including
things like size.  Therefore, in a complex math expression containing
many different sizes of the same font, the font file was opened once for
each of those sizes.

Font files are opened and kept open (rather than opened, read,
and closed) so that FreeType only needs to load the actual glyphs that
are used, rather than the entire font.  In an era of cheap memory and
fast disk, it probably doesn't matter for our current fonts, but once
 matplotlib#5214 is merged, we will have larger font files with many more glyphs
and this loading time will matter more.

The solution here is to do all font file loading in one place and to use
`lru_cache` (available since Python 3.2) to do the caching, and to use
only the file name and hinting parameters as a cache key.  For earlier
versions of Python, the functools32 backport package is required.  (Or
we can discuss whether we want to vendor it).
mdboom added a commit to mdboom/matplotlib that referenced this issue Oct 27, 2015
@mdboom
Reduce number of font files opened
bd5812b 
This should hopefully address the long-reported "Too many open files"
error message (Fix matplotlib#3315).

To reproduce: On a Mac or Windows box with starvation for file
handles (Linux has a much higher file handle limit by default), build
the docs, then immediately build again.  This will trigger the caching
bug.

The font cache in the mathtext renderer was broken.  It was caching a
font file once for every *combination* of font properties, including
things like size.  Therefore, in a complex math expression containing
many different sizes of the same font, the font file was opened once for
each of those sizes.

Font files are opened and kept open (rather than opened, read,
and closed) so that FreeType only needs to load the actual glyphs that
are used, rather than the entire font.  In an era of cheap memory and
fast disk, it probably doesn't matter for our current fonts, but once
 matplotlib#5214 is merged, we will have larger font files with many more glyphs
and this loading time will matter more.

The solution here is to do all font file loading in one place and to use
`lru_cache` (available since Python 3.2) to do the caching, and to use
only the file name and hinting parameters as a cache key.  For earlier
versions of Python, the functools32 backport package is required.  (Or
we can discuss whether we want to vendor it).
mdboom added a commit to mdboom/matplotlib that referenced this issue Oct 28, 2015
@mdboom
Reduce number of font files opened
349762b 
This should hopefully address the long-reported "Too many open files"
error message (Fix matplotlib#3315).

To reproduce: On a Mac or Windows box with starvation for file
handles (Linux has a much higher file handle limit by default), build
the docs, then immediately build again.  This will trigger the caching
bug.

The font cache in the mathtext renderer was broken.  It was caching a
font file once for every *combination* of font properties, including
things like size.  Therefore, in a complex math expression containing
many different sizes of the same font, the font file was opened once for
each of those sizes.

Font files are opened and kept open (rather than opened, read,
and closed) so that FreeType only needs to load the actual glyphs that
are used, rather than the entire font.  In an era of cheap memory and
fast disk, it probably doesn't matter for our current fonts, but once
 matplotlib#5214 is merged, we will have larger font files with many more glyphs
and this loading time will matter more.

The solution here is to do all font file loading in one place and to use
`lru_cache` (available since Python 3.2) to do the caching, and to use
only the file name and hinting parameters as a cache key.  For earlier
versions of Python, the functools32 backport package is required.  (Or
we can discuss whether we want to vendor it).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mdboom mdboom

Labels
OS: Apple
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Reduce number of font file handles opened mdboom/matplotlib
3 participants
@mdboom @matthew-brett @tacaswell