tex cache lockfile retries should be configurable #7776

Open
gerritholl opened this Issue Jan 9, 2017 · 2 comments

Projects

None yet

3 participants

@gerritholl

I am doing data processing and generating many plots with many scripts running in parallel. The scripts use texmanager, and many fail when drawing to png due to below. My plots are large, with many subplots, and the filesystem I'm writing to is slow. Therefore, the hardcoded limit of 50 tries with 0.1 second waits is not appropriate.

Those hardcoded values should be configurable so that the user can change them if needed.

Traceback (most recent call last):
  File "/home/users/gholl/venv/stable-3.5/bin/plot_hirs_field_timeseries", line 11, in <module>
    load_entry_point('FCDR-HIRS==0.0.4', 'console_scripts', 'plot_hirs_field_timeseries')()
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/FCDR_HIRS/analysis/timeseries.py", line 1225, in main
    "calibpos": p.corr_calibpos})
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/FCDR_HIRS/analysis/timeseries.py", line 592, in plot_noise_with_other
    alltemp='_'.join(temperatures), tb=t[0], te=t[-1]))
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/pyatmlab/graphics.py", line 208, in print_or_show
    fig.canvas.print_figure(str(outf))
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/backend_bases.py", line 2192, in print_figure
    **kwargs)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 545, in print_png
    FigureCanvasAgg.draw(self)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 464, in draw
    self.figure.draw(self.renderer)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/artist.py", line 63, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/figure.py", line 1142, in draw
    renderer, self, dsu, self.suppressComposite)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/image.py", line 139, in _draw_list_compositing_images
    a.draw(renderer)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/artist.py", line 63, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 2405, in draw
    mimage._draw_list_compositing_images(renderer, self, dsu)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/image.py", line 139, in _draw_list_compositing_images
    a.draw(renderer)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/artist.py", line 63, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/axis.py", line 1138, in draw
    renderer)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/axis.py", line 1078, in _get_tick_bboxes
    extent = tick.label1.get_window_extent(renderer)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/text.py", line 967, in get_window_extent
    bbox, info, descent = self._get_layout(self._renderer)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/text.py", line 362, in _get_layout
    ismath=ismath)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 230, in get_text_width_height_descent
    renderer=self)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/texmanager.py", line 676, in get_text_width_height_descent
    dvifile = self.make_dvi(tex, fontsize)
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/texmanager.py", line 406, in make_dvi
    with Locked(self.texcache):
  File "/home/users/gholl/venv/stable-3.5/lib/python3.5/site-packages/matplotlib/cbook.py", line 2738, in __enter__
    raise self.TimeoutError(err_str)
matplotlib.cbook.TimeoutError: LOCKERROR: matplotlib is trying to acquire the lock
    '/home/users/gholl/.cache/matplotlib/tex.cache/.matplotlib_lock-*'
and has failed.  This maybe due to any other process holding this
lock.  If you are sure no other matplotlib process is running try
removing these folders and trying again.

Unfortunately, the nature of the problem makes it hard to write a minimum code snippet that reproduces it, because it only occurs with big plots when running many processes in parallel.

I'm using Matplotlib 2.0.0rc2 (installed through pip) on Python 3.5.

@anntzer
Contributor
anntzer commented Jan 10, 2017

Rather, the locking should be done per-entry, rather than globally for the whole folder, as I don't think writing to separate cache entries creates any problems?

The current lock implementation can only lock folders though (because it writes to a file within that folder). We can either move each cache entry into its own subfolder (not too hard?), or change the lock implementation to be able to lock single files (off the top of my head, one should be able to do so by creating lockfiles named $MPLCACHEDIR/locks/$(resolved-path-to-file-or-dir-to-be-locked-encoding-slashes-appropriately)).

@tacaswell tacaswell added this to the 2.1 (next point release) milestone Jan 10, 2017
@tacaswell
Member

Another option would be to isolate your process's tex cache from each other by launching your python processes as

MPLCONFIGDIR=/tmp/some_pathN python foo.py

which if your /tmp is a ramdisk might get you a reasonable speed up to boot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment