OSError: Too many open files #396

Closed
sander76 opened this Issue Dec 2, 2016 · 9 comments

Comments

Projects
None yet
2 participants
@sander76

sander76 commented Dec 2, 2016

I am creating pdf's from a (static) site which is built every night.
But after some time weasyprints ends with the above OSError.
I have been trying to simplify the issue and have come up with the following python script:
(Running Python 3.5.2, Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-51-generic x86_64))

from weasyprint import HTML
import logging

logging.basicConfig(level=logging.DEBUG)

URL = "http://tools.hde.nl/menc/site/guides/Pliss%C3%A9%20%26%20Duette%C2%AE%20Bottom-Up%20programming/"
OUT = "/home/admin-s/test.pdf"

for lp in range(0, 300):
    try:
        HTML(URL).write_pdf(OUT)
    except OSError as e:
        logging.exception("**************** ERROR AT ATTEMPT: {} *******************".format(lp))
        break

In my case the error occurs after approx 60 loops (varies per situation).
Stacktrace:

ERROR:root:**************** ERROR AT ATTEMPT: 63 *******************
Traceback (most recent call last):
  File "tester.py", line 11, in <module>
  File "/home/admin-s/envs/weasy/lib/python3.5/site-packages/weasyprint/__init__.py", line 178, in write_pdf
  File "/home/admin-s/envs/weasy/lib/python3.5/site-packages/weasyprint/__init__.py", line 143, in render
  File "/home/admin-s/envs/weasy/lib/python3.5/site-packages/weasyprint/document.py", line 325, in _render
  File "/home/admin-s/envs/weasy/lib/python3.5/site-packages/weasyprint/css/__init__.py", line 712, in get_all_computed_styles
  File "/home/admin-s/envs/weasy/lib/python3.5/site-packages/weasyprint/css/__init__.py", line 162, in find_stylesheets
  File "/home/admin-s/envs/weasy/lib/python3.5/site-packages/weasyprint/__init__.py", line 261, in __init__
  File "/home/admin-s/envs/weasy/lib/python3.5/site-packages/weasyprint/css/__init__.py", line 692, in preprocess_stylesheet
  File "/home/admin-s/envs/weasy/lib/python3.5/site-packages/weasyprint/fonts.py", line 229, in add_font_face
OSError: [Errno 24] Too many open files: '/tmp/tmp1a03t715'

@liZe liZe added the bug label Dec 2, 2016

@sander76

This comment has been minimized.

Show comment
Hide comment
@sander76

sander76 Dec 4, 2016

Doing a little digging around I found out the following:

ulimit -n shows the amount of allowed open files. It was set at 1024 which was pretty low compared to a docker instance I tried (where the issue was not occurring).

I edited /etc/security/limits.conf by adding

*               soft    nofile          65535
*               hard    nofile          65535

Now ulimit -n shows the number above and the issue like described in the first post is gone.
Don't know though whether this is a workaround for an actual weasyprint issue. But for now it works for me !

sander76 commented Dec 4, 2016

Doing a little digging around I found out the following:

ulimit -n shows the amount of allowed open files. It was set at 1024 which was pretty low compared to a docker instance I tried (where the issue was not occurring).

I edited /etc/security/limits.conf by adding

*               soft    nofile          65535
*               hard    nofile          65535

Now ulimit -n shows the number above and the issue like described in the first post is gone.
Don't know though whether this is a workaround for an actual weasyprint issue. But for now it works for me !

@liZe

This comment has been minimized.

Show comment
Hide comment
@liZe

liZe Dec 6, 2016

Member

There's a real problem about this, temporary files used for fonts management are deleted but still open by the python process. That's why the 5th item in #376 is needed for example.

Member

liZe commented Dec 6, 2016

There's a real problem about this, temporary files used for fonts management are deleted but still open by the python process. That's why the 5th item in #376 is needed for example.

@sander76

This comment has been minimized.

Show comment
Hide comment
@sander76

sander76 Dec 7, 2016

@liZe Alright. But the 5th item is windows related if I understand correctly. My issue is all linux...

sander76 commented Dec 7, 2016

@liZe Alright. But the 5th item is windows related if I understand correctly. My issue is all linux...

@liZe

This comment has been minimized.

Show comment
Hide comment
@liZe

liZe Dec 7, 2016

Member

Yes, the file is open on both Windows and Linux, but Windows doesn't accept deleting open files when Linux does. The file is marked as open+deleted on Linux, when it should be closed and deleted afterwards.

Member

liZe commented Dec 7, 2016

Yes, the file is open on both Windows and Linux, but Windows doesn't accept deleting open files when Linux does. The file is marked as open+deleted on Linux, when it should be closed and deleted afterwards.

@sander76

This comment has been minimized.

Show comment
Hide comment
@sander76

sander76 Dec 7, 2016

sander76 commented Dec 7, 2016

@liZe

This comment has been minimized.

Show comment
Hide comment
@liZe

liZe Dec 7, 2016

Member

Obvious fact: this bug is not related at all to the Windows support 😄.

I really wonder if
anyone else has succeeded...

At least I did, and it works well.

There are lots of bugs related to the Windows support in the bug tracker, so I assume that it's working for some people too. There's almost no Windows-specific code in WeasyPrint, the only hard part is the installation. The documentation about that has been improved during the last months, you should give another try!

Member

liZe commented Dec 7, 2016

Obvious fact: this bug is not related at all to the Windows support 😄.

I really wonder if
anyone else has succeeded...

At least I did, and it works well.

There are lots of bugs related to the Windows support in the bug tracker, so I assume that it's working for some people too. There's almost no Windows-specific code in WeasyPrint, the only hard part is the installation. The documentation about that has been improved during the last months, you should give another try!

@liZe liZe self-assigned this Dec 7, 2016

@liZe liZe added this to the 0.34 milestone Dec 7, 2016

@liZe liZe closed this in bf6911d Dec 7, 2016

@liZe

This comment has been minimized.

Show comment
Hide comment
@liZe

liZe Dec 7, 2016

Member

It's now fixed, thanks!

Member

liZe commented Dec 7, 2016

It's now fixed, thanks!

@sander76

This comment has been minimized.

Show comment
Hide comment

sander76 commented Dec 7, 2016

Greatn

@sander76

This comment has been minimized.

Show comment
Hide comment

sander76 commented Dec 7, 2016

Great!

jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Jan 15, 2017

kleink
Update py-weasyprint to 0.34.
Version 0.34
------------

Released on 2016-12-21.

Bug fixes:

* `#398 <Kozea/WeasyPrint#398>`_:
  Honor the presentational_hints option for PDFs.
* `#399 <Kozea/WeasyPrint#399>`_:
  Avoid CairoSVG-2.0.0rc* on Python 2.
* `#396 <Kozea/WeasyPrint#396>`_:
  Correctly close files open by mkstemp.
* `#403 <Kozea/WeasyPrint#403>`_:
  Cast the number of columns into int.
* Fix multi-page multi-columns and add related tests.


Version 0.33
------------

Released on 2016-11-28.

New features:

* `#393 <Kozea/WeasyPrint#393:
  Add tests on MacOS.
* `#370 <Kozea/WeasyPrint#370>`_:
  Enable @font-face on MacOS.

Bug fixes:

* `#389 <Kozea/WeasyPrint#389>`_:
  Always update resume_at when splitting lines.
* `#394 <Kozea/WeasyPrint#394>`_:
  Don't build universal wheels.
* `#388 <Kozea/WeasyPrint#388>`_:
  Fix logic when finishing block formatting context.


Version 0.32
------------

Released on 2016-11-17.

New features:

* `#28 <Kozea/WeasyPrint#28>`_:
  Support @font-face on Linux.
* Support CSS fonts level 3 almost entirely, including OpenType features.
* `#253 <Kozea/WeasyPrint#253>`_:
  Support presentational hints (optional).
* Support break-after, break-before and break-inside for pages and columns.
* `#384 <Kozea/WeasyPrint#384:
  Major performance boost.

Bux fixes:

* `#368 <Kozea/WeasyPrint#368>`_:
  Respect white-space for shrink-to-fit.
* `#382 <Kozea/WeasyPrint#382>`_:
  Fix the preferred width for column groups.
* Handle relative boxes in column-layout boxes.

Documentation:

* Add more and more documentation about Windows installation.
* `#355 <Kozea/WeasyPrint#355:
  Add fonts requirements for tests.


Version 0.31
------------

Released on 2016-08-28.

New features:

* `#124 <Kozea/WeasyPrint#124>`_:
  Add MIME sniffing for images.
* `#60 <Kozea/WeasyPrint#60>`_:
  CSS Multi-column Layout.
* `#197 <Kozea/WeasyPrint#197>`_:
  Add hyphens at line breaks activated by a soft hyphen.

Bux fixes:

* `#132 <Kozea/WeasyPrint#132>`_:
  Fix Python 3 compatibility on Windows.

Documentation:

* `#329 <Kozea/WeasyPrint#329>`_:
  Add documentation about installation on Windows.


Version 0.30
------------

Released on 2016-07-18.

WeasyPrint now depends on html5lib-0.999999999.

Bux fixes:

* Fix Acid2
* `#325 <Kozea/WeasyPrint#325>`_:
  Cutting lines is broken in page margin boxes.
* `#334 <Kozea/WeasyPrint#334>`_:
  Newest html5lib 0.999999999 breaks rendering.


Version 0.29
------------

Released on 2016-06-17.

Bug fixes:

* `#263 <Kozea/WeasyPrint#263:
  Don't crash with floats with percents in positions.
* `#323 <Kozea/WeasyPrint#323>`_:
  Fix CairoSVG 2.0 pre-release dependency in Python 2.x.


Version 0.28
------------

Released on 2016-05-16.

Bug fixes:

* `#189 <Kozea/WeasyPrint#189>`_:
  ``white-space: nowrap`` still wraps on hyphens
* `#305 <Kozea/WeasyPrint#305>`_:
  Fix crashes on some tables
* Don't crash when transform matrix isn't invertible
* Don't crash when rendering ratio-only SVG images
* Fix margins and borders on some tables


Version 0.27
------------

Released on 2016-04-08.

New features:

* `#295 <Kozea/WeasyPrint#295>`_:
  Support the 'rem' unit.
* `#299 <Kozea/WeasyPrint#299>`_:
  Enhance the support of SVG images.

Bug fixes:

* `#307 <Kozea/WeasyPrint#307>`_:
  Fix the layout of cells larger than their tables.

Documentation:

* The website is now on GitHub Pages, the documentation is on Read the Docs.
* `#297 <Kozea/WeasyPrint#297>`_:
  Rewrite the CSS chapter of the documentation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment