Infinite loop and memory leak with NBSP after a long word

[greycat70]

The following example causes weasyprint 0.42 (on Debian 9 amd64) to go into an infinite (or practically infinite) loop and consume all possible memory. I've worked around it in the application that generates the HTML, partially, but this is really quite nasty and my band-aid won't cover all possible user inputs. I've also had to add resource limits on the script that runs weasyprint.

This is a line.
This is another line.
----------------------------------------------------------------------------------------------------------------------------------------  
No more lines.

Ugh, the bug tracking system is rendering the HTML. I don't want it to render the HTML. Here, here's the HTML in a text file that you can download: http://wooledge.org/~greg/weasybug614

[liZe]

Thank you for this bug report.

I've tried to reproduce the bug, but everything works fine for me, even with older versions of WeasyPrint. Could you please:

• tell me what is your default font (you can find it by rendering another document with no CSS and check what font is included in your PDF file, or just attach it here if you don't know how to do this),
• try if possible the current master version of WeasyPrint (warning: it's Python 3 only) and tell me if the bug is gone.

[greycat70]

Here's the PDF result from http://wooledge.org/~greg/weasy614.html

weasy614.pdf

[greycat70]

Following the "git clone" instructions with a python3 virtualenv, the problem still occurs. I get this warning: "UserWarning: There are known rendering problems with cairo < 1.15.4. WeasyPrint may work with older versions, but please read the note about the needed cairo version on the "Install" page of the documentation before reporting bugs."

Debian 9 has libcairo 1.14.8

[liZe]

Here's the PDF result from http://wooledge.org/~greg/weasy614.html

I have the same default font as you (DejaVu Serif) but I can't reproduce this bug, and in this situation it's hard to help you…

Debian 9 has libcairo 1.14.8

Your issue is probably not related to Cairo.

Do you have the problem when launching weasyprint --verbose http://wooledge.org/~greg/weasybug614 test.pdf?

[greycat70]

(env) wooledg:~$ weasyprint --verbose http://wooledge.org/~greg/weasybug614 test.pdf
/home/wooledg/weasy/WeasyPrint/weasyprint/document.py:32: UserWarning: There are known rendering problems with cairo < 1.15.4. WeasyPrint may work with older versions, but please read the note about the needed cairo version on the "Install" page of the documentation before reporting bugs.
  'There are known rendering problems with cairo < 1.15.4. WeasyPrint '
INFO: Step 1 - Fetching and parsing HTML - http://wooledge.org/~greg/weasybug614
INFO: Step 3 - Applying CSS
INFO: Step 4 - Creating formatting structure
INFO: Step 5 - Creating layout - Page 1
[…]
INFO: Step 5 - Creating layout - Page 111
^CTraceback (most recent call last):
  File "/home/wooledg/weasy/WeasyPrint/env/bin/weasyprint", line 11, in <module>
    load_entry_point('WeasyPrint', 'console_scripts', 'weasyprint')()
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/__main__.py", line 173, in main
    getattr(html, 'write_' + format_)(output, **kwargs)
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/__init__.py", line 185, in write_pdf
    font_config=font_config).write_pdf(
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/__init__.py", line 146, in render
    font_config)
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/document.py", line 329, in _render
    [Page(p, enable_hinting) for p in page_boxes],
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/document.py", line 329, in <listcomp>
    [Page(p, enable_hinting) for p in page_boxes],
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/__init__.py", line 53, in layout_document
    context, root_box, html, cascaded_styles, computed_styles))
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/pages.py", line 600, in make_all_pages
    context, root_box, page_type, resume_at, page_number)
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/pages.py", line 519, in make_page
    positioned_boxes, positioned_boxes, adjoining_margins)
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/blocks.py", line 60, in block_level_layout
    adjoining_margins)
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/blocks.py", line 111, in block_box_layout
    page_is_empty, absolute_boxes, fixed_boxes, adjoining_margins)
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/blocks.py", line 646, in block_container_layout
    adjoining_margins)
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/blocks.py", line 60, in block_level_layout
    adjoining_margins)
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/blocks.py", line 111, in block_box_layout
    page_is_empty, absolute_boxes, fixed_boxes, adjoining_margins)
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/blocks.py", line 518, in block_container_layout
    for line, resume_at in lines_iterator:
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/inlines.py", line 50, in iter_line_boxes
    device_size, absolute_boxes, fixed_boxes, first_letter_style)
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/inlines.py", line 107, in get_next_linebox
    waiting_floats, line_children=[])
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/layout/inlines.py", line 771, in split_inline_box
    last_letter + first, child.style['lang'])
  File "/home/wooledg/weasy/WeasyPrint/weasyprint/text.py", line 1188, in can_break_text
    return any(attr.is_line_break for attr in log_attrs[1:length - 1])
KeyboardInterrupt

[liZe]

I've tried WeasyPrint versions between 0.42 and current master branch, I've tried with Cairo 1.14.x (just in case), I've tried with the same font as you, I've tried to change the number of hyphens, I didn't reproduce your bug.

The only solution I can think of now is to find an old version of WeasyPrint that works for you and to use git bisect to find which commit caused this problem.

[liZe]

May be related to #585.

[greycat70]

WeasyPrint version 0.40 ("git checkout v0.40") works.
WeasyPrint version 0.42 fails.
WeasyPrint version 0.41 works.

Starting from there, git bisect leads me to:
883cbac is the first bad commit
commit 883cbac
Author: Guillaume Ayoub
Date: Mon Dec 11 17:22:54 2017 +0100

Fix more line-breaking use cases

[Tontyna]

Can reproduce the infinite loop. On Windows 7 (64bit), having Pango 1.40.11. With this reduced html:

<html><body>
<br>--------------------------------------------------------------------------------------------------------------------&nbsp;
<br>
</body></html>

The   and the two <br> are required to force the bug. The long line must be wider than the page margins and it must follow the first <br> immediately. If it starts on a new line #585 is generated.

When I force the otherwise never-ending make_all_pages to return (e.g. at page_number==13) a PDF filled with pages like that is produced:

Tried to debug but must confess: The way WeasyPrint layouts the pages is a mystery to me. Anyway: It looks like Pango in can_break_text reports that the long line can be broken, but the PangoLayout, used in split_first_line fails to split the line and WeasyPrint gets stuck forever which this non-breakable breakable line.

Definitely resume_at in make_page never reaches None, it stays forever at [0, [0, [1, None]]], whatever that means.

[liZe]

Can reproduce the infinite loop. On Windows 7 (64bit), having Pango 1.40.11

The bug is in Pango and have been fixed in 1.40.13 (see bug 788115 and bug 785978).

It's becoming harder and harder to have recent versions of WeasyPrint working with older versions of Cairo, Pango, GhostScript, etc. Changes introduced in 0.41 and 0.42 rely on a lot of PDF- and Unicode-related features, and having a bleeding-edge system doesn't help me to find corner cases like this one.

I'm seriously thinking about upgrading the needed library versions on the install page (I already did with Cairo), and adding warnings at runtime and in the documentation if the needed library versions are not installed. WeasyPrint 0.40 seems to be a good candidate for users with older versions. If I had time, I would even backport some fixes of the 0.42.x branch on 0.40 and make the next 0.x versions from this hybrid version (basically with Python 2 support, no pdfrw, no #528).

[Tontyna]

Not to mention how hard it is for Windows users like me to keep their GTK up-to-date.

Looking at the Pango bugs and the Pango source, I'm not shure whether even with an up-to-date Pango there won't be similiar issues -- in this case: discrepancies between pango.pango_get_log_attrs(), reporting that a string can be wrapped and PangoLayout not wrapping it at all.

More debugging and experimenting revealed the problem and an attempt to solve it.

Problem

After asking Pango whether a text is breakable -- weasyprint.text.can_break_text() -- WeasyPrint relies on that information and assumes that the text actually will be wrapped when layouted in weasyprint.text.split_first_line().
But that's not always the case. E.g. in Pango 1.40.11 not for

• xxxöxxx
• xxx xxx
• --- ---

The last letter in those strings is attributed as is_line_break by pango.pango_get_log_attrs(). Dont know why the ö or the   (and probably a lot more 'special chars') have that effect (in Pango < 1.40.13), but they have and as one can see in the Pango source code it obviously tries to fulfill an internationalization-Unicode-whatnot-standard, which looks rather complicated, so bugs are to be expected.
When those strings are layouted, PangoLayout does not wrap the last letter onto a new line -- which seems ok to me, but gives the impression that PangoLayout doesnt coordinate its job with PangoLogAttr 😁

This inconsistency is the cause of #585 and #614.

Idea to work around the Pango bug

If split_inline_level() returns a child_resume_at of None we know that PangoLayout decided not to wrap the text.
Not shure how to handle that case properly (as I said: I don't really understand WeasyPrint*s layout process), but the following patch, prevents crash and infinite loop.

diff --git a/weasyprint/layout/inlines.py b/weasyprint/layout/inlines.py
index 0270de50..57d0d126 100644
--- a/weasyprint/layout/inlines.py
+++ b/weasyprint/layout/inlines.py
@@ -809,8 +809,12 @@ def split_inline_box(context, box, position_x, max_x, skip_stack,
                         if (child.is_in_normal_flow() and
                                 can_break_inside(child)):
                             # This waiting child is in flow and can be broken,
-                            # let's break it!
-                            break_found = True
+                            # let's TRY TO break it!
+
+                            # cant be shure about that!
+                            # Thats what PangoLogAttr told us, have to wait for
+                            # PangoLayout
+                            # break_found = True

                             # We break the waiting child at its last possible
                             # breaking point.
@@ -826,6 +830,13 @@ def split_inline_box(context, box, position_x, max_x, skip_stack,
                                     absolute_boxes, fixed_boxes,
                                     line_placeholders, waiting_floats,
                                     line_children))
+                            # prevent #585 and #614
+                            # TODO: Expert's review required!
+                            break_found = not child_resume_at is None
+                            if child_resume_at is None:
+                                # PangoLayout decided NOT to break the child
+                                child_resume_at = (0, None)
+
                             children = children + waiting_children_copy
                             if child_new_child is None:
                                 # May be None where we have an empty TextBox.

As you can see, there are definitely issues with the <br> boxes -- expert required!

[liZe]

Idea to work around the Pango bug

According to what's in the Pango bugs, your workaround seems to be a perfect solution!

As you can see, there are definitely issues with the
boxes -- expert required!

I think you hit the only xfail test of WeasyPrint 😉. git blame following renamed files tells us that it was already known in 2012 (search xfail in d6b78b3).