[qt] generated PDF does not pass the PDF validations #1573

lindi2 · 2014-03-05T08:54:35Z

Steps to reproduce:

Ensure that you are using a Debian 7 amd64 system
Download and unpack http://downloads.sourceforge.net/project/wkhtmltopdf/0.12.0/wkhtmltox-linux-amd64_0.12.0-03c001d.tar.xz
Download PDF validator from the Apache project: http://www.apache.org/dyn/closer.cgi/pdfbox/1.8.4/preflight-app-1.8.4.jar

cat > hello.html <<EOF
<html>
<head>
<title>title</title>
</head>
<body>
<h1>h1</h1>
<p>p</p>
</body>
</html>
EOF

wkhtmltox/bin/wkhtmltopdf hello.html hello.pdf
java -jar preflight-app-1.8.4.jar hello.pdf

Expected results:
6) output of wkhtmltopdf passes the PDF validator

Actual results:
6) the PDF validator complains:

The filehello.pdf is not valid, error(s) :
1.1 : Body Syntax error, Second line must contains at least 4 bytes greater than 127
1.0 : Syntax error, Object (9:0) at offset 326 does not end with 'endobj'.

More info:

The file hello.pdf can be downloaded from http://lindi.iki.fi/lindi/wkhtmltopdf/hello.03c001de254b857f08eba80b62d4b6490ffed41d.pdf -- its md5sum is efe87c6b91ad3b8c93071d8689fe144e.
If I look at the PDF internals I can see that object 9:0 indeed does not end with endobj (comments with "#" added by me):

...
9 0 obj      # obj 9:0 starts here
<<
/__WKANCHOR_2 8 0 R
>>
11 0 obj      # obj 11:0 starts here
<</Title (\376\377^@h^@1)
  /Parent 10 0 R
  /Dest /__WKANCHOR_2
  /Count 0
>>
endobj       # obj 11:0 ends here
10 0 obj     # obj 10:0 starts here
...

This seems to be the only object that does not end with endobj since there are 26 objs and 25 endobjs:

$ grep -ca " obj$" hello.pdf
26
$ grep -ca "endobj" hello.pdf
25

The text was updated successfully, but these errors were encountered:

ashkulz · 2014-03-06T06:15:13Z

First of all, thank you for the very detailed and comprehensive bug report! I wish all bug reports were like this 👍

I think the following patch for the patched QT should fix the issue you mentioned:

diff --git a/src/gui/painting/qprintengine_pdf.cpp b/src/gui/painting/qprintengine_pdf.cpp
index fd4e297..5dac92b 100644
--- a/src/gui/painting/qprintengine_pdf.cpp
+++ b/src/gui/painting/qprintengine_pdf.cpp
@@ -175,7 +175,7 @@ bool QPdfEngine::begin(QPaintDevice *pdev)
 bool QPdfEngine::end()
 {
     Q_D(QPdfEngine);
-   
+
     uint dests;
     if (d->anchors.size()) {
         dests = d->addXrefEntry(-1);
@@ -185,8 +185,8 @@ bool QPdfEngine::end()
             d->printAnchor(i.key());
             d->xprintf(" %d 0 R\n", i.value());
         }
-        d->xprintf(">>\n");
-    }  
+        d->xprintf(">>\nendobj\n");
+    }

     if (d->outlineRoot) {
         d->outlineRoot->obj = d->requestObject();
@@ -608,7 +608,7 @@ int QPdfEnginePrivate::gradientBrush(const QBrush &b, const QMatrix &matrix, int
                 ">>\n"
                 "stream\n"
               << content
-              << "endstream\n"
+              << "\nendstream\n"
                 "endobj\n";

             int softMaskFormObject = addXrefEntry(-1);
@@ -1236,7 +1236,7 @@ int QPdfEnginePrivate::writeImage(const QByteArray &data, int width, int height,
             xprintf(">>\nstream\n");
         len = writeCompressed(data);
     }
-    xprintf("endstream\n"
+    xprintf("\nendstream\n"
             "endobj\n");
     addXrefEntry(lenobj);
     xprintf("%d\n"
@@ -1376,7 +1376,7 @@ void QPdfEnginePrivate::embedFont(QFontSubset *font)
             "/CapHeight " << properties.capHeight.toReal()*scale << "\n"
             "/StemV " << properties.lineWidth.toReal()*scale << "\n"
             "/FontFile2 " << fontstream << "0 R\n"
-            ">> endobj\n";
+            ">>\nendobj\n";
         write(descriptor);
     }
     {
@@ -1394,7 +1394,7 @@ void QPdfEnginePrivate::embedFont(QFontSubset *font)
             "stream\n";
         write(header);
         int len = writeCompressed(fontData);
-        write("endstream\n"
+        write("\nendstream\n"
               "endobj\n");
         addXrefEntry(length_object);
         xprintf("%d\n"
@@ -1622,7 +1622,7 @@ void QPdfEnginePrivate::writePage()
     xprintf("stream\n");
     QIODevice *content = currentPage->stream();
     int len = writeCompressed(content);
-    xprintf("endstream\n"
+    xprintf("\nendstream\n"
             "endobj\n");

     addXrefEntry(pageStreamLength);

However, the following errors are still coming, and I don't really have a clue why they are being generated:

The filehello.pdf is not valid, error(s) :
1.1 : Body Syntax error, Second line must contains at least 4 bytes greater than
 127
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, The operator "f" can't be used without Color Profil
e
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
3.1.11 : Invalid Font definition, The CIDSet entry is missing for the Composite
Subset
3.2.3 : Font damaged, The FontFile can't be read
2.4.3 : Invalid Color space, DestOutputProfile is missing
3.1.11 : Invalid Font definition, The CIDSet entry is missing for the Composite
Subset
3.2.3 : Font damaged, The FontFile can't be read
2.4.3 : Invalid Color space, DestOutputProfile is missing
1.4.1 : Trailer Syntax error, The trailer dictionary doesn't contain ID
1.2.5 : Body Syntax error, Stream length is invalide
1.2.5 : Body Syntax error, Stream length is invalide
7.1 : Error on MetaData, Missing Metadata Key in catalog
1.4.9 : Trailer Syntax error, Outline Hierarchy doesn't have Count entry

ashkulz · 2014-03-10T11:32:33Z

@lindi2: any updates? otherwise I will commit the above changes and close the issue.

lindi2 · 2014-04-01T11:18:44Z

@ashkulz I don't have build environment setup right now. Did you start getting those other validation errors when you updated QT? I don't get them with the exact versions listed in the report.

ashkulz · 2014-04-01T11:55:48Z

@lindi2: I am getting them on the updated QT while testing on Windows. I don't think there has been a change in the relevant code for a long time... can you check after compiling with the latest code?

lindi2 · 2014-04-01T12:07:31Z

@ashkulz I can put that to my todo list but it'll take some time to setup the build system.

ashkulz · 2014-04-01T12:17:07Z

let me see if I can build and upload a new development snapshot, will take a day or two for compiling all the combinations.

ashkulz · 2014-04-02T11:12:38Z

@lindi2: setting up a development environment is very simple, see INSTALL.md. Either way, a new development snapshot is available and linked from the website.

lindi2 · 2014-04-02T12:51:00Z

@ashkulz sure, I just need to find the time to do it :) Anyways, wkhtmltox-0.12.1-b3e000e_linux-wheezy-amd64.tar.xz generates the following validation errors here:

1.1 : Body Syntax error, Second line must contains at least 4 bytes greater than 127
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, The operator "f" can't be used without Color Profile
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
3.1.11 : Invalid Font definition, The CIDSet entry is missing for the Composite Subset
3.2.3 : Font damaged, The FontFile can't be read
2.4.3 : Invalid Color space, DestOutputProfile is missing
3.1.11 : Invalid Font definition, The CIDSet entry is missing for the Composite Subset
3.2.3 : Font damaged, The FontFile can't be read
2.4.3 : Invalid Color space, DestOutputProfile is missing
1.4.1 : Trailer Syntax error, The trailer dictionary doesn't contain ID
1.2.5 : Body Syntax error, Stream length is invalide
1.2.5 : Body Syntax error, Stream length is invalide
7.1 : Error on MetaData, Missing Metadata Key in catalog
1.4.9 : Trailer Syntax error, Outline Hierarchy doesn't have Count entry

Could git bisect be used to figure out which commit created these? Does git bisect work sensibly with git submodules (I'm assuming that qt is a git submodule)?

ashkulz · 2014-04-02T17:30:15Z

Are you saying you didn't get any other errors with 0.12.0?

lindi2 · 2014-04-03T11:54:11Z

That's right. With wkhtmltox-linux-amd64_0.12.0-03c001d.tar.xz I only get those two validation errors. Anyways, I got a VM for building wkhtmltopdf now and built b3e000e which gives me

The filehello.pdf is not valid, error(s) :
1.1 : Body Syntax error, Second line must contains at least 4 bytes greater than 127
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, The operator "f" can't be used without Color Profile
2.4.3 : Invalid Color space, DestOutputProfile is missing
2.4.3 : Invalid Color space, DestOutputProfile is missing
3.1.11 : Invalid Font definition, The CIDSet entry is missing for the Composite Subset
3.2.3 : Font damaged, The FontFile can't be read
2.4.3 : Invalid Color space, DestOutputProfile is missing
3.1.11 : Invalid Font definition, The CIDSet entry is missing for the Composite Subset
3.2.3 : Font damaged, The FontFile can't be read
2.4.3 : Invalid Color space, DestOutputProfile is missing
1.4.1 : Trailer Syntax error, The trailer dictionary doesn't contain ID
1.2.5 : Body Syntax error, Stream length is invalide
1.2.5 : Body Syntax error, Stream length is invalide
7.1 : Error on MetaData, Missing Metadata Key in catalog
1.4.9 : Trailer Syntax error, Outline Hierarchy doesn't have Count entry

just like with the static binary as expected.

ashkulz · 2014-04-03T13:14:43Z

There aren't any changes to the PDF rendering code, let me recompile 0.12.0 and verify it.

lindi2 · 2014-04-03T15:00:37Z

Build instructions seem to have changed between these two versions so it's not exactly trivial to git bisect I guess?

ashkulz · 2014-04-04T12:41:03Z

Verified, just two errors reported for 0.12.0. Bisection is not straightforward because QT is a submodule and the changes are on two separate branches (rebased with the wkhtmltopdf-specific patches). I'll look at it over the weekend.

ashkulz · 2014-04-06T12:17:26Z

It looks like you start getting these errors as soon as you add the missing endobj -- probably the other errors were getting masked by it and are now getting reported. Do you have any idea on what they mean?

lindi2 · 2014-04-07T08:05:37Z

1.1 : Body Syntax error, Second line must contains at least 4 bytes greater than 127

The PDF specification at http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/devnet/pdf/pdfs/PDF32000_2008.pdf says 👍
"If a PDF file contains binary data, as most do (see 7.2, "Lexical Conventions"), the header line shall be
immediately followed by a comment line containing at least four binary characters—that is, characters whose
codes are 128 or greater.
"

ashkulz · 2014-04-07T10:40:06Z

If you can give me explanations like that, I think we can crack this very soon 👍 I've added a potential fix in the issue1573 branch, am compiling and testing it right now.

lindi2 · 2014-04-07T10:42:56Z

I already made such a patch and tested that it works. Working on more patches at the moment.

ashkulz · 2014-04-07T10:44:48Z

Great 👍 I'll wait for a pull request from you, then.

lindi2 · 2014-04-07T10:46:07Z

Is there some way to incrementally rebuild only changed files? "scripts/build.py wheezy-amd64" doesn't seem to pick up my changes to qt/src/gui/painting/qprintengine_pdf.cpp

ashkulz · 2014-04-07T10:48:00Z

You'll have to do it manually, I'm afraid. What I do is make install inside static-build/wheezy-amd64/qt_build and make inside static-build/wheezy-amd64/app. Having support for incremental builds is on my TODO list.

lindi2 · 2014-04-07T14:29:13Z

Couldn't figure out how to do incremental builds so had to wait a while for full builds. Does wkhtmltopdf/qt#6 look ok to you?

ashkulz · 2014-04-08T16:00:45Z

I've implemented incremental builds in 3995b6e, let me know if it works for you.

pinx · 2014-06-17T16:05:39Z

Using the latest version solved my problems. I haven't investigated, but I found a bug in my haml file, that was the basis for the pdf. I accidentally put the body tag inside the head tag. A html validator pointed out that I was generating invalid html. Solved this bug and installed the latest wkhtmltopdf, and things are working for me.
Do you check generated pdf with a pdf validator? I got some warnings, still (although my pdf viewers no longer complain).

ashkulz · 2014-06-18T06:36:52Z

The issues are with the upstream Qt PDF generation code, so will have to be fixed/pushed upstream first. Considering that it works fine in most viewers, it is not a high priority as such 😄

ashkulz added the Verified label Mar 6, 2014

ashkulz added this to the 0.12.1 milestone Mar 6, 2014

ashkulz closed this as completed in bc91350 Mar 28, 2014

ashkulz added Fixed and removed Verified labels Mar 28, 2014

ashkulz mentioned this issue Apr 2, 2014

PDF Accessibility #1605

Closed

ashkulz reopened this Apr 4, 2014

ashkulz added Verified and removed Fixed labels Apr 4, 2014

lindi2 mentioned this issue Apr 7, 2014

Pdf validation fixes wkhtmltopdf/qt#6

Closed

ashkulz modified the milestones: future, 0.12.1 Apr 14, 2014

ashkulz mentioned this issue Jun 2, 2014

Format identifier invalid (zero) #1754

Closed

ashkulz mentioned this issue Dec 17, 2014

Width for character x in font 'XYZ' does not match. #2075

Closed

ashkulz mentioned this issue May 1, 2018

Creating a PDF/A document #2585

Closed

alloylab added the UpstreamChangeNeeded label Jun 15, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[qt] generated PDF does not pass the PDF validations #1573

[qt] generated PDF does not pass the PDF validations #1573

lindi2 commented Mar 5, 2014

ashkulz commented Mar 6, 2014

ashkulz commented Mar 10, 2014

lindi2 commented Apr 1, 2014

ashkulz commented Apr 1, 2014

lindi2 commented Apr 1, 2014

ashkulz commented Apr 1, 2014

ashkulz commented Apr 2, 2014

lindi2 commented Apr 2, 2014

ashkulz commented Apr 2, 2014

lindi2 commented Apr 3, 2014

ashkulz commented Apr 3, 2014

lindi2 commented Apr 3, 2014

ashkulz commented Apr 4, 2014

ashkulz commented Apr 6, 2014

lindi2 commented Apr 7, 2014

ashkulz commented Apr 7, 2014

lindi2 commented Apr 7, 2014

ashkulz commented Apr 7, 2014

lindi2 commented Apr 7, 2014

ashkulz commented Apr 7, 2014

lindi2 commented Apr 7, 2014

ashkulz commented Apr 8, 2014

pinx commented Jun 17, 2014

ashkulz commented Jun 18, 2014

[qt] generated PDF does not pass the PDF validations #1573

[qt] generated PDF does not pass the PDF validations #1573

Comments

lindi2 commented Mar 5, 2014

ashkulz commented Mar 6, 2014

ashkulz commented Mar 10, 2014

lindi2 commented Apr 1, 2014

ashkulz commented Apr 1, 2014

lindi2 commented Apr 1, 2014

ashkulz commented Apr 1, 2014

ashkulz commented Apr 2, 2014

lindi2 commented Apr 2, 2014

ashkulz commented Apr 2, 2014

lindi2 commented Apr 3, 2014

ashkulz commented Apr 3, 2014

lindi2 commented Apr 3, 2014

ashkulz commented Apr 4, 2014

ashkulz commented Apr 6, 2014

lindi2 commented Apr 7, 2014

ashkulz commented Apr 7, 2014

lindi2 commented Apr 7, 2014

ashkulz commented Apr 7, 2014

lindi2 commented Apr 7, 2014

ashkulz commented Apr 7, 2014

lindi2 commented Apr 7, 2014

ashkulz commented Apr 8, 2014

pinx commented Jun 17, 2014

ashkulz commented Jun 18, 2014