Add support for testing based on uncompressed PDF streams

[josephwright]

It is possible to produce uncompressed PDF streams, which allow those versed in PDF to examine the detailed output of a TeX run and to check on aspects that may be difficult/impossible to test or debug from the macro layer/\tracingall. Adding support for this area would allow both testing and debugging abilities to be enhanced.

The interface for this is to be decided, but seems likely to use a marker in the .lvt which tells l3build to read the PDF, followed by manipulation of the latter to extract the 'useful' parts.

[wspr]

I initially thought “well, if TeX is writing the data to the PDF stream via XYZ approach then it could just write it to the log as well”, but obviously you don’t want to have duplicate code paths just for testing purposes! So makes sense to me.

[josephwright]

A suggested test environment here is based around

   \STREAMTEST#1#2 =>
     \pdfliteral direct {\%\space l3build test \space"#1"}
     #2
     \pdfliteral direct{\%\space l3build test end}

with some marker to the .log to tell l3build to look at the PDF (something like \CHECKPDF or perhaps \CHECKPDFSTREAM).

[josephwright]

The -p switch would probably be dropped in this approach, likely replaced by \CHECKPDFBINARY or something similar.

[blefloch]

Yes, a command in the test source seems better than a switch that one has to remember (and presumably make texlua build.lua ctan correctly pass along).

[josephwright]

@blefloch Good point: I'd not considered that (also applies to 'mixed' testing situation).

I'll try to come up with some proposals today if I have time: probably another branch so we can argue over the detail.

[FrankMittelbach]

Am 02.07.17 um 02:32 schrieb Bruno Le Floch:

Yes, a command in the test source seems better than a switch that one has to remember (and presumably make |texlua build.lua ctan| correctly pass along).

agreed. one would want to have tests that are self-contained and automatically work switches should only do things like restricting what to test, what to show from the results, how to deal with errors and the like but not needed to make a test work in the first place, ie a simple "check" target should always be able to correctly process all tests automatically

[josephwright]

OK, I need some names for the relevant macros. I was thinking \CHECKPDFSTREAM and \CHECKPDFBINARY. (I considered a two-part \GENERATEPDF + \CHECKSTREAM/\CHECKBINARY, but that is more work for the user, and what happens if the first marker is missed.) Names OK?

[blefloch]

The names sound good, assuming they do roughly the following:

• \CHECKPDFBINARY instructs l3build to compare the binary pdf resulting from the run to some saved binary pdf result (and with texlua build.lua save -exetex test001 say, the result of the xetex run is presumably saved as test001.xetex.ext for some extension ext)

• \CHECKPDFSTREAM instructs l3build to cancel pdf compression (it's also ok if that's always the case) and extract from the resulting pdf some regions marked by some markers in the pdf (I didn't follow the discussions you had with Javier on these markers).

If both \CHECKPDFBINARY and \CHECKPDFSTREAM are given, error.

[car222222]

This could need a new issue?

As maybe Will was suggesting, it is essential to test exactly what is written out to external files, including the .pdf file, by any mechanism: writes, pdfliteral etc. This should, as Will said, be done without explicitly reproducing the material and writing it to the log file; and is better done without explicitly in the external file. Except for \immediate operations, this information needs to be located within the boxes traced by \showoutput .

Reasons: For output such as pdfliteral, or a pdf: special, testing at this stage (without looking into the .pdf file) tests precisely 'what LaTeX does' and this is what the test suite should test. (If you only compare what gets into the pdf file then, in the case of a diff, you do not know whether something in the LaTeX setup has caused the change or if it is caused by a problem in the process that produces the content of the pdf file (and that is external to LaTeX).

Note that \writes, \specials (if not rejected) and \pdfliterals appear in the box data from \showoutput

I did not know how to trace the (expanded) content of an \immediate write.

Are there other primitives for output that need to be traced?

[josephwright]

@car222222 I'm not sure I follow: Javier's request was related to areas that are hard/impossible to debug at the macro end as one has to be sure that the binaries are 'behaving'. (The ordering and exact nature of \special instructions can be vital in this regard, and that's simply not visible to the macro layer.)

[car222222]

I said that it might be a new issue.

I do not understand the relevance of 'binaries', as Javier needs an uncompressed PDF for his tests.

With the correct settings, the content of a \special is written to the .log file as part of he contents of the box that is shipped out. Thus visibility to the macro layer (even simply) is not required to test this output. Same for \pdfliteral and \write but the contents of an \immediate \write do not appear to be tracable.

[FrankMittelbach]

Am 03.07.17 um 10:25 schrieb Chris Rowley:

With the correct settings, the content of a \special is written to the .log file as part of he contents of the box that is shipped out. Thus visibility to the macro layer (even simply) is not required to test this output. Same for \pdfliteral and \write but the contents of an \immediate \write do not appear to be tracable.

as I understand it the specials are only there to mark regions in the pdf and it is then region that he checks for spotcolor or something (didn't quite follow the discussion, nor is this something I understand much about).

[josephwright]

@car222222 What we ask for in specials and what the binaries (pdfTeX, dvips, dvipdfmx, ...) do aren't necessarily the same, so if you are seeking a particular outcome you have to check the 'end result'. This shows up once one starts doing colour/graphics/hyperlink/bookmark stuff of any complexity. So my understanding is that Javier wants a way to test what the user actually gets for some particular cases, not what we think the macro layer asks for. It's a somewhat specialist area but it's not unreasonable to cover it.

[josephwright]

@FrankMittelbach Yup, that's more or less it: \pdfliteral here is just being used to do the equivalent of \OMIT/\TIMO or \TEST, but for the PDF stream rather than the .log.

[car222222]

Sorry: clearly I should have made it new issue. I was not suggesting that this could be used for Javier's case. I was mainly responding to Will, as he seems to be implying that it was impossible to get such things into the .log file. Also to note that the contents of immediate things are, I think, unloggable, or are they?

[car222222]

What we ask for in specials and what the binaries (pdfTeX, dvips, dvipdfmx, ...) do aren't necessarily the same, so if you are seeking a particular outcome you have to check the 'end result'.

That is what I said earlier. It is the reason why, when possible, one should also (in many cases and where possible) test what LaTeX outputs. Then you can tell if a fail is due to LaTeX or due to some change in what 'the binaries' have done to the output from LaTeX.

(BTW: not sure I like executables/processes being called 'binaries' as they may or not be in binary form; is that standard jargon now?)

[josephwright]

@car222222 One can of course can (and should) test the macro layer, but that's what we already have, so I'm not sure what the issue is there.

On 'binaries', I tend to use that to differentiate from the macro layer: they are executable programs, not scripts. (The latter to me as a 'Windows person' won't be executables either in the main ...)

[car222222]

So is 'the macro layer' what I call 'LaTeX'? Just getting this clear.

In that case the only issues left are how to log the contents of \immediate output; and whether I interpreted Will's input correctly:-).

Am I correct that \immediate \write etc cannot be traced in detail?

[josephwright]

A couple of notes from the TeX Live list re. behaviour of (x)dvipdfmx in this area.

dvipdfmx deliberately drops literal lines starting % and also drops whitespace. @davidcarlisle therefore suggested something like

\special{pdf:literal direct (OMIT THIS LINE) pop}

as an approach that would work (with care).

To get no PDF compression you need a couple of not-really documented specials:

\special{dvipdfmx:config z 0}% ~ \pdfcompresslevel
\special{dvipdfmx:config C 0x40}% ~ \pdfobjcompresslevel

which avoids needing to manually run xdvipdfmx for XeTeX and does seem to have the desired effect.

[josephwright]

Also worth noting that the pdfTeX and dvipdfmx produce rather different results at the low level: dvipdfmx doesn't start a new line for the special and surrounds it with ET/BT, whereas for pdfTeX it is literally inserted directly in the text but does stand on a line on its own. All probably workable as the results will be highly route-dependent anyway ...

[josephwright]

I've now looked at binary-based PDF comparison in a137f8c. That shows up that whilst you can automatically make PDFs, the binaries are not platform-independent. Thus it may well be the case that the best set up for us is stream-based testing. That would simplify some aspects here, as we then don't need two separate PDF routes.

If no one objects, I'll drop binary comparison and re-work for stream support instead.

[wspr]

Right, it sounds like testing at the binary level isn't feasible. Thanks for clarifying.

[josephwright]

@wspr Well, for 'reproducible build' uses work on a binary level, but those are single-platform. That was the original motivation for looking at PDF-based testing, but I agree that for us it probably doesn't help so much.

[josephwright]

This is now done.