Reproducible PS/PDF output (master) #6597

ghost · 2016-06-16T21:45:21Z

This is a rebase of #6595 to the master branch.

Several software packages use matplotlib in their building process (mainly to produce PS or PDF documents). To make their build reproducible, it would be great to make matplotlib output reproducible.

To allow reproducible PS and PDF output:

honour SOURCE_DATE_EPOCH for timestamps in PS and PDF files.
See https://reproducible-builds.org/specs/source-date-epoch/
get keys sorted so that hatch patterns, images and markers are included with
a reproducible order in the PDF file. Another solution is to sort self.hatchPatterns in writeHatches (and similar ordering in writeImages and writeMarkers), but this consumes more memory.

This patch has been submitted in debian bug #827361

See also https://reproducible-builds.org/

* honour SOURCE_DATE_EPOCH for timestamps in PS and PDF files. See https://reproducible-builds.org/specs/source-date-epoch/ * get keys sorted so that hatchPatterns, images and markers are included with a reproducible order in the PDF file. See https://reproducible-builds.org/

tacaswell · 2016-06-17T15:53:37Z

I think we are already sorting all of those keys on the way out so do not need the ordered dicts.

Can you add an entry in https://github.com/matplotlib/matplotlib/tree/master/doc/users/whats_new and a test (like the deterministic svg test) that this actually works?

attn @jkseppan

jkseppan · 2016-06-17T16:39:58Z

I agree that a test is needed. I don't see the reason for the ordered dicts either, but there could be some subtlety that I'm missing. It would be neat to make the date settable by the user (via e.g. the PdfPages constructor) but of course not necessary for this particular use case.

- SOURCE_DATE_EPOCH support - reproducible output

ghost · 2016-06-18T16:16:15Z

I added a test for reproducible PDF output (you're right: this is very important!).
The test shows that the ordered dicts are needed for markers and hatches, due to the dict iterations at the beginning of writeHatches and writeMarkers, which are unreproducible with python3.
Even if the same iteration occurs in writeImages, the tests passes without ordering _images. I think this is because the _images keys are integers.
I also added two orderings (related to fonts), that appeared to be essential thanks to the test.

jkseppan · 2016-06-18T16:35:18Z

Oh, of course! I was thinking about the page stream, but it also matters in which order the objects are output at the top level of the pdf file. I suspect that the ordering is needed for _images too - maybe the dict implementation is a list for integer-keyed small dictionaries, or something.

ghost · 2016-06-18T19:45:35Z

I would like to add tests for postscript too, but I'm afraid the code will be nearly the same as for PDF: do you think I should move the common code somewhere (maybe in a new testing/reproducible.py file), or duplicate it in test_backend_ps.py and test_backend_pdf.py?

…pendent.

tacaswell · 2016-07-08T01:24:15Z

.SK...
======================================================================
FAIL: Test SOURCE_DATE_EPOCH support for PS output
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Miniconda-x64\envs\test-environment\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 152, in wrapped_callable
    func(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\tests\test_backend_ps.py", line 186, in test_source_date_epoch
    _test_source_date_epoch("ps", b"%%CreationDate: Sat Jan  1 00:00:00 2000")
  File "c:\projects\matplotlib\lib\matplotlib\testing\determinism.py", line 122, in _test_source_date_epoch
    assert string in buff
AssertionError
--------------------------------------------------------------------

These failures on appveyor look real.

tacaswell · 2016-07-08T01:24:56Z

My naive guess is that there is a unicode vs bytes issues

ghost · 2016-07-08T09:22:15Z

These failures on appveyor look real.

Sure! I'm working on it but for now I can't reproduce the problem at home.

…and test_determinism_all

…environment is shared between threads).

…th, day of month and times. This way we don't mind if timestamps are written with leading 0 or space.

ghost · 2016-07-11T09:54:03Z

I think this is better now.

For parallel tests, I think threads are used, and they share the same environment. So setting the SOURCE_DATE_EPOCH environment variable in a test is not good. So I used subprocesses instead.
The timestamp was 'Jan 1' or 'Jan 01' sometimes, so I used Dec 15 instead to solve this issue.

Appveyor is now reporting only a HTTPError: 500 Server Error. I think this is not related to Matplotlib code, but I don't know if I can restart the build.

tacaswell · 2016-07-11T23:43:48Z

👍 from me, attn @jkseppan can you do a final review?

jkseppan · 2016-07-12T05:53:31Z

doc/users/whats_new/reproducible_ps_pdf.rst

+
+The ``SOURCE_DATE_EPOCH`` environment variable can now be used to set
+the timestamps value in the PS and PDF outputs, which are then
+reproducible.


We should list the known limitations, because when downstream users see this promise, they may start depending on it and run into the limitations later. At least usetex with the ps backend should be ruled out, based on comments in the tests.

Please add a link to the specification that describes the value of this environment variable.

Other possible limitations include at least mathtext, ps.usedistiller, ps.fonttype, and pdf.fonttype. These are features or settings that I think could easily hide a source of nondeterminism and that don't seem to get exercised by the new test. (To get a more reliable list of relevant features than just my guesses, you'd need to run the test with coverage checking and see which parts of the backends get no coverage.)

To be clear, I don't mean we should delay merging until there are tests for everything, but we should be realistic in how we document this. I imagine the reproducible-builds community would appreciate the feature as it is now, with careful documentation of what exactly they can rely on.

jkseppan · 2016-07-12T06:01:00Z

Commit c007f49 changes the tests to use a two-digit date in December instead of January 1st. I think the underlying problem should be fixed instead of avoiding it in the tests, since we don't want to tell users that their SOURCE_DATE_EPOCH values must be two-digit days.

ghost · 2016-07-12T07:05:20Z

I considered reproducibility this way: with the same tools (same versions of all what you installed on your computer), the same commands lead to the same results.
With this point of view, I accept that with some different versions/implementations of python, the date can be written different ways (leading space or 0 for example). The problem with the tests was that we face different implementations with different behaviors. We don't pretend that the output will be exactly the same with all future versions of all the tools, all implementations of python.
I also accept that ghostscript can change its way to format the timestamp in a future version, but I did not include a test with usetex relying to the current behavior of ghostscript. Maybe this is too prudent.
Let me try to be more explicit about that in the doc.

Kojoley · 2016-10-13T13:01:16Z

I think the failure is due to ghostscript (some versions now honors SOURCE_DATE_EPOCH, but some don't).

The fail happens on the build where ghostscript is not installed, so bf7387e commit is not the right way to handle such situation.

ghost · 2016-10-15T07:58:10Z

The [XPASS(strict)] This test needs a ghostscript installation fail happened because the test should have failed (tagged @needs_ghostscript) but did not fail, because usetex was not properly passed to _determinism_save before af4213e, so that the test ran without using ghostscript.
Correcting this, I then saw that now that usetex is properly passed, the test using usetex failed because of ghostscript, which does not honour SOURCE_DATE_EPOCH in the installed version. So I applied bf7387e so that we can see if the test pass (good ghostscript version) or not, but without failing.

tacaswell · 2016-11-01T02:39:02Z

'power cycled to restart CI (appveyor failures looked due to qt-related packaging issues).

tacaswell

Resuse UTC timezone from dates.py

tacaswell · 2016-11-01T03:06:40Z

lib/matplotlib/backends/backend_pdf.py

@@ -135,6 +136,20 @@ def _string_escape(match):
    assert False


+# tzinfo class for UTC
+class UTCtimezone(tzinfo):


We already have this in mpl/dates.py

https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/dates.py#L155

Yes! Sorry I missed that.

tacaswell

Resuse UTC timezone from dates.py

tacaswell

Resuse UTC timezone from dates.py

tacaswell · 2016-11-01T03:39:11Z

Other than one minor change, this looks good to go!

Don't worry about the coverall failure, that is un-related.

Sorry this has dragged on so long.

QuLogic · 2016-11-02T19:35:19Z

The coveralls drop is probably due to a change in master, and not this PR; a rebase to latest master might help that, but is not strictly necessary.

QuLogic

A couple of minor things.

QuLogic · 2016-11-02T19:35:37Z

doc/users/whats_new/reproducible_ps_pdf.rst

+------------------------------
+
+The ``SOURCE_DATE_EPOCH`` environment variable can now be used to set
+the timestamps value in the PS and PDF outputs. See


QuLogic · 2016-11-02T19:58:30Z

lib/matplotlib/testing/determinism.py

+        default value is "mhi", so that the test includes all these objects.
+    format : str
+        format string. The default value is "pdf".
+    uid : str


Not seeing this being used for anything?

The uid option is used in test_determinism_all_tex from lib/matplotlib/tests/test_backend_ps.py

But it was only useful when files were used to store the figures. I will remove that.

QuLogic · 2016-11-02T20:01:39Z

lib/matplotlib/testing/determinism.py

+        format string, such as "pdf".
+    string : str
+        timestamp string for 2000-01-01 00:00 UTC.
+    keyword : str


It appears to be bytes, not str.

ghost · 2016-11-03T20:53:06Z

@QuLogic: thanks a lot for your review. I corrected those points.

QuLogic · 2016-12-07T06:09:43Z

A rebase might fix coveralls (as I'm pretty sure it was a change in master that dropped coverage), but it's not strictly needed. I guess we aren't really waiting on anything else here.

ghost · 2016-12-09T15:45:23Z

Thanks to you all for your kind support!

tacaswell · 2016-12-14T21:47:25Z

@JojoBoulix Thanks for taking care of this! Sorry it was such a protracted process.

mdboom added the status: needs review label Jun 16, 2016

ghost mentioned this pull request Jun 16, 2016

Reproducible PS/PDF output #6595

Closed

tacaswell added this to the 2.1 (next point release) milestone Jun 17, 2016

Alexis Bienvenüe added 3 commits June 18, 2016 17:42

Reproducible PDF output: sort TTF characters

4e477dc

Reproducible PDF output: sort fonts

946ae45

Tests for determinist PDF output:

22f71a2

- SOURCE_DATE_EPOCH support - reproducible output

Alexis Bienvenüe added 5 commits June 27, 2016 20:35

Share determinism test code for PS and PDF output.

ad660d7

Reproducible PS/tex output.

37c28b4

Removes test_source_date_epoch_tex test, since this is ghostscript de…

2e1c773

…pendent.

PEP8

541f97e

Add what's new section

2a6ebc8

Add some insight when test_source_date_epoch fails.

6ee8967

Alexis Bienvenüe added 4 commits July 8, 2016 12:16

Allow parallel execution of test_backend_ps:test_determinism_all_tex …

a3185e6

…and test_determinism_all

Use subprocess for _test_source_date_epoch, to allow parallel calls (…

8f095f6

…environment is shared between threads).

Change SOURCE_DATE_EPOCH test date, to use two-digits numbers for mon…

c007f49

…th, day of month and times. This way we don't mind if timestamps are written with leading 0 or space.

PEP8

ecbdd55

jkseppan reviewed Jul 12, 2016
View reviewed changes

tacaswell closed this Nov 1, 2016

tacaswell reopened this Nov 1, 2016

tacaswell changed the title ~~Reproducible PS/PDF output (master)~~ [MRG+1] Reproducible PS/PDF output (master) Nov 1, 2016

tacaswell requested changes Nov 1, 2016

View reviewed changes

tacaswell changed the title ~~[MRG+1] Reproducible PS/PDF output (master)~~ [MRG] Reproducible PS/PDF output (master) Nov 1, 2016

Reuse UTC timezone from dates.py

76bec02

tacaswell approved these changes Nov 2, 2016

View reviewed changes

QuLogic reviewed Nov 2, 2016

View reviewed changes

Alexis Bienvenüe added 2 commits November 3, 2016 21:46

Removes now useless option uid for _determinism_check

bbab0c5

Typo

1a5ada6

QuLogic changed the title ~~[MRG] Reproducible PS/PDF output (master)~~ [MRG+1] Reproducible PS/PDF output (master) Dec 7, 2016

QuLogic merged commit eadafc6 into matplotlib:master Dec 9, 2016

QuLogic changed the title ~~[MRG+1] Reproducible PS/PDF output (master)~~ Reproducible PS/PDF output (master) Dec 9, 2016

QuLogic removed the status: needs review label Dec 9, 2016

QuLogic mentioned this pull request Dec 9, 2016

PDF file generation is not deterministic - results in different outputs on the same input #6317

Closed

QuLogic added backend: pdf backend: ps labels Dec 9, 2016

jkseppan mentioned this pull request Jan 5, 2017

MAINT: Deterministic SVG and PDF tests #7748

Merged

edemaine mentioned this pull request Feb 25, 2018

Compiled PDFs are not reproducibly built edemaine/svgtiler#24

Closed

Reproducible PS/PDF output (master) #6597

Reproducible PS/PDF output (master) #6597

Conversation

ghost commented Jun 16, 2016

tacaswell commented Jun 17, 2016

jkseppan commented Jun 17, 2016

ghost commented Jun 18, 2016

jkseppan commented Jun 18, 2016

ghost commented Jun 18, 2016

tacaswell commented Jul 8, 2016

tacaswell commented Jul 8, 2016

ghost commented Jul 8, 2016

ghost commented Jul 11, 2016

tacaswell commented Jul 11, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkseppan commented Jul 12, 2016

ghost commented Jul 12, 2016

Kojoley commented Oct 13, 2016

ghost commented Oct 15, 2016

tacaswell commented Nov 1, 2016

tacaswell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tacaswell left a comment

Choose a reason for hiding this comment

tacaswell left a comment

Choose a reason for hiding this comment

tacaswell commented Nov 1, 2016

QuLogic commented Nov 2, 2016

QuLogic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Nov 3, 2016

QuLogic commented Dec 7, 2016

ghost commented Dec 9, 2016

tacaswell commented Dec 14, 2016