Unit tests failing for 6.9.9-33 in Fedora Rawhide (28)

[mooninite]

Description

FAIL: tests/validate-formats-map.tap 1
FAIL: tests/validate-formats-disk.tap 1
ERROR: tests/validate-formats-memory.tap - too few tests run (expected 1, got 0)

Steps to Reproduce

$ fedpkg clone ImageMagick
$ vi ImageMagick.spec #update Patchlevel to 33
$ fedpkg mockbuild

System Configuration

Fedora Rawhide (28) - all architectures
ImageMagick 6.9.9-33

[urban-warrior]

The likely fix is already in ImageMagick 6.9.9-34 Beta, however, we can't sure without seeing the output of tests/validate-formats-memory.log to see where the failure occurs. We are ready to push out the 6.9.9-34 release but are waiting on the Windows build, which is currently failing due to the new OSS fuzzers we added.

[mooninite]

I tried a -34 beta (2018-01-15) and it fails to compile. I'll wait for a different beta or the final release and try again.

Q16.xs:145:3: error: redeclaration of enumerator 'UndefinedReference'
   UndefinedReference = 0,
   ^~~~~~~~~~~~~~~~~~
In file included from ../../magick/annotate.h:21:0,
                 from ../../magick/MagickCore.h:81,
                 from Q16.xs:60:
../../magick/draw.h:125:3: note: previous definition of 'UndefinedReference' was here
   UndefinedReference,
   ^~~~~~~~~~~~~~~~~~

Q16.xs:227:45: error: 'NullReference' undeclared here (not in a function); did you mean 'FileReference'?
     { "Despeckle", { { (const char *) NULL, NullReference } } },
                                             ^~~~~~~~~~~~~
                                             FileReference

[urban-warrior]

Try the beta now. It should compile without complaint.

[mooninite]

Yes, it compiles now, but the unit tests still fail. The issue is that Fedora Rawhide ghostscript has re-structured its sub-packages. I am discussing this change with the ghostscript maintainer. It may or may not require an ImageMagick change.

[urban-warrior]

As you likely know, ImageMagick builds with the Ghostscript libs under Fedora 27 when we set the --with-gslib=yes configure script command-line option. If we could get access to Fedora Rawhide, we could look toward a configure.ac mod to link with the Ghostscript development libraries under Rawhide. If in fact the Ghostscript development libraries are broken, the Ghostscript developers will of course need to fix that problem.

[mooninite]

It's very easy to build in a chrooted Rawhide environment from a Fedora 27 machine. That's how I am reporting all of this to you...

$ vi ImageMagic.spec #update to beta build -34
$ fedpkg srpm
$ mock -v -r fedora-rawhide-x86_64 --rebuild ./ImageMagick-6.9.9.34-1.fc28.src.rpm
If you need a shell:
$ mock -v -r fedora-rawhide-x86_64 --shell
Otherwise you can see the Rawhide file system:

• Root: /var/lib/mock/fedora-rawhide-x86_64/root
• Build directory: /var/lib/mock/fedora-rawhide-x86_64/root/builddir/build/BUILD/ImageMagick-6.9.9-34/

[deekej]

Hello guys! :)

This is the new ghostscript package layout:

• libgs - library providing Ghostscript's core functionality. Includes all necessary files for this library to function:
  - Ghostscript's library - /usr/share/lib64/libgs.so.*
  - ICC profiles - /usr/share/ghostscript/iccprofiles/
  - library startup/configuration files - /usr/share/ghostscript/lib/
  - necessary resources - /usr/share/ghostscript/Resource/

• libgs-devel - Ghostscript's header files & unversioned symlink to library

• ghostscript - main package providing typical Ghostscript's functionality from command line
  - binaries: gs, gsnd, ghostscript
  - typical conversion scripts: eps2*, pdf2*, ps2*

• ghostscript-tools-dvipdf - contains dvipdf script (this package pulls in texlive-dvips as its dependency)

• ghostscript-tools-fonts - scripts related to work with AFM, PFB, PFA files
  - pf2afm, pfbtopfa, printafm

• ghostscript-tools-printing - scripts related to direct text printing on several printers
  - gsbj, gsdj, gsdj500, gslj, gslp
  - pphs utility

• ghostscript-x11 - X11 driver which enables displaying of PS/PDF documents with Ghostscript

• ghostscript-gtk - GTK-based binary for displaying of PS/PDF documents with Ghostscript

• ghostscript-doc - documentation files

---

The current version of Ghostscript package in Fedora Rawhide no longer has its version as part of the file paths:
/usr/share/ghostscript/9.22/Resource/Init/gs_resmp.ps
->>
/usr/share/ghostscript/Resource/Init/gs_resmp.ps

It is part of the libgs package now:
https://koji.fedoraproject.org/koji/fileinfo?rpmID=12537549&filename=/usr/share/ghostscript/Resource/Init/gs_resmp.ps

[urban-warrior]

We tried fedpkg but we got a authentication exception when we tried to clone even after we created a Fedora account. Keep in mind, its difficult for us to debug because we have more than 200 issues pending leaving us with little free time. Can you post a URL to the config.log file from this command: ./configure --with-gslib. It will tell us exactly why Ghostscript failed to validate under Fedora Rawhide. Hopefully with config.log and your details about how Ghostscript is laid out in Rawhide, we might be able to figure out a proper patch.

If Ghostscript exported a pkg-config file, it would help avoid this sorts of issues when the distribution layout changes.

[deekej]

If Ghostscript exported a pkg-config file, it would help avoid this sorts of issues when the distribution layout changes.

I will suggest this to upstream to add the support for this for their next major release. :)

[deekej]

Hello guys, any news on this issue? What do you currently need from us to be able to resolve this issue? :)

EDIT: Scratch my comment, I'll try to get you the logs.

[deekej]

Here is the config.log of ./configure --with-gslib from mock-build in Fedora Rawhide (28):
https://paste.fedoraproject.org/paste/RJ0c23YHqkTywOsqkW85kg

[urban-warrior]

We need access to a Fedora 28 host. Can someone spin up a Docker and provide credentials? We can provide our SSH public key. If we had direct access to Fedora 28, we likely could resolve this issue in about 20 minutes.

[mooninite]

You don't need a F28 host. Everything can be done from a Fedora 26 or 27 host.

To use fedpkg anonymously without being a Fedora user, just add '--anonymous' to your clone.

fedpkg clone --anonymous ImageMagick
fedpkg srpm
mock -v -r fedora-rawhide-x86_64 --rebuild ./ImageMagick-6.9.9.33-1.fc28.src.rpm
mock -v -r fedora-rawhide-x86_64 --shell

[urban-warrior]

Remove the --with-gslib option from the configure script command-line in your ImageMagick.spec rpmbuild file. The unit tests should pass now. There is something screwy about Ghostscript in Fedora 28 and we're investigating.

The units tests complete for mock fedora-27-x86_64 with ImageMagick 7.0.7-24, the current release. However for fedora-rawhide-x86_64, the same code base / units tests start leaking file descriptors when it utilizes the Ghostscript library. Utilizing Ghostscript from the command line works fine (e.g. convert logo.ps logo.png). We're trying to trace the source of the file leak and this may take several days.

[deekej]

The Ghostscript's code base remained the same for Fedora 28, but as I mentioned before, there were changes to its packaging process. It should be much closer to vanilla build now. More info here:
https://fedoraproject.org/wiki/Changes/GhostscriptPackageUpdate28

[deekej]

@urban-warrior, could you please let me know what capabilites would you need from Ghostscript's pkg-config file? Would Libs and Cflags be enough for you? Or do you want to access some other specific variables for paths?

Once I have the info, I can create the necessary pkg-config file for Ghostscript's upstream.

[urban-warrior]

Let's go with the expected configuration. Here is an example from libwebp:

prefix=/usr
exec_prefix=/usr
libdir=/usr/lib64
includedir=/usr/include

Name: libwebp
Description: Library for the WebP graphics format
Version: 0.6.1
Cflags: -I${includedir}
Libs: -L${libdir} -lwebp
Libs.private: -lm -pthread

Once its adopted by Ghostscript upstream, we would need to wait until it propagates to a variety of Linux distros before ImageMagick could rely on utilizing it.

[urban-warrior]

There appears to be a file leak when calling the Ghostscript library. ImageMagick pairs calls to gsapi_new_instance() with calls to gsapi_exit() and gsapi_delete_instance() (see coders/{ps.c,pdf.c}). This works fine under mock / Fedora 27 and other OS's such as Windows, CentOS, Mac, etc.. However, the file leak appears under mock / Fedora 28. We debugged with strace and it shows an increasing file descriptor number while Ghostscript is loading resources from the /usr/share/ghostscript folder. The file descriptor increases for each Postscript or PDF document the unit tests processes. Eventually the unit tests run out of file descriptors and the fails.

The failure can be replicated from the ImageMagick command-line. Try this command:

magick convert rose: rose.ps
magick rose.ps rose.ps rose.ps ... null:

Where the file count for rose.ps is > 1000. The command-line exits with the same failure, too many open file descriptors.

When we add --without-gslib to the configure script command-line, the unit test relies on the Ghostscript delegate program (gs) rather than the Ghostscript delegate library (gslib). With this setup, the unit tests pass.

[deekej]

Where the file count for rose.ps is > 1000. The command-line exits with the same failure, too many open file descriptors.

So, is this a bug in Ghostscript? Should I report it to upstream then? :)

Regarding the pkg-config file - that's what I was expecting, but I wasn't sure if it was enough for you or not. :) I will try to prepare the pkg-config and submit it to Ghostscript's upstream. In case it is missing something, it can be added as necessary. ;)

[urban-warrior]

Is it a bug in Ghostscript? Likely, but we can't confirm other than looking at an strace and seeing the file descriptor number jumping for each invocation of the Ghostscript library with the new_instance() API call. Its curious that this behavior is not seen with Fedora 27 (or Windows or Mac), so is it a Ghostscript version issue? Not sure. We have exhausted the extent of our knowledge and need someone else to follow the crumbs to identify the source of the file leak.

[remicollet]

Is it a bug in Ghostscript?

Perhaps not, as F27 have the same version. I'd rather think about some issue related to the build tool chain (gcc 8, glibc 2.27...)

So, is this a bug in Ghostscript? Should I report it to upstream then? :)

IMHO, Yes

[deekej]

Could you try to reproduce this issue with some small C "test harness" (not C++) that illustrates the problem? We can then open a new BZ for it in upstream. Right now, I don't have much I could give them to fix this. :)

[urban-warrior]

If its the same Ghostscript release, it could very well be a toolchain related problem. To visualize the problem, build ImageMagick and then run the unit tests with strace:

cd ImageMagick-7.0.7-26
strace tests/.libs/validate -validate formats-memory

Search for ghostscript in the strace log. You'll notice Ghostscript opens perhaps 20 resources all in the /usr/share.../Resource/ folder. The first instance of ImageMagick reading a Postscript / PDF image with the Ghostscript delegate library returns a file descriptor of 4, 5, 6, etc., for each resource it opens. The next time Ghostscript reads these same resource files, it returns 4, 10, 11, 12, etc. The next time it returns 4, 17, 18, 19, etc., suggesting a file leak. If you run the same test under Fedora 27, it returns file descriptors of 4, 5, 6, etc. for each instance the Ghostscript API is called.

Could we be wrong and the bug is in localized to ImageMagick? Of course, but this file leak does not exhibit in Fedora 27 or CentOS or Mac OS 10 or Windows 10.

If you have other debugging suggestions, let us know and we'll try to help debug this issue. We're fairly overbooked right now. If we find the time, we will try to build a small C demonstration program.

[chris-liddell]

I'm a Ghostscript developer, deekej pointed me at this.

I've done a naive little test program that does:
gsapi_new_instance()
gsapi_init_with_args()
gsapi_run_string()
gsapi_exit()
gsapi_delete_instance()

and loops around doing that (currently sixteen times). I never see the file descriptor in use going above 5

However, there are just so many variables in how this stuff can be used, and I'm not at all familiar with the ImageMagick code/architecture.

So, with the information I have available right now, I cannot reproduce this issue.

[urban-warrior]

Just to confirm, did you try your test under Fedora 28? Also can you post a link to your test program? We can check your API workflow against what we do in ImageMagick.

[chris-liddell]

I use Ubuntu, I'm afraid. I'll try to install Fedora 28 in a VM and try it again.

All I did was hack our existing main function:

https://ghostscript.com/~chrisl/gs.c

[urban-warrior]

The problem only exhibits in Fedora 28. ImageMagick works fine with the Ghostscript API in Fedora 27 and Ubuntu (and Windows, and ...).

[chris-liddell]

Ah, sorry, I misunderstood - I thought you were switching from calling out to a sub-process for gs, to using the gsapi directly. I've got Fedora 28 installing - hopefully I'll have some time to try it again today.

[chris-liddell]

So, I did install a Fedora rawhide build, and I was able to reproduce the problem. The leaking file descriptors are actually happening in libpaper, because the default /etc/papersize is empty (except for a comment). deekej and I are hoping to get this fixed in libpaper, but a workaround is to add a valid paper size line to /etc/papersize (a4 for Europe etc, letter for US etc), that should allow your tests to run without leaking file descriptors.

Either deekej or I will update you when we know more.

[deekej]

The leaking file descriptors are actually happening in libpaper, because the default /etc/papersize is empty (except for a comment).

This also explains why this is only happening in Fedora 28 and was not happening before. I have enabled the libpaper support for Ghostscrip in F28 when I was doing the cleanup. :) It would never occur to me it could cause this...

[remicollet]

Only FYI, fixed in Fedora Rawhide (and libpaper maintainer pinged for F28)

[deekej]

Only FYI, fixed in Fedora Rawhide (and libpaper maintainer pinged for F28).

It's already fixed in F28 as well AFAIK. There were some issues in mock, which caused the delay. :)