PyFerret graphics output (PDF, PS, PNG)

[karlmsmith]

Reported by @AndrewWittenberg on 8 Apr 2014 22:36 UTC
This report crystallizes some graphics issues previously described in #2086 and #2007. This is the main thing preventing our transition to PyFerret at GFDL, and so is at the very top of our priority list.

I'll attempt to attach a gzipped tar file of the full demo, and if that doesn't work, I'll email it to you separately.

In your working directory, save the attached scripts as:

pyferret/mytest.csh
pyferret/mytest.jnl
ferret/mytest.csh
ferret/mytest.jnl
ferret/util/ps_thicken
ferret/util/ps_no_bleedthrough
ferret/util/fps2eps_p
ferret/util/fps2eps_l
ferret/util/fps2eps_l_upright

Now "cd" into the ferret subdirectory and run the "mytest.csh"; then repeat for the pyferret subdirectory. This should produce a tree of figures:

.
|-- ferret
|   |-- display
|   |   |-- asp_0.3
|   |   |   |-- plot.pdf
|   |   |   |-- plot.png
|   |   |   |-- plot.ps
|   |   |   |-- plot_upright.png
|   |   |   `-- plot_upright.ps
|   |   |-- asp_0.77
|   |   |   |-- plot.pdf
|   |   |   |-- plot.png
|   |   |   |-- plot.ps
|   |   |   |-- plot_upright.png
|   |   |   `-- plot_upright.ps
|   |   `-- asp_3
|   |       |-- plot.pdf
|   |       |-- plot.png
|   |       `-- plot.ps
|   |-- mytest.csh
|   |-- mytest.jnl
|   |-- nodisplay
|   |   |-- asp_0.3
|   |   |   |-- plot.pdf
|   |   |   |-- plot.png
|   |   |   |-- plot.ps
|   |   |   |-- plot_upright.png
|   |   |   `-- plot_upright.ps
|   |   |-- asp_0.77
|   |   |   |-- plot.pdf
|   |   |   |-- plot.png
|   |   |   |-- plot.ps
|   |   |   |-- plot_upright.png
|   |   |   `-- plot_upright.ps
|   |   `-- asp_3
|   |       |-- plot.pdf
|   |       |-- plot.png
|   |       `-- plot.ps
|   `-- util
|       |-- fps2eps_l
|       |-- fps2eps_l_upright
|       |-- fps2eps_p
|       |-- ps_no_bleedthrough
|       `-- ps_thicken
`-- pyferret
    |-- display
    |   |-- asp_0.3
    |   |   |-- plot.pdf
    |   |   |-- plot.png
    |   |   `-- plot.ps
    |   |-- asp_0.77
    |   |   |-- plot.pdf
    |   |   |-- plot.png
    |   |   `-- plot.ps
    |   `-- asp_3
    |       |-- plot.pdf
    |       |-- plot.png
    |       `-- plot.ps
    |-- mytest.csh
    |-- mytest.jnl
    `-- nodisplay
        |-- asp_0.3
        |   |-- plot.pdf
        |   |-- plot.png
        |   `-- plot.ps
        |-- asp_0.77
        |   |-- plot.pdf
        |   |-- plot.png
        |   `-- plot.ps
        `-- asp_3
            |-- plot.pdf
            |-- plot.png
            `-- plot.ps

For backward compatibility, we'd like to be able to reproduce all the graphics in the ferret/ directory -- as closely as possible in dimensions, alignment, orientation.

First off, you'll note that the graphics generation syntax & pipeline are VASTLY simpler in PyFerret -- yay!

But there are a number of critical remaining issues:

1. MOST IMPORTANT by far -- the following graphics are not centered, and run off the edge of the page:

pyferret/display/asp_{0.3,3}/plot.{ps,pdf}

2. The "pyferret -nodisplay" output for PDF is "shrink-wrapped" about the figure, which may usually (but not always) be preferable to expanding the margins to a letter-sized page.

3. In cases where the PyFerret graphics don't run off the page, there are still differences of plot sizes and margins, and colorbar widths, between vanilla Ferret and PyFerret. Can these differences be minimized?

4. In vanilla Ferret, graphics produced with or without a display were bitwise-identical, or very nearly so. Not the case yet in PyFerret. While these display/nodisplay differences are fairly minor in PyFerret (i.e. less than 15%), it'd be good to minimize them if it's not too difficult. In addition, the PNGs produced by "pyferret" with a display are somehow clearer than those produced by "pyferret -nodisplay", which are a bit fuzzy. Why?

5. The vector (PS, PDF) files produced by "pyferret" are twice the size of those produced by either "pyferret -nodisplay" or vanilla Ferret. Why?

To address (1) & (2), it'd help to have a way to rotate, resize-to-fit, and center a graphic (with dimensions xinch,yinch) on a given output page (with dimensions pxinch,pyinch):

FRAME/PXINCH=8.5/PYINCH=11/ROTATE=90/FIT/XALIGN=0/YALIGN=0 output.pdf

where

   PXINCH,PYINCH: dimensions of output page in inches
      (defaults are XINCH,YINCH)

   ROTATE: counter-clockwise rotation angle of graphic on page
      (valid values are -90, 0, 90, 180, 270; default is 0)

   FIT: rescale the graphic so it just fits the output page

   XALIGN, YALIGN:
      -1 = align graphic with left/bottom edge of page
      0 = center graphic on page (this is the default)
      1 = align graphic with right/top edge of page

The XALIGN,YALIGN values would be mirror those used to align text labels in Ferret.

Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/2167

[karlmsmith]

Comment by @karlmsmith on 3 Jul 2014 04:07 UTC
The Qt viewer updated so that if it can "shrink-wrap" the page to the image, it will. I have verified it is working on flat (RHL6-64 at PMEL using Python-2.6) as well as on atw and an analysis system at GFDL (RHEL6-64 using Python-2.7.3). For RHEL-5 systems it should revert to the old behavior, but have not verified that yet.

So displayed and -nodisplay for PDFs are nearly identical in graphics display.

For (3), the sizes had been tweaked to make the default page size 10.5" x 8.5" to fit on a letter-size page. I could reset these so the defaults match that of traditional Ferret (10.8" x 8.8" I think) if that is desirable. Since the user can specify the final size and pyferret will scale the image, seems like a reasonable think to do for consistency.

For (4) and (5), the files from displayed images are produced by the Qt graphics software, whereas the files from -nodisplay images are produced by Cairo graphics software. So there is little hope in the underlying contents of images files being nearly the same; there is even differences between versions of Cairo (RHEL5 uses Cairo v1.2 and RHEL6 uses Cario v1.8, the latter does better compression than the former).

The fuzziness in -nodisplay PNG plots had been noticed before and some effort was made to resolve the problem (which resulted in the change of behavior of set win /size). The reason is Qt does a better job with anti-aliasing than Cairo. If one makes a larger plot with proportionally thicker lines (e.g., set win /xinch=20 /thick=2) and then save as the regular size image (e.g., frame /xinch=10) then sharper image are obtained with -nodisplay. But sometimes this can makes lines too "sharp", so not sure whether to try to do any of this automatically.

It does appear the PNG line thickness appears to be thicker than that of the PDFs. Appears to be more than just from antialiasing differences. So may need some tweaking of the default line thickness, at least with PNG plots, may be needed.

Given that the page is now always "shrink-wrapped" to the image, I do wonder how important is supporting positioning, orientation, and additional margins; e.g., Office programs do a much better job for positioning an image within a larger document.

[karlmsmith]

Comment by @AndrewWittenberg on 4 Jul 2014 00:29 UTC
Thanks Karl -- your fixes resolve items (1) & (2).

With (4), the display & nodisplay graphics look very similar in size now within PyFerret, at least for PS and PDF graphics. But for PNG for some reason, the default sizes are different (with nodisplay a bit smaller) -- could this difference be eliminated?

It almost looks like the Cairo (nodisplay) graphics draw an outline around each grid cell of a SHADE, so that the cells expand slightly (relative to Qt/display) into data-void regions like land. In data-rich regions (like the middle of the Pacific), these outlines overlap each other, and the order in which they're drawn results in us seeing only the bottom-left part of each cell outline so that each cell appears to shift downward & to the left as we go from a Qt PDF to a Cairo PDF. (E.g. switching back & forth in acroread, using Ctrl-Tab.)

It seems that the Cairo (nodisplay) graphics are better in at least one sense, namely at the interfaces between grid cells of a SHADE. With Qt (display) I see the familiar blending-toward-background problem (which in the case of a white background, lightens up the interfaces between cells, making a "grid" appear in the plot). This is familiar from vanilla Ferret. The Cairo (nodisplay) graphics don't have that problem at all. Is this related to the antialiasing differences that you mentioned?

Regarding (5), it'd still be very helpful to have the page-layout syntax that I described. You're right that Office would be great for manipulating a single figure; but we need to think about our automated figure generation, which creates thousands of figures. We want to be able to print any one of these PS or PDF figures at will, on letter paper, and have things looking nice, without the graphic sitting way up in one corner of the page. If you can think of a postprocessing tool (psutils?) outside of Ferret that would do all this in a scriptable way, that'd be fine too.

[karlmsmith]

Comment by @karlmsmith on 14 Jul 2014 20:29 UTC
When drawing filled rectangles and polygons, the border is drawn as well using a "cosmetic" pen, as Qt calls it, which is suppose to be the thinnest line possible for the format (one pixel for PNG; not sure what for PDF). This is done in an attempt to solve the problem you mention - the white lines in SHADE and FILL plots (which arise because of plotting continuous-value coordinates on a discrete-value surface).

For Qt/display, since the cosmetic pen is a Qt term, Qt understands this cosmetic line concept and handles it for me. But this also means I am not sure what width is used for PDFs.

For Cairo/nodisplay, there is no such cosmetic pen concept, so I draw it one pixel wide. With my pretend resolution of 96 DPI, this ends up being 72/96 (0.75) of a point wide for PDFs. But this is definitely thicker than what Qt uses for PDF.

I can switch to not using the cosmetic pen in Qt and instead specify the width directly. But it sounds like I need to make the width thinner that it is with Cairo to minimize the shift due to order-of-drawing and still cover the gaps between rectangles/polygons.

For external tools to manipulate PDF and/or PS....

pdftops can be used to center a PDF on a page of any size desired (and convert back to PDF with ps2pdf)
I created an example 6"x4" test image as test.pdf, then centered it on an 8.5"x5.5" page (612 pt x 396 pt) using the command

pdftops -paperw 612 -paperh 396 test.pdf - | ps2pdf - test_halfpage.pdf

pstops can be inserted for finer manipulation

pdftops -paper letter -nocenter test.pdf - | pstops '0(1.25in,5.75in)' | ps2pdf - test_fullpage.pdf

The pdftops command makes a PS on my desired final page size (letter sized) but with -nocenter leaves it at the bottom left corner - PS coordinates (0,0). The pstops command then moves it 1.25" to the right and 5.75" up to give the 6"x4" image on a letter-sized page with a 1.25" left, right, and top margins.

The pstops command can also be used to rotate and scale the image. If I get multi-page output into PyFerret, then multiple images could also be combined onto one page.
See http://www.novell.com/documentation/suse91/suselinux-adminguide/html/ch06s08.html for helpful information on reformatting with pstops (the diagrams help a lot to understand).

[karlmsmith]

Comment by @AndrewWittenberg on 2 Aug 2014 00:22 UTC
I've written the following script, "frame_to_page.jnl" (see below) to implement the features that I had requested above, with a few alterations. This is a wrapper around the FRAME command, to be used with PostScript and PDF output.

By default the script will center and auto-rotate the figure to best fit an 8.5" wide x 11" tall page, setting the BoundingBox and PageOrientation so that it will display properly (upright and cropped) in GhostView. That way the PostScript file can be printed directly, without it sitting way up in one corner of the printed page.

Additional arguments are provided to (a) expand/shrink the figure to the full page (minus a specified minimum margin); (b) specify the rotation explicitly; (c) specify the page dimensions explicitly; and (d) specify the exact size of the figure on the page.

This script enables PyFerret to reproduce the output of vanilla Ferret very closely, at least in "display" mode (Qt4 graphics). So if it's not too much work, I'd suggest incorporating all of these features into the syntax of the FRAME command.

There are still a number of problems with the "nodisplay" (Cairo) graphics, as I'll point out in the next comment.

Comments? Suggestions?

\can mode verify
! Usage: go frame_to_page file [page_minmarg fig_angle page_wid page_ht fig_wid fig_ht]
!                          $1  [    $2          $3        $4      $5       $6     $7  ]
! Output a figure to a PostScript or PDF page.
!
! Examples:
! yes? def sym c1 go magnify\; plot/vs/line/nolab/hl=0:1/vl=0:1 {0,1,,0,1},{0,1,,1,0}\; label .5 .5 0 0 .6 TEST
! yes? set win/xin=6/yin=3 1; ($c1)   ! short wide window
! yes? go frame_to_page "a.ps"        ! auto-rotate
! yes? go frame_to_page "a.ps" .25    ! auto-rotate and scale, with minimum 0.25" margin
! yes? go frame_to_page "a.ps" 0      ! no margin (expand/shrink to page edges)
! yes? go frame_to_page "a.ps" " " 0  ! no rotation
! yes? go frame_to_page "a.ps" " " " " 11 8.5      ! landscape page
! yes? go frame_to_page "a.ps" " " 0 8.5 11 " " 11 ! expand graphic to fit page height
!
! yes? set win/xin=3/yin=6 1; ($c1)   ! tall narrow window
! yes? go frame_to_page "a.ps"        ! auto-rotate
! yes? go frame_to_page "a.ps" .25    ! auto-rotate and scale, with minimum 0.25" margin
! yes? go frame_to_page "a.ps" 0      ! no margin
! yes? go frame_to_page "a.ps" " " 90 ! left rotation
! yes? go frame_to_page "a.ps" " " " " 11 8.5      ! landscape page
! yes? go frame_to_page "a.ps" " " 0 8.5 11 8.5 ! expand to graphic to page width
!
! yes? set win/xin=8/yin=10.5 1; ($c1)   ! optimized portrait window 
! yes? go frame_to_page "a.ps"
! yes? set win/xin=10.5/yin=8 1; ($c1)   ! optimized landscape window 
! yes? go frame_to_page "a.ps"
!
! atw 2014aug1

def sym fp_file "($1)"
let fp_out_type = spawn("echo ($fp_file) | sed 's/.*\.\([a-z]*\)/\1/'")
IF `fp_out_type NE "pdf" AND fp_out_type NE "ps"` THEN
   say "ERROR in frame_to_page.jnl: file type '`fp_out_type`' is not one of pdf, ps."
   exit/prompt
ENDIF

let fp_minmarg = $2"-1" ! Minimum margin (inches) between rescaled graphic & page edges.
                        ! If negative, then don't rescale the graphic.
let fp_angle = $3"-1|-1|0|90|180|270"  ! counter-clockwise angle (degrees) of graphic on page
                         ! If -1, rotate to best match the aspect ratio of output page.

! dimensions (inches) of output page
let fp_xo = $4"8.5"
let fp_yo = $5"11"

! dimensions (inches) of input graphic, before rotation & rescaling
IF `$6"0|*>1" AND $7"0|*>1"` THEN
   say "ERROR in frame_to_page.jnl: only one of FIG_WID and FIG_HT can be given."
   exit/prompt
ELIF `$6"0|*>1"` THEN
   let fp_xi = $6
   let fp_yi = fp_xi * ($ppl$height)/($ppl$width)
ELIF `$7"0|*>1"` THEN
   let fp_yi = $7
   let fp_xi = fp_yi * ($ppl$width)/($ppl$height)
ELSE
   let fp_xi = ($ppl$width)
   let fp_yi = ($ppl$height)
ENDIF

IF `fp_angle EQ -1` THEN
   ! auto-determine orientation of graphic
   IF `(fp_yi-fp_xi)*(fp_yo-fp_xo) GE 0` THEN
      ! graphic aspect ratio matches output page: don't rotate
      let fp_angle = 0
   ELSE
      ! rotate graphic to better fit output page
      let fp_angle = 90
   ENDIF
ENDIF

IF `fp_angle EQ 0 OR fp_angle EQ 180` THEN
   def sym fp_orient "Portrait"
   let fp_xr = fp_xi
   let fp_yr = fp_yi
ELSE
   def sym fp_orient "Landscape"
   let fp_xr = fp_yi
   let fp_yr = fp_xi
ENDIF

IF `fp_minmarg LT 0` THEN
   let fp_scale = 1
ELSE
   ! fit graphic to output page, without changing aspect ratio
   let fp_scale = min((fp_xo-2*fp_minmarg)/fp_xr, (fp_yo-2*fp_minmarg)/fp_yr)
   IF `fp_scale LE 0` THEN
      say "ERROR in frame_to_page.jnl: minimum page margin (`fp_minmarg`in) is too large for the `fp_xo`in x `fp_yo`in page."
      exit/prompt
   ENDIF
ENDIF

let fp_xmarg = (fp_xo - fp_scale*fp_xr)/2
let fp_ymarg = (fp_yo - fp_scale*fp_yr)/2

! dimensions and shifts for various rotations
IF `fp_angle EQ 0` THEN
   let fp_rcom = ""
   let fp_xshift = fp_xmarg
   let fp_yshift = fp_ymarg
ELIF `fp_angle EQ 180` THEN
   let fp_rcom = "U"
   let fp_xshift = fp_xmarg + fp_scale*fp_xr
   let fp_yshift = fp_ymarg + fp_scale*fp_yr
ELIF `fp_angle EQ 90` THEN
   let fp_rcom = "L"
   let fp_xshift = fp_xmarg + fp_scale*fp_xr
   let fp_yshift = fp_ymarg
ELSE
   let fp_rcom = "R"
   let fp_xshift = fp_xmarg
   let fp_yshift = fp_ymarg + fp_scale*fp_yr
ENDIF

def sym fp_tmp "($fp_file).($SESSION_PID)"

frame/file="($fp_tmp).pdf"/xin=`fp_xi`

! wait for Ferret to flush its frame buffer
sp sleep 1

sp pdftops -paperw `72*fp_xi,p=0` -paperh `72*fp_yi,p=0` -nocenter -noshrink -nocrop ($fp_tmp).pdf - \
   | pstops -q -w`72*max(fp_xr,fp_yr),p=0` -h`72*max(fp_xr,fp_yr),p=0` '0`fp_rcom`@`fp_scale,p=6`(`72*fp_xshift,p=0`,`72*fp_yshift,p=0`)' \
   | sed -e 's/\(^%%.*BoundingBox:\).*/\1 `72*fp_xmarg,p=0` `72*fp_ymarg,p=0` `72*(fp_xo-fp_xmarg),p=0` `72*(fp_yo-fp_ymarg),p=0`/' -e 's/\(^%%.*PageOrientation:\).*/\1 ($fp_orient)/' -e '/%%DocumentMedia/d' -e '/%%PageMedia/d' \
   > ($fp_tmp).ps

IF `fp_out_type EQ "pdf"` THEN
   sp ps2pdf -dDEVICEWIDTHPOINTS=`72*fp_xo,p=0` -dDEVICEHEIGHTPOINTS=`72*fp_yo,p=0` ($fp_tmp).ps ($fp_file)
   sp rm ($fp_tmp).ps
ELIF `fp_out_type EQ "ps"` THEN
   sp mv ($fp_tmp).ps ($fp_file)
ENDIF

sp rm ($fp_tmp).pdf

can var fp_xo fp_yo fp_angle fp_minmarg
can var fp_xi fp_yi fp_xr fp_yr fp_scale
can var fp_rcom fp_xshift fp_yshift fp_out_type
can var fp_xmarg fp_ymarg
can sym fp_file fp_tmp fp_orient

set mode/last verify

[karlmsmith]

Comment by @AndrewWittenberg on 2 Aug 2014 01:01 UTC
I'll attach a new graphics.tgz with the new scripts, including the above "frame_to_page.jnl" for PyFerret.

In the new output tree, there's a now a much closer match between the Ferret and PyFerret graphics, mostly resolving issues (1)-(3) above. But issues (4) and (5) remain. One remaining graphics difference between versions (comparing ferret/display/ against pyferret/display/) is that they appear to have slightly different aspect ratios for the plot region, perhaps due to different margins around the graphics. Can we eliminate those differences?

In vanilla Ferret the nodisplay graphics were bitwise identical to the displayed versions. But in PyFerret, the nodisplay (Cairo) graphics are much different (and seemingly worse) than the displayed (Qt4) versions.

The Cairo graphics produce vastly bigger PostScript files (by up to a factor of 30!) than vanilla Ferret, such that even a simple plot like the one in my example becomes a 10Mb file. Very slow to produce, and slow to open and view. Looking inside the PostScript files, the Cairo output has loads and loads of repeated phrases -- not sure what these are.

For some reason, the Cairo graphics also get rasterized in the PostScript output, and have darker lines. And for an unknown reason, the pyferret/*/asp_0.77/plot.ps gets rotated on its side compared to the Qt4 result (really puzzling, since the asp_0.3 case looks ok).

The PNG output also differs between Cairo and Qt4; the Cairo output has smaller pixel dimensions. How can that be, when XPIXEL/YPIXEL are set the same in the two cases?

[karlmsmith]

Comment by @karlmsmith on 25 Sep 2014 22:37 UTC
Modified so the "cosmetic" line used to outline polygons is 1 pixel wide (in the unscaled plot) regardless of whether Qt or Cairo. Thus changed so Qt matches what I am doing in Cairo which has no concept of a cosmetic line.

Removed setting a fallback resolution in vector graphics files so that (hopefully) it will never put raster images into vector files. This might have just been a "just in case" image that was never really used and should not have been in there. But takes up space and we don't need it.

Found a bug (hopefully fixed now) in no-display saving of PostScript where the image scaling was not being performed when a landscape image was rotated for saving.

Will build and test on RHEL6 machines at PMEL and GFDL.

[karlmsmith]

Comment by @AndrewWittenberg on 1 Oct 2014 23:13 UTC
Items (1)-(3) of my original report have now been resolved in the 9/29/14 version of PyFerret (based on Ferret v6.925), with Karl's latest fixes and my frame_to_page.jnl.

Closing this ticket, and moving remaining Cairo (nodisplay) graphics issues to ticket #2202.

[karlmsmith]

Attachment from @AndrewWittenberg on 8 Apr 2014 22:38 UTC
Gzipped tar file, containing the demo scripts.
graphics.tar.gz

[karlmsmith]

Attachment from @AndrewWittenberg on 2 Aug 2014 01:02 UTC
Version 2 of the graphics tests.
graphics.2.tar.gz