Skip to content
This repository

Transparent PNGs memory intensive #324

Closed
luckydev opened this Issue · 63 comments

8 participants

Anand Brad Ediger Alex Dowad Gregory Brown Alexander Mankuta Mark D. Blackwell James Healy Benjamin Curtis
Anand

We use prawn for all PDF generation in our raise app.

we have delayed_job background workers which picks up certain records from database and generate a PDF document of them. In that document, there can be user uploaded PNGs.

When generating the PDF with transparent PNGs, the memory consumed by delayed_job worker quickly grows from 60MB to 500MB.. where as in the case of normal PNGs with some backgrounds, the memory consumed is very low.

Im using Prawn 0.8.4, Ruby 1.9.2 and Rails 3.0.5 for my app. Is this problem fixed in recent versions?

Brad Ediger
Collaborator

Transparent PNGs have to have their alpha channel split in order to be embedded in PDFs. That's a memory-intensive process in Ruby. The workaround, if you're willing to depend on ImageMagick and RMagick, is to install prawn-fast-png:

https://github.com/amberbit/prawn-fast-png

Alex Dowad

We encountered the exact same issue and were baffled by why simply adding a PNG to a PDF could use hundreds of megabytes of memory. Using ObjectSpace.each_object revealed that Prawn was allocating more than 500,000 Arrays and 80,000 Strings for that operation. This should be mentioned in the documentation for Prawn::Document#image!!!

Brad Ediger
Collaborator

@alexdowad: It is mentioned in the documentation and has been for over two years:

https://github.com/prawnpdf/prawn/blob/master/lib/prawn/images.rb#L17

As much as I would like not to have this problem, it's intrinsic to the PDF format, which does not natively support transparency in images. Prawn has to convert the PNG file into an image blob and an SMask for the alpha channel, and the processing has to be done somewhere. Since Prawn by design is a pure-Ruby library, this can be pretty allocation-intensive. I'd definitely accept patches that reduce the number of allocations, but ImageMagick, being written in C, is likely to always be faster and more memory-efficient.

-be

Alex Dowad

@bradediger, thanks for pointing that out. I did see that in the docs, but didn't connect it to the fact that our application was running out of memory. The docs seem to indicate that processing PNGs is CPU-intensive, not memory-intensive. Actually, I thought using prawn-fast-png might make the problem worse, because RMagick can be a memory hog itself.

Brad Ediger
Collaborator

@alexdowad, good point. I've added a note about the memory consumption. Thanks!

Alex Dowad
Anand

If anyone is using paperclip, we can fix this without prawn-fast png. we ask imagemagick directly to save the image with white background.

 has_attached_file :header, :styles => {:std => "450x50>"}, :convert_options => {:all => "-flatten" } 
Gregory Brown
Owner

I am going to re-open this issue just because I'm interested in whether we're doing this in a way that's well-optimized or not. I don't expect we're going to have great performance or memory efficiency with tweaking (especially compared to C libraries), but I wonder if we can do better than what we're doing now, at least.

Maybe it's worth revisiting ChunkyPNG and seeing what it can offer.

Gregory Brown sandal reopened this
Alexander Mankuta
Collaborator

AFAIK, it's as good as it gets without investing substantial effort into it. Mostly because of how PDF implements transparency. In PDF transparency is supported only via masking. So every transparent PNG has to be decomposed into two different images. Essentially, alpha channel has to be extracted into bw-image. ChunkyPNG has to decompress image to do that.

The only thing I can think of is checking if decompressed images are getting GC'd ASAP. Not that that would help to avoid high total peak mem usage but at least it can lover average mem usage over time.

Gregory Brown
Owner

@cheba: Yes, we can't avoid splitting out the alpha channels, as @bradediger already mentioned above.

I'm mostly interested in whether our implementation creates some unnecessary intermediate objects that we might be able to eliminate by reading in larger chunks at a time, or doing some other tricks. We may come up empty, but it's worth investigating. When we wrote the PNG code, we weren't necessarily thinking about what would be the optimal way of doing things.

Mark D. Blackwell

In practical terms (unlike, from some perspectives, the 'ideal' world of programming), 'hugely slow' is equivalent to 'unworkable'.

Prawn (in effect) IMHO simply does not support certain kinds of image files (in that important sense).

The kinds of image files without Prawn support (practically speaking) include at least PNGs with transparency.

By default, Prawn should reject these files—that is, reject them immediately, rather than in the mathematical sense of rejecting them ultimately, by making their processing practically interminable. ;/

This would be a service to Prawn's users.

Because Prawn (I believe) lacks such a configurable option, I needed to write code to prevent passing such alpha transparent PNGs to Prawn. Please let me know if Prawn does have such an option!

To catch them, I just looked at the 'alpha' bit located in PNG's easily findable IHDR color type byte.

However, generally speaking, obviously, Prawn users who use PNGs would find this unattractive. Presumably, they wouldn't wish to write this kind of code!

Arguably, Prawn should have a passable (or configurable) setting, allowing it optionally to process PNGs with transparency.

This setting should be 'reject' by default in order to protect Prawn's reputation for snappiness. Knowledgeable users can set this quite easily to 'accept', after they decide the extreme slowness is warranted in their special cases (perhaps for quite small PNG files).

BTW, relative to this method, ChunkyPNG was extremely slow to load a PNG for this check. And, ChunkyPNG doesn't even provide the color type (I believe).

Gregory Brown
Owner

@MarkDBlackwell

We have the constraint of working in pure Ruby. Simply installing prawn-fast-png mitigates this problem, but involves pulling in a heavyweight dependency.

The use cases for Prawn are vast. Someone may be hand generating a single report on a decently powerful machine, once. In that case, memory and speed are not a concern. Anyone who is doing server-side generation of PDFs however will be badly hurt by this issue, and we've been aware of that for a long time.

I think we can possibly add a warning at least, so that this issue will be detected at development time rather than in production. I might even go as far as to have a variable that's off by default, and spit an error until someone sets it otherwise. That would work something like this:

require "prawn"

Prawn::Image::PNG.allow_transparency = true

# your code here 

Without that value set, you'd get an error like:

Prawn's support for transparency in PNGs is very processor and memory intensive, 
and is disabled by default. To enable it, set Prawn::Image::PNG.allow_transparency = true. 
Alternatively, consider installing the prawn-fast-png extension

I think either of these options (a warning or off-by-default feature) are worth discussing. But I'm also still interested in verifying how much of the slowness and memory usage is due to inherent problems (of which we know there are some), and how much is due to implementation issues.

But in any case, I agree we should do something about this before 1.0

Alexander Mankuta
Collaborator

@sandal What would happen if transparency is disabled but transparent PNG is added to the document?

Would it result in an exception? This would basically break the functionality. Also it still requires at least some checks (meaning, leaking knowledge about PNG into a PDF lib).

Would it simply ignore it and just pass the data into the document? It might be useless in most cases.

PNG has two major advantages: lossless (images not crippled by JPEG compression or GIF reduced palettes) and alpha-channel.

Reducing alpha-channel support out of the box might be a major downside.

It would be hard to reduce CPU/mem usage without really processing PNG like it's done with JPGs. JPGs are basically inscribed inscribed in the PDF format. JPXDecode is the one of the supported stream filters. On the other hand PNGs have to be unpacked. It's impossible to just include PNG data as it's done with JPGs.

It's worth to try and optimize PNG parsing. I wouldn't turn off transparency. It breaks a major feature and still it doesn't eliminate the work required for PNG support at all.

Gregory Brown
Owner

@cheba: It would raise an exception. We have a very easy place to put this, directly where we split alpha channels.

The argument @MarkDBlackwell is making is that support for PNG with transparency in Prawn is effectively broken due to its efficiency issues, even if the feature produces technically correct output. I'm not sure that I'd go to that extreme, but I definitely think of it as being a feature that is problematic.

What we are trying to figure out is the best possible user experience, while keeping our constraints in mind. In my view, being explicit is better than being implicit, and giving feedback early is better than later. A warning could do this, but forcing to set a variable would allow us to be certain the user understood the potential problems, and also alert them to an alternative solution.

The thing we need to remember when working on this library is no one is going to care about why there are all the technical reasons for our inefficiency, they're just going to see memory spikes and slowdown and be frustrated by them. We should fix these issues where we can, and do better with communication where we cannot.

James Healy
Collaborator
Gregory Brown
Owner

I would imagine it wouldn't create any support requests at all if we include an explanation and possibly a link in the error message. But it may prevent someone from building an entire project only to realize later that these performance issues were a show-stopper for them.

Again, the message I would want to show is something like this:

Prawn's support for transparency in PNGs is very processor and memory intensive, 
and is disabled by default. To enable it, set Prawn::Image::PNG.allow_transparency = true. 
Alternatively, consider installing the prawn-fast-png extension

How would this create problems that would require a support request?

Alexander Mankuta
Collaborator

@sandal Prawn's PNG support is very processor and memory intensive. Full stop.

The issue has little to do with the transparency as such. The fact that we have to decode PNGs for any support at all (even non-transparent ones) is why it's not as efficient as JPGs.

Gregory Brown
Owner

@cheba: I would like to see the numbers on the different PNG types, it's my recollection that most of the performance issues we had were with alpha channel splitting, but I could be misremembering. Can you try running some benchmarks?

Alexander Mankuta
Collaborator

@sandal I'll try doing something over the weekend.

Gregory Brown
Owner

@cheba: Thanks! It'll be great for us to take a more quantitative approach to problems like these, so we can make better informed decisions.

James Healy
Collaborator
Mark D. Blackwell

@sandal @cheba @yob It would be wonderful to have such a configurable option as sandal's suggested: Prawn::Image::PNG.allow_transparency (even if it defaults to true) with sandal's suggested error message, but only if it raises an exception that stops the program when it is set to `false'.

In my own view, it is fine to default it to true to avoid extra support requests on the mailing list. :)

Merely emitting a warning about this issue (when set to false) would not help us Prawn users who write programs to deliver to end users. We Prawn users ourselves are well aware—or soon become aware—of Prawn's tendency to hang on transparent PNGs.

Handling transparency in PNGs isn't such a big need, I would think. After all, as I understand it, the PDF format itself doesn't even include the concept of transparency. Is it true that we can draw anything atop any opaque PNGs we drew (or placed) earlier in a PDF? (Please correct me if I'm wrong.)

I believe the big problem here is that end users may give us a transparent PNG someday (or soon), intended for placement in a PDF, without any transparency truly, actually being desired. And how do we handle that? (It has already happened to me, out of their graphics shop.)

As you know, end users don't much look at the stream of warning messages. But they would look, if an error stopped the program they were running. And, users of Prawn could set that configurable option easily to false: so much more easily than writing code, themselves, to look inside any 'arriving' PNGs, programmatically, for their color type, before passing them on to Prawn!

With an exception raised, end users would see the exception message immediately and get the point not to use transparent PNGs, rather than just believing that the Prawn user's (that is, my) program is bothersomely slow and hangs too much, and perhaps, then, drawing a mistaken, general conclusion that Prawn, itself, is 'just a bad gem'. ;/

BTW, aren't we Prawn users who write programs—for delivery to end users—a major portion of all Prawn users? Presumably, Prawn intends to help us with our needs, too.

I tend to think that defaulting a configurable option: allow_transparency to false on balance would evangelize Prawn much better. But, whichever way it defaults doesn't matter to me, for my own purposes. :)

Gregory Brown
Owner

@yob: If I were to introduce an exception, I could probably wait until the time of the 1.0 release. Also, I'm not against using a warning rather than exception in theory, except in practice it's very likely the user won't see that and it'll just be ignored.

My main priority for 1.0 is just to have clear messaging about what is stable and reliable, and what is problematic. I'm open to any possible way of communicating that, as long as its effective.

What's more... if I make a change people hate, I won't stubbornly stick to it. But I rather try something out than wonder what the effect will be in theory.

Gregory Brown
Owner

@MarkDBlackwell: I totally understand the concern from your perspective, and I feel like there are plenty of users who will end up in similar situations. But I also know that we have a hugely diverse userbase, and so I can't really make claims about what the "majority" use case will be.

Another option we have here is exception raising behavior that is OFF by default. This could be documented in the manual, would address the use case you mentioned (presenting an error to end users rather than hanging for a long time), and would not introduce a behavior change to Prawn's default behavior.

Let's leave all options on the table, start with optimization first, and see where we're at by the next release (not 0.13.0 which will ship Sunday, but 0.14.0 a month from now).

James Healy
Collaborator
Mark D. Blackwell

@sandal I presume you have good reasons for wanting to wait until the 1.0 release to introduce an exception. :)

However, it seems arguably important in pre-1.0 time, when things (API, etc.) can be more fluid, to make changes to solve the known, various problems such as this one, for Prawn users who supply programs to end users.

Especially in this case, whose solution involves only a non-default configuration, which won't impact the usual mass of users at all!

Gregory Brown
Owner

@MarkDBlackwell, I'd be OK with a patch for an exception that is raised only when the flag is set, and does nothing by default. I can't promise it will stick around until 1.0, but I'd encourage you to try it out, see if it works for your use case, and if so, submit a pull request.

Mark D. Blackwell

@sandal @yob Agreed it should be OFF by default. :)

I already have my own code written to check the PNG's myself. I raise this recommendation to benefit (I think) Prawn, and the other Prawn users who are like me, in the future. :)

In other words, I don't foresee my submitting a pull request for this, not that I don't think it's desirable.

Gregory Brown
Owner

@MarkDBlackwell It's a good suggestion. It definitely will get looked at faster with a patch though, so feel free to send one in if you get a chance.

In any case, thanks for sharing your experiences with us.

Alexander Mankuta
Collaborator

@MarkDBlackwell

Handling transparency in PNGs isn't such a big need, I would think. After all, as I understand it, the PDF format itself doesn't even include the concept of transparency. Is it true that we can draw anything atop any opaque PNGs we drew (or placed) earlier in a PDF? (Please correct me if I'm wrong.)

Technically that is not entirely correct. PDF has concept of blending. You can use soft masks (basically alpha-channels) to produce semi-transparent output. All the drawing operations would be preformed in the memory before actually being rendered (to a screen or a printer, or anything else actually). That includes blending too. These masks are the PDF's transparency concept. If you want to go into greater details you can refer to the section 7. Transparency of the PDF Reference v1.7.

Alexander Mankuta
Collaborator

I believe the big problem here is that end users may give us a transparent PNG someday (or soon), intended for placement in a PDF, without any transparency truly, actually being desired. And how do we handle that? (It has already happened to me, out of their graphics shop.)

This is hardly ever true. It's really hard to predict how it would look like if you just strip alpha-channel. Here's an example:

key

The black rectangle on the right is the same key without the transparency. Many graphic editors do really weird things to the transparent pixels. So it might look like this, or have ugly grey background, or might be full of Satan hailing where original image was transparent.

I can believe that users intend the image have the image blended over some sane background (like white, for instance in case of printing on a white paper) but I don't believe that users really want just transparency removed. Note, that technically these two are not the same and blending in Prawn would be even slower than just extracting alpha-channel as it's done now.

Mark D. Blackwell

@cheba Continually, I am greatly impressed by your knowledge!

So, I take it I can indeed draw something opaque over an earlier-drawn opaque object in a PDF. Perhaps it's obvious.

I take your point that transparent images can contain some hidden things which shouldn't be brought into the light of total opaqueness! ;/

Well, in this particular case, using Photoshop, I flattened the image which I was given and removed its transparency. I don't know if that's the same as merely stripping the alpha channel. Then I verified with my own eyes that it looked the same as the original PNG. Perhaps people (not Prawn) should flatten their PNG images for Prawn's use, and not merely strip their alpha channel—I don't know.

This particular image was entirely at the 100% level (of opaqueness) except for its surrounding area, which was 100% transparent (or clear), somewhat in the manner of a PNG for a visual object on a web page. I verified this by loading it in a browser and changing the browser's background color.

The image also included a central, opaque white background. This is perhaps not so good for a web page—unless the intended web page's background happens to be white. So, here, this client's graphics shop didn't do such good work, I would think. Or, which seems more likely, they don't understand PDFs.

Anyway, placed on a white background on a web page, this kind of transparent PNG looks just like its non-transparent, transformed version.

In cases like these, the alpha channel is totally unneeded. Nevertheless, it included the bothersome alpha channel.

I don't really know whether (coming out of graphics shops) this kind of PNG is common or uncommon. Based on my sample size (of one) and for other reasons, I suspect it might be quite common.

I would inform you, if you didn't know, that end users often don't know much about what they are doing. :) That's a key point, I think. That's why they easily could ask that a PNG with an almost totally useless alpha channel be used for making a PDF.

I suppose transparency isn't usually deeply desired by the end user for PDFs, because usually they can convert their transparent PNG images to a workable, non-transparent form.

In other words, after human beings hand-convert the usual web PNG's transparent areas to white, using their visual judgment, everything should be fine whenever their desired PDF is printed on white paper. Don't you think so?

I suspect that in practice people present Prawn with an actual business need for PDF soft masks—for a visible element involving PNG transparency—only in rare cases. :)

On the other hand, if a client were to tell me they want to include soft masks in their PDF output, then I would simply change my Prawn configuration settings to allow PNG alpha transparency—but I can make that change only in those cases, and protect the good Prawn processing speed for my other clients.

Gregory Brown
Owner

@MarkDBlackwell @cheba: Feel free to continue discussing this, but perhaps head to the Prawn mailing list or take the conversation to private email. I want to keep the thread for this ticket focused on the optimization of the PNG stuff.

Alexander Mankuta
Collaborator

@MarkDBlackwell > In other words, after human beings hand-convert the usual web PNG's transparent areas to white, using their visual judgment, everything should be fine whenever their desired PDF is printed on white paper. Don't you think so?

This is what I was talking about. It's blending transparent PNG over white background, not stripping down alpha-channel.

I just reiterate that PDF doesn't have native PNG support. That means, that Prawn has to decode it and convert to something PDF renderer can understand. That is true for all PNGs, not only the transparent ones.

Alexander Mankuta
Collaborator

@luckydev @alexdowad Could you please provide the images you've observed abnormal CPU/mem usage with?

Anand
Alexander Mankuta
Collaborator

Alpha decomposition seem to be extremely expensive.

Here's a profile of opaque PNG image:

Thread 1: total running time: 0.042860147s

  %   cumulative   self                self     total
 time   seconds   seconds      calls  ms/call  ms/call  name
------------------------------------------------------------
  10.50    0.02      0.01         81     0.07     0.25  Prawn::Core.PdfObject
   7.57    0.00      0.00          1     4.31     4.31  GC.collect_young
   5.63    0.01      0.00          2     1.60     3.93  Rubinius::Sprinter::Builder#parse
   5.49    0.00      0.00        288     0.01     0.01  Array#include?
   5.43    0.01      0.00         47     0.07     0.14  Array#map
   4.98    0.01      0.00          1     2.84     5.24  Prawn::Images#build_image_object
   4.78    0.00      0.00          1     2.72     2.72  Prawn::Document#start_new_page
   4.63    0.00      0.00          4     0.66     0.66  Rubinius::Sprinter::Builder::FloatAtom#bytecode
   4.30    0.00      0.00        288     0.01     0.02  Prawn::Core::PdfObject<93> {}
   3.88    0.00      0.00         19     0.12     0.12  String#length
   3.78    0.00      0.00         47     0.05     0.05  Array::join<849> {}
   3.44    0.01      0.00          3     0.65     2.09  Encoding::Converter::TranscodingPath.get_converters
   2.67    0.01      0.00          6     0.25     1.13  Encoding::Converter::TranscodingPath.[]
   2.64    0.02      0.00         32     0.05     0.62  Prawn::Core::PdfObject<102> {}
   2.50    0.00      0.00         12     0.12     0.12  Rubinius::Tuple#each
   2.48    0.03      0.00         25     0.06     1.02  Array#each
   2.45    0.00      0.00         50     0.03     0.05  Thread.detect_recursion
   2.09    0.00      0.00         47     0.03     0.06  Array#join
   2.05    0.02      0.00         10     0.12     1.99  Hash#each
   2.02    0.01      0.00          2     0.58     4.64  Module#dynamic_method
   1.99    0.00      0.00          1     1.14     1.14  Prawn::Document::Internals#render_trailer
   1.61    0.02      0.00          1     0.92    17.64  Prawn::Document#initialize
   1.45    0.02      0.00          1     0.83    24.58  Prawn::Document#render
   1.11    0.00      0.00         15     0.04     0.19  StringIO#read
   1.06    0.01      0.00          1     0.60     8.61  Prawn::Images#embed_image
   1.05    0.01      0.00          2     0.30     3.95  String#encode!
   0.87    0.02      0.00          6     0.08     3.27  Prawn::Core::Reference#object
   0.80    0.00      0.00         14     0.03     0.22  Rubinius::Sprinter::Builder::parse<1190> {}
   0.78    0.01      0.00          7     0.06     1.44  String#%
   0.69    0.02      0.00          6     0.07     3.34  Prawn::Core::DocumentState::render_body<70> {}
   0.58    0.01      0.00          2     0.16     3.65  Encoding::Converter#initialize
   0.58    0.01      0.00          7     0.05     1.38  Rubinius::Sprinter.get
   0.52    0.00      0.00         12     0.02     0.15  MatchData#to_a
   0.43    0.01      0.00          2     0.12     4.05  Rubinius::Sprinter::Builder#build
   0.42    0.00      0.00         12     0.02     0.13  MatchData#captures
   0.40    0.01      0.00          2     0.11     4.07  Prawn::Core.utf8_to_utf16
   0.29    0.00      0.00         12     0.01     0.15  Rubinius::Type.coerce_to_array
   0.29    0.02      0.00         10     0.02     1.78  Class#new
   0.27    0.01      0.00          4     0.04     1.44  Encoding::Converter::initialize<154> {}
   0.24    0.00      0.00          6     0.02     0.37  StringIO#printf
   0.24    0.02      0.00         12     0.01     1.87  Prawn::Core::ObjectStore::each<73> {}
   0.15    0.00      0.00         10     0.01     0.12  Prawn::Core::PdfObject<79> {}
   0.15    0.00      0.00          6     0.01     0.39  Prawn::Document::Internals::render_xref<150> {}
   0.12    0.01      0.00          1     0.07    13.92  Prawn::Images#image
   0.12    0.00      0.00          1     0.07     2.50  Prawn::Document::Internals#render_xref

10 methods omitted

55 methods called a total of 1,133 times

And here's one with transparent PNG:

Thread 1: total running time: 72.057737892s

  %   cumulative   self                self     total
 time   seconds   seconds      calls  ms/call  ms/call  name
------------------------------------------------------------
  24.90   20.82     17.94   10678380     0.00     0.00  Array#[]
  22.10   35.36     15.93    1222400     0.01     0.03  Prawn::Images::PNG::unfilter_image_data<306> {}
  11.74   64.50      8.46       1730     4.89    37.28  Array#each
   3.95    7.48      2.85          2  1424.13  3741.28  String#each_byte
   3.42    3.18      2.47    6242171     0.00     0.00  Array#<<
   3.00    3.81      2.16    1920000     0.00     0.00  Enumerable::each_slice<400> {}
   2.79   53.70      2.01    1920600     0.00     0.03  Prawn::Images::PNG::unfilter_image_data<274> {}
   2.41    6.93      1.74     480000     0.00     0.01  Prawn::Images::PNG::unfilter_image_data<350> {}
   2.27    1.88      1.63    3667200     0.00     0.00  Numeric#abs
   2.22   41.27      1.60    1609600     0.00     0.03  Enumerable::each_with_index<200> {}
   1.75    2.72      1.26     185600     0.01     0.01  Prawn::Images::PNG::unfilter_image_data<297> {}
   1.68    2.19      1.21    1920600     0.00     0.00  Enumerable::to_a<150> {}
   1.61    4.39      1.16    1920600     0.00     0.00  Enumerable::Enumerator::each_with_block<70> {}
   1.58    1.75      1.14   27541261     0.00     0.00  Fixnum#<
   1.48    1.60      1.07   24124934     0.00     0.00  Fixnum#+
   1.47    1.31      1.06     960637     0.00     0.00  String#<<
   1.47    1.22      1.06    2444800     0.00     0.00  Fixnum#zero?
   1.37    3.20      0.99    1920600     0.00     0.00  Enumerable::Enumerator::each<60> {}
   1.09    1.13      0.79   15652806     0.00     0.00  Rubinius::Tuple#at
   0.82    0.61      0.59     960343     0.00     0.00  Array#pack
   0.80    0.72      0.58    6242171     0.00     0.00  Array#set_index
   0.66    1.11      0.48     112000     0.00     0.01  Prawn::Images::PNG::unfilter_image_data<291> {}
   0.66    0.71      0.48   10678512     0.00     0.00  Fixnum#>=
   0.63    0.68      0.45   10348190     0.00     0.00  Fixnum#-
   0.63    0.69      0.45   10678383     0.00     0.00  Fixnum.===
   0.52    0.38      0.38        383     0.99     0.99  GC.collect_young
   0.42    0.53      0.30     480000     0.00     0.00  Prawn::Images::PNG::unfilter_image_data<338> {}
   0.41    0.39      0.30    4350472     0.00     0.00  Fixnum#%
   0.29    0.31      0.21    4666547     0.00     0.00  Fixnum#==
   0.29    0.41      0.21      89600     0.00     0.00  Prawn::Images::PNG::unfilter_image_data<285> {}
   0.24    0.19      0.17     960000     0.00     0.00  Array#new_range
   0.18    0.15      0.13     960637     0.00     0.00  String#append
   0.16    0.17      0.12    2472984     0.00     0.00  Fixnum#<=
   0.15    0.15      0.11    1609840     0.00     0.00  Array#[]=
   0.15    0.15      0.11    1705600     0.00     0.00  Integer#floor
   0.14    0.14      0.10    1705603     0.00     0.00  Fixnum#/
   0.14    0.14      0.10    1920600     0.00     0.00  Rubinius::ByteArray#get_byte
   0.07    0.07      0.05     961043     0.00     0.00  Kernel#kind_of?
   0.07    0.07      0.05     960752     0.00     0.00  Rubinius::Type.infect
   0.06    0.04      0.04          2    21.75    21.75  Zlib::Deflate.deflate
   0.06    0.06      0.04     960848     0.00     0.00  Fixnum#>
   0.02    0.03      0.02     391382     0.00     0.00  Fixnum#-@
   0.02    0.02      0.02          1    16.74    16.74  Zlib::Inflate.inflate
   0.01    6.74      0.01        600     0.01    11.23  Enumerable#each_slice
   0.01    0.01      0.01        657     0.01     0.01  Array#shift

68 methods omitted

113 methods called a total of 167,634,034 times

The code:

doc = Prawn::Document.new do
  image png # png = StringIO.new(File.read('image.png'))
end
doc.render

Rubinius 2.2.1, Prawn master

The problem seem to be somewhere in Prawn::Images::PNG.unfilter_image_data. I'm trying to pinpoint the exact problem and to find how to fix it.

Gregory Brown
Owner

@cheba: Thanks for looking into this! Be sure to try either JRuby 1.7 or MRI 1.9.3+ as well, because those are the Ruby implementations we're targeting. But I think this issue is going to show up in any Ruby implementation, so we can start anywhere and just verify that later.

Alexander Mankuta
Collaborator

@sandal Sure. Though, I find it highly unlikely that 5 orders of magnitude in method calls number is solely an rbx issue.

Gregory Brown
Owner

@cheba: Absolutely, we know the problem is in our code. I was actually considering the opposite issue... that RBX generational GC might be a little bit more kind to us than MRI's GC.

Alexander Mankuta
Collaborator

@sandal I'm currently looking into performance issues.

Gregory Brown
Owner

@cheba: The performance issues are possibly related to excess object creation, and possibly also due to GC of temporary throwaway objects. For example, this code creates two strings for every single pixel. If we assembled all the values in a single array and then ran one pack statement, we'd massively reduce the amount of time spent on temporary object allocation and GC:

        color_byte_size = self.colors * self.bits / 8
        alpha_byte_size = alpha_channel_bits / 8
        pixels.each do |this_row|
          this_row.each do |pixel|
            @img_data << pixel[0, color_byte_size].pack("C*")
            @alpha_channel << pixel[color_byte_size, alpha_byte_size].pack("C*")
          end
        end
      end

https://github.com/prawnpdf/prawn/blob/master/lib/prawn/images/png.rb#L358-L366

For a single 640x480 image, the current implementation is going to create over 300,000 strings in this area of the code. I imagine there are ways to avoid that!

Gregory Brown
Owner

It's really just a shot in the dark, but I'd be interested in seeing what kind of results people with the code I just put up on #599. I am seeing minor improvements in memory and speed running our png_type6 benchmark, but I would like to see more realistic tests.

James Healy
Collaborator

For example, this code creates two strings for every single pixel

I think it's actually creating at least 4 strings per pixel. the 2 "C*" stings will be re-created on each loop. Thanks past-me.

James Healy
Collaborator

I've just rebased my old chunky_png experiment onto current master. The rebase is a bit rough, I'd want to clean it up before opening a pull request but it's useful as a quick experiment.

Here's the diff: master...chunky_png

Here's a script I used to check object allocations on MRI 2.0.0p353

$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
require "prawn"

GC.disable

before = GC.stat

Prawn::Document.new do
  image "#{Prawn::DATADIR}/images/dice.png"
end.render

after = GC.stat
total = after[:total_allocated_object] - before[:total_allocated_object]

puts "allocated objects: #{total}"

Here's the output on master:

⚡ ruby bench/png_type_6_objects.rb
allocated objects: 541656

Here's the output on the chunky_png branch:

⚡ ruby bench/png_type_6_objects.rb
allocated objects: 13853

I haven't benchmarked speed changes. It might be slower on PNGs with no alpha channel.

I used the following script to visually check various PNGs render correctly:

$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), 'lib'))
require "prawn"

images = [
      ["Type 0", "#{Prawn::BASEDIR}/data/images/web-links.png"],
      ["Type 2", "#{Prawn::BASEDIR}/data/images/ruport.png"],
      ["Type 3", "#{Prawn::BASEDIR}/data/images/indexed_color.png"],
      ["Type 4", "#{Prawn::BASEDIR}/data/images/page_white_text.png"],
      ["Type 6", "#{Prawn::BASEDIR}/data/images/dice.png"],
]

Prawn::Document.generate("png_types.pdf", :page_size => "A5") do
  images.each do |header, file|
    start_new_page unless header.include?("0")
    text header 
    image file, :at => [50,450]
  end 
end
Gregory Brown
Owner

@yob Neat stuff. I ran your allocation script against #599 and got 235900 objects... so about half of our current implementation, but MUCH more than ChunkyPNG. This doesn't surprise me, because ChunkyPNG uses a numeric bitmap to store the data rather than a ton of nested arrays.

Gregory Brown
Owner

I committed your objects benchmark to master... may be helpful for you @cheba if you're still looking at this too.

Gregory Brown
Owner

@yob: Now down to 5260 objects on #599 (although memory footprint didn't change much)

James Healy
Collaborator

Interesting. How are you tracking peak memory usage?

Gregory Brown
Owner

@yob: Very coarsely via OS X activity monitor (though something like top should also work) while running the png_type_6 benchmark. I bumped up the number of iterations on that so you have enough time to watch it.

Master is showing about 45-50mb, and my branch 25-30mb. Time is 45s vs. 33s on my machine.

Gregory Brown
Owner

I've thrown up a test repository for us to try out the different implementations with some giant PNGs:
https://github.com/prawnpdf/png-testing

I've just been running time ruby png_example.rb and then changing the branch in the Gemfile whenever I want to swap between implementations.

Here are the results I got on my machine (Ruby 2.0.0, OS X Snow Leopard). All of them use the 2402 × 1707 book.png file.

master: Runtime ~27 seconds, peak memory ~832 MB
png_optimization: Runtime ~18 seconds, peak memory ~ 325 MB
chunky_png: Runtime ~50s, peak memory ~ 583 MB

So my results are indicating that my recent set of optimizations are winning out in both memory usage and speed. It's still very slow, but then again, 2402x1707 is a giant file to be processing in Ruby.

I need to clean up the code on #599, but it's basic design might actually be an improvement, in that it extracts the filtering logic into its own class. I need to tweak naming, APIs, and use of variables throughout to make it suitable for merging though.

If I do that, is anyone opposed to me merging my optimizations as an incremental improvement? I will probably leave this ticket open until we're fairly sure we've exhausted our options, but it would be a step towards making things at least a little better!

Gregory Brown
Owner

It's worth noting two other points:

1) prawn-fast-png is currently not compatible with Prawn's master branch. Hopefully someone will fix it, otherwise we should probably offer a patch ourselves.

2) when reverting to Prawn 0.12.0, prawn-fast-png runs in about 1 second, and takes up < 100mb at peak memory in the Ruby process (unsure if ImageMagick ends up kicking other processes under the hood).

Maybe we should be digging into ImageMagick for algorithmic inspiration?

Gregory Brown
Owner

Whoa! Using my https://github.com/prawnpdf/png-testing code, @cheba's work on #600 is chewing through the test PNG in 4.7s and using only ~70MB of memory. That's awesome!

Can someone try his code in a real project for a proper smoke test?

Alexander Mankuta
Collaborator

Also please check the output for correctness. I've tried a few images and for all the output was correct.

While theory behind the method seem to be pretty solid, I doubt it was intended by the PNG spec authors.

Gregory Brown
Owner

The output looked correct, also if you run the spec/acceptance/png.rb I checked into master the other day, it appears to be generating good output for all PNG types. I put a bright red background in it to prove transparency is working.

Alexander Mankuta
Collaborator

Generally, Prawn tries to do as little manipulations to images as possible. Though, there's one exception. When you add a 16bit PNG to a document its alpha channel gets downsampled to 8bits.

The comment states that Adobe Reader can't handle 16bit PNG channels. But in the code only alpha channel is downsampled.

I tried to use 16bit data for both color and alpha channels and it seem to work just fine in all renderers I have access to (Apple Preview, PDF.js, Adobe Acrobat Pro X, Adobe Reader XI).

Can you please verify that 16bit images are rendered correctly on all renderes that we support (BTW, is there a list?).
16bit alpha PDF

It should look like this:
screen shot 2013-12-17 at 4 58 06 pm

Gregory Brown
Owner

Our list is very small and ought to be expanded, but officially it's Adobe Reader on all platforms, and
Preview on OS X. Rendering looks correct on both of those on OS X for me.

Alexander Mankuta
Collaborator

Probably some versions should be specified for those renderers.

Specifically, in this case the comment was written when the latest version of Adobe Reader was 9. Do we target only Adobe Reader XI now? It's been out for over a year now.

Gregory Brown
Owner

We officially target the latest released version for each platform, and do the best we can with older versions. In practice we rarely run into viewer-specific problems with Adober Reader and Preview, as long as we follow the spec.

Keep in mind our support policy is based on our available resources, rather than what we'd like to be
able to support. We can definitely refine and extend this in time.

Gregory Brown
Owner

Also, I am totally okay with shipping something that we may need to revise or revert later. We can learn a lot by releasing code to our actual users, and as long as we act on the feedback we receive, it's okay for us to incrementally improve our stability.

Gregory Brown
Owner

With @cheba's patch merged, we have about as decent performance as I'd expect us to be able to get out of Ruby. Additional improvements are welcome, but I think it's OK for us to close this ticket for now.

Please report back with results, though!

Benjamin Curtis

Since you asked for reports, here's one -- This change made a significant difference in an app I'm working on. Running 0.12 I was hitting memory limits on Heroku, and running 0.14 I'm not. Awesome work, thanks!

Alexander Mankuta
Collaborator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.