vips fails with SVGs > 10MB

[tokee]

I am trying to use vips with 30-50MB SVGs, but it fails fast:

> vips copy '1998_to_2003.svg[dpi=10]' t1.png
glib: Error domain 1 code 1 on line 789142 column 7 of file:///home/te/projects/graph_presenter/1998_to_2003.svg: internal error: Huge input lookup

Looking around, it seems to be a security feature of rsvg that caps SVG processing to 10MB SVGs. Running from the command line, the flag --unlimited can be passed to rsvg-convert to disable this feature and testing locally confirms that it works:

> /usr/bin/rsvg-convert -w 320 -h 200 -o T111815_test.png T111815_test.svg 
Error reading SVG:Error domain 1 code 1 on line 15 column 287195 of data: internal error: Huge input lookup

 > /usr/bin/rsvg-convert -w 320 -h 200 --unlimited -o T111815_test.png T111815_test.svg 
 > identify T111815_test.png 
T111815_test.png PNG 320x200 320x200+0+0 8-bit sRGB 97.2KB 0.000u 0:00.000

I looked in svgload.c in the vips project and it seems that the flag RSVG_HANDLE_FLAG_UNLIMITED can be used to programmatically disable the feature.

Alas, my c-skills are nearly non-existing, so I am unable to figure out how to provide the flag in the code. Could this be made into a vips feature or is there another way I can bypass the SVG size limitation?

[jcupitt]

!!!!!

[jcupitt]

Sorry @tokee, I was reeling in horror :)

I guess there would need to be a new option to svgload to remove the limit. How about:

vips copy '1998_to_2003.svg[dpi=10,unlimited]' t1.png

Would that work?

[lovell]

This problem has previously been reported against sharp - see lovell/sharp#1339 (comment) - I took a quick look at the time but IIRC it's not a straightforward change as the RSVG_HANDLE_FLAG_UNLIMITED flag is not available on the librsvg API that libvips currently uses.

[tokee]

Thank you for your very fast follow ups, @jcupitt & @lovell!

There is also a Wikimedia thread about this which handily provides

curl -s 'https://upload.wikimedia.org/wikipedia/commons/e/e7/Thousand_Island%2C_St._Lawrence_River.svg' > T111815_test.svg

to get a sample SVG that fails. That's the one I used for my report above.

[jcupitt]

Oh I see, yes you're right @lovell, we'd need to switch to gio for this, ouch.

[jcupitt]

I had a quick hack and I now see:

john@kiwi:~/pics$ time vips copy T111815_test.svg[dpi=10] t1.png
glib: Error domain 1 code 1 on line 15 column 287195 of data: internal error: Huge input lookup
real	0m0.579s
user	0m0.501s
sys	0m0.078s
john@kiwi:~/pics$ time vips copy T111815_test.svg[dpi=10,unlimited] t1.png
real	0m3.389s
user	0m3.170s
sys	0m0.218s

It'll need some testing. I've not used gio before and I don't know how well it works on macos / windows / freebsd / etc. We'll need to bump the minimum version of librsvg in configure.ac as well.

[tokee]

Awesome response time! I checked out the branch and it seems to work (Ubuntu). Thanks a lot!

There also seems to be a problem with text labels on my SVG graphs that started it all, but I'm pretty sure that's on rsvg and not vips. I'll report back if it turns out to be vips-related after all.

[kleisauke]

The add-unlimited-to-svgload branch seems to work on Windows with the latest librsvg (v2.45.6):

C:\test-images>vips copy T111815_test.svg[dpi=10] t1.png
glib: XML parse error: Error domain 1 code 77 on line 15 column 287195 of data: Premature end of data in tag svg line 3

C:\test-images>vips copy T111815_test.svg[dpi=10,unlimited] t1.png

(vips:10608): Pango-WARNING **: 15:43:04.035: couldn't load font "Helvetica Not-Rotated 36.25", falling back to "Sans Not-Rotated 36.25", expect ugly output.

(vips:10608): Pango-WARNING **: 15:43:04.049: couldn't load font "Helvetica Not-Rotated 40.830078125", falling back to "Sans Not-Rotated 40.830078125", expect ugly output.

(vips:10608): Pango-WARNING **: 15:43:04.055: couldn't load font "Helvetica Not-Rotated 39.98046875", falling back to "Sans Not-Rotated 39.98046875", expect ugly output.

(vips:10608): Pango-WARNING **: 15:43:04.062: couldn't load font "Helvetica Not-Rotated 37.0498046875", falling back to "Sans Not-Rotated 37.0498046875", expect ugly output.

I'm also going to try to test with librsvg v2.40.19 on Windows.

Note that according to the librsvg documentation loading an SVG with GIO is the preferred and most resource-efficient way, so this can also be seen as a nice improvement.

Otherwise, loading an SVG without GIO is not recommended, since librsvg will need to buffer your entire data internally before actually being able to parse it. ... Still, please try to use the GIO stream functions instead.

[jcupitt]

That's good to know. I was worried that there might be extra linking requirements on some platforms. I'll test on macOS as well.

[jcupitt]

It seems to work on macOS too, with both clang and gcc.

Should we merge to master?

[tokee]

I can happily confirm that the experimental version performs perfectly under Ubuntu for our use case: Rendering a 70MB SVG to 25 gigapixel took 1 hour (it seems to be single threaded?) on a 16GB desktop and produced https://labs.statsbiblioteket.dk/linkgraph/1998_to_2003/ which, besides the resolution jump from our previous GraphicsMagic rendered version, also solved some problems we had with font kerning.

I remain in awe of your response time and problem solving skills. Thanks again!

[jcupitt]

Recent versions of librsvg will thread some operations, like gaussblur, but it's quite limited. It does not support several threads rendering different parts of the image (I think).

I'll merge. Thanks for the feedback.

[jcupitt]

OK, this will be in 8.9. Thanks!

[bluecorridor]

Will there be a 'unlimited' flag option for writing out image pyramids (dzsave)? I seem to be encountering a similar message when processing a large .SVG file.

[jcupitt]

Yes, it's an option for input, not output, so you just do:

vips dzsave huge.svg[unlimited] my-deepzoom-pyramid