Resized images pixelated - option for better downsampling available? [$300 awarded]

[NewsteadEric]

Regarding https://groups.google.com/forum/?hl=en#!topic/fabricjs/N5LOujX-qMA it would be great to have an option for better resampling when scaling images with fabric:

It looks like fabric.js currently uses only the browser's native/default (bilinear?) resampling capabilities, which usually result in very pixelated, low-quality images when scaling down.

I would like to request a feature that avoids pixelation and resamples images to a higher quality especially when scaling down (similar to http://jsfiddle.net/qwDDP/), e.g. when the user uses the scaling handles on an image element in a fabric.js canvas, or when the scaleX/Y or width/height properties are used.

Depending on performance impact, this could happen during scaling, or when scaling has completed, but should yield a high quality scaled image in any case.

Addendum: As far as text resampling suffers the same issue (which it may not), it would be ideal if that was addressed as well.

The $300 bounty on this issue has been claimed at Bountysource.

[luron01]

Got the exact same issue than NewsteadEric. Am adding a $50 bounty.
Guys, christmas is not very far away, come get those 300 bucks :).

On a side note, am curious on how teespring.com handles this issue.

[luron01]

Hi Zolomon,
Thanks for replying. I think NewsteadEric described pretty well the issue we're facing: when scaling down an image on fabricjs, we got a very pixelated image. See his fiddle, you'll understand straight away: http://jsfiddle.net/pVE33/1/.
We need the scaled down image to be of proper quality, without all these pixels. "Depending on performance impact, this could happen during scaling, or when scaling has completed, but should yield a high quality scaled image in any case."

From what I understood on Eric's post, he suggests handling the issue by stepping down several times the scaling. For instance, instead of dividing once the scale by 8, divide it by half 3 times. It gives better results see here (http://jsfiddle.net/qwDDP/). There may be even more improvements to bring. Perhaps looking at the interpolation method (linear vs bicubic vs lanczos). All in all, when the image is scaled down server-side with Imagemagick, the sizing is very clean. When scaled down with fabric.js, the quality is as poor as you can see on the 1st fiddle.

I'll let @NewsteadEric comment further on this as he may actually have more precised info.

[schumskie]

Hi to everybody,
I would also try to solve this solution, and this is my thoughts:
Eric has a good idea, to scale image down multiple times, but optimization stays as problem, cause calling a scale method multiple times can be expensive I suppose. I think that making multiple samples of an image could do the trick, and storing them into memory (something like openGL does) will be more efficient. So this is what I think it should be done:

1. Load Image
2. Make sample images out of the original with sizes which are power of two (example: if image is 1000x800, it will be scaled first to 1024x1024, and then those new image will be scaled down recursively by 0.5 factor (so we will have 1024x1024, 512x512, 256x256, 128x128, 64x64...)
3. Choose best fitted image by scale parameter and scale it to calculated dimensions(if scale is 0.3 then 1000_0.3x800_0.3 would be 300x300 so our best would be to scale 512x512 image)

On this way we shouldn't lose any performance, we shall need a little bit more memory but I think it shouldn't be concern, or maybe I'm wrong? So when we need scaling we will just use 3) point, cause first two's had to be done on initialization.
If you think that this strategy is good, or have any suggestions or pointers please let me know.
I would be also grateful if some of fabric.js developers give me some brief walk through the code, so I could implement this feature faster.
Also, I think it isn't wise to implement new scaling function, cause I'm pretty sure it will be slow because of java script...

[asturur]

would nt be better to have some resize filter implemented in the fabric filter class?
so we could choose between wich filter and also building it as plugin to do not enlarge too much the basic library.
this is what i m trying to do as now.

[luron01]

@ByteRogue > you mention creating sample images that are powers of 2, wouldn't it stretch the image? Unless you have an idea to overcome stretching the image, we may be better off scaling down recursively the image starting from its original ratio.

Ok, here's an idea to overcome the streching problem... we could add transparent pixels to a side until we reach a power of 2 ratio, then scaling down as you proposed, and then cropping the artificially added transparent pixels. I read something interesting here: http://stackoverflow.com/questions/6355954/resize-image-without-distortion-keeping-aspect-ratio-then-crop-excess-using-wide?rq=1
But... that may just bring too much complexity and... actually... not bring much of any good.

[kangax]

The image filter idea is excellent. It would be the most idiomatic solution to this problem, and judging by http://fabricjs.com/image-filters/, the performance is probably not going to be (much of) an issue.

[asturur]

Some progress report:
i create a filter that use fabricJs filter structure and that implements the "multiple resize hack"
How i use it:

fabric.Image.fromURL('pug.jpg', function(img) {
  img.filters.push(new fabric.Image.filters.Resize({scaleX: 0.2, scaleY: 0.2}));
  img.applyFilters(canvas.renderAll.bind(canvas));
  canvas.add(img);
});

Left is scaled by hack, right is normally resized.
Now some logic in the scale operation to always scale by the filter instead of applymanually is required.

[asturur]

I added this:

 if (this.isMoving === false && this.resizeByFilter && fabric.Image.filters && this._needResize()) {
   this.applyResizeFilters();
 }

in the _render so the image is filtered just when needed, otherwise the performance hit is visible.

this is a comparision screenshot:

Tomorrow i try to add different filter method and i will try to finalize the filter class and submit it for a first review from kangax.

[kangax]

Looks pretty good, Andrea. Thanks!

On Sun, Nov 2, 2014 at 1:36 AM, Andrea Bogazzi notifications@github.com
wrote:

I added this:

if (this.isMoving === false && this.resizeByFilter && fabric.Image.filters && this._needResize()) {
this.applyResizeFilters();
}

in the _render so the image is filtered just when needed, otherwise the
performance hit is visible.

this is a comparision screenshot:
[image: image]
https://cloud.githubusercontent.com/assets/1194048/4873832/2c32c926-6228-11e4-960d-79f2d794c718.png

Tomorrow i try to add different filter method and i will try to finalize
the filter class and submit it for a first review from kangax.

—
Reply to this email directly or view it on GitHub
#1741 (comment).

[asturur]

I added even hermite fast resize alghoritm, working pretty good.
Now i'll give a shot to lanzcos3.
Is cool, just i need help to solve some async problem ( the filter finishes after the rendering process, so either i re render the scene, or i re render just the image, but this could make mess with z ordering ).

[asturur]

@NewsteadEric any though on the work so far? do you like it, do you think is good for the bounty, should i go on?

[mobidev111]

Any branch available to test the result? Can you create one working version, i.e., plug one of the filter in, fix the mentioned async problem, etc.

Please document: How can I apply the filter?

Then I can support in testing the performance of the different algorithms on mobile devices.

[asturur]

Yes, as soon as i get an idea from @NewsteadEric i will finish it and give a repo for testing.
I have lot of things to do for this library in my freetime, this is a nice addition, but without support from the baker, is going back on the queque.
I hope no one get offended from what i said.

[asturur]

Asyncronous effect is solved, of course the effect is not immediate, because there are the timings of resize alghoritm to wait.
Effect is ok and code is not too much patched in my opinion.

[asturur]

@luron01 @NewsteadEric i would make a demo and a PR.
Are you still interested in this?

[luron01]

@asturur
Of course, I am :).
For me it would be awesome to have a jsfiddle... though I don't know if that's possible at this stage.

[asturur]

I think a will make a demo page, ok?

2014-11-07 15:55 GMT+01:00 luron01 notifications@github.com:

@asturur https://github.com/asturur
Of course, I am :).
For me it would be awesome to have a jsfiddle... though I don't know if
that's possible at this stage.

—
Reply to this email directly or view it on GitHub
#1741 (comment).

[luron01]

sure

[asturur]

@luron01, @NewsteadEric , @kangax , @Kienz ( what a spammer i am )
Ok i did a jsFiddle demo finally

http://jsfiddle.net/asturur/6qaacwea/12/

or to test more free

http://www.deltalink.it/andreab/fabric/resize.html

The fiddle has 4 buttons, for normal image, "divide by 2 hack" , hermite algorithm and another try that is a step resize ( instead of dividing by 2 i resize by equal steps to the destination, but is not so good )

You can compare them.

comparing to first screenshot i semplified the parameters and the functionality.

I will add even lanczosresize maybe.

[kangax]

Very cool. "slice by 2" seems the blurriest one, at small size. But also the "softest" one, naturally. "Steps hack" still looks a bit pixelated to me, and hermite seems the best. Especially, if you shrink image even more:

Notice how hermite is so much clearer than "slice by 2" (which is really blurry).

[asturur]

the alghoritm are used even for upscaling, just to do not leave that
operation unchecked. we could even make a double decision, when upscaling
use that, when downscaling this other.
i tweaked it so it works even if you scale the image in a way that one
dimension is shrinked, the other is enlarged. (did not test with hermite
code really now that i think of it )
Il 09/nov/2014 16:41 "Juriy Zaytsev" notifications@github.com ha scritto:

Very cool. "slice by 2" seems the blurriest one, at small size. But also
the "softest" one, naturally. "Steps hack" still looks a bit pixelated to
me, and hermite seems the best. Especially, if you shrink image even more:

[image: ckkfx4b0luulx3ah0zwermtwwf-kuubyli97mcltcbq]
https://cloud.githubusercontent.com/assets/383/4967814/94490864-6826-11e4-82e5-4273918a5e38.png

Notice how hermite is so much clearer than "slice by 2" (which is really
blurry).

—
Reply to this email directly or view it on GitHub
#1741 (comment).

[NewsteadEric]

I only just had a chance to check in on this again, and it looks very nice already so far!

I'd like to test this with a few different images - the above jsfiddle link (http://jsfiddle.net/asturur/6qaacwea/12/) produces a 404 for me. Where would I find the latest version to test? (I see that http://jsfiddle.net/asturur/6qaacwea/11/ seems to work, but that might be an older version)

Eric

[mobidev111]

Mayby not so important, but just to mention it: What about very small scale? any improvement possible there? See example with a SVG: http://jsfiddle.net/6qaacwea/13/

[NewsteadEric]

Just got around to testing this at http://jsfiddle.net/asturur/6qaacwea/11/ - works great so far. A few things:

• Sometimes the resize quality still seems to be a little hit or miss - a smaller size may actually render nicely, and then scaling up a little will result in more pixelation, or even more blur
• When using the Slice by 2 Hack or the Steps Hack, transparent PNGs will render with resized duplicates of themselves on top of each other; I assume this is due to the approach you explained above, but can this be fixed?
• Do you see any way to improve scaling of line drawings even further - for example, http://upload.wikimedia.org/wikipedia/commons/f/fe/Drum_kit_illustration.png

[asturur]

this is the correct link, http://jsfiddle.net/6qaacwea/12/ but the difference is only chrome related. Resize is the same.

For your first point, the hit and mis , yes. I think is like that. When you have a zebra, for example, and you have to draw 2 stripes but your scaling gives you just on 1.7 pixel to do that, the result is questionable, if you scale up a little bit and you get to 2 pixels, you can have a prettier render. I think is nothing related to fabric or resize code, i think is normal.

Good find for the Png's with transparency, this is gonna be fixed for sure.

To improve further we can just try some other algorithms (lanczos and classic trilinear)

[asturur]

thanks for reporting, i m gonna check it.

[lionellei]

Hi @asturur any update? I am pretty new to this so I am not much a help but reading the documentation, it looks like JSON.stringify(canvas) will call each of its object's toJSON method. So is it this resize filter object's toJSON method not correct?

[lionellei]

Hi @asturur, I am sorry to bug you again, but we really would like to have this issue fix for our release. I would post bounty if need be (have not figured out how to yet, or if only certain people could post bounty).
Also, text is pixelated too.

[asturur]

i m sorry i got other issues. if you mamage to make a jsfiddle for me, so i have the code causing the problem, i will look into it today.

[lionellei]

Hi, I did not manage to get a jsfiddle, I am not too familiar with that but it will be a quick test to just try to load this JSON:
"{"objects":[{"type":"image","originX":"left","originY":"top","left":7.64,"top":11.36,"width":880,"height":942,"fill":"rgb(0,0,0)","stroke":null,"strokeWidth":1,"strokeDashArray":null,"strokeLineCap":"butt","strokeLineJoin":"miter","strokeMiterLimit":10,"scaleX":0.31,"scaleY":0.31,"angle":0,"flipX":false,"flipY":false,"opacity":1,"shadow":null,"visible":true,"clipTo":null,"backgroundColor":"","fillRule":"nonzero","globalCompositeOperation":"source-over","src":"http://teeyeah-images.b0.upaiyun.com/1430384957/I_love_bipolar_black.png","filters":[{"type":"Resize","scaleX":0.3,"scaley":0.3,"resizeType":"hermite","lanczosLobes":3}],"crossOrigin":"","alignX":"none","alignY":"none","meetOrSlice":"meet"}],"background":""}"

once you load it, you will see that the canvas has one object and that object has width and height of 0.

Also once the resize filter is applied, and then if you serialize the canvas, the original image's width and height is changed.
For example, if original image is 1000 x 1000, and you apply a filter of scaleX and scaleY both are 0.5. And then you serialize the canvas, the JSON contains an image object of size 500 x 500, and a filter of scaleX and scaleY being 0.5. So what happen if you load canvas from this then? will the newly loaded image be 250 x 250? I can not try that since when I load it I get an object of size 0x0.

[asturur]

hi @lionellei,
i thought the issue was fixed.
i m out some days, i ll be back sunday and a i will check this situation.

which of the filter are you using? static resize on image load or dynamic resize or both?

[lionellei]

Hi, @asturur , no the issue was there for quite some time, I am just busy with other part of the project that I put off this by tolerating a pixelated scaled down image.

I was using static resize filter. The easiest reproduction will be to just load the json I provided above into a canvas and you will see nothing.

So just to summarize, the problem is two fold:

1. Resizing filter changes the original image's dimension, and serialization will save the resize filter in the image's filters property. If everything works accordingly, upon loading, will the original image be resize again? i.e., the filter be applied to the already scaled image? I tried removing the filter from the image after the first static resize, then serialize, but when I loaded back the image, the size is the pixelated scaled down version.

2. Loading a canvas containing an image from URL and with resize filters will yield a canvas object with dimension 0x0

[lionellei]

It's also possible that I maybe doing something wrong. So it will be nice if you could provide some example code, I am sure you must've done some testing when you implemented this. Thanks in advance.

[asturur]

as soon as i m back ( sunday ) i will make a demo with the functions. so we
find the bugs.
Il 30/apr/2015 04:57 PM, "Lionel Li" notifications@github.com ha scritto:

It's also possible that I maybe doing something wrong. So it will be nice
if you could provide some example code, I am sure you must've done some
testing when you implemented this. Thanks in advance.

—
Reply to this email directly or view it on GitHub
#1741 (comment).

[asturur]

@lionellei i checked the code and i found those errors:

• the filter of typer resize was not considering options when restored from object
• the exported option had a mispelled scaley instead of scaleY
• the filters at image initialize where applied twice
• the resizeFilters ( used for dynamic filtering on resize ) where not exported nor enlived from object.

now everything looks like it is working, as soon as i can i push a PR

i have also to check interaction with cropping as a separate issue of course.

[lionellei]

@asturur very nice. I was about to report the last bullet point and you beat me to it :)

[lionellei]

Hi @asturur, is there any fix to this issue? Thanks.

[asturur]

it has been merged recently.
it looks like it works good.

If you want to have a look, another check is even better.
Download the latest source and build it to test it.

[lionellei]

Hi @asturur, yes it works good in terms of now the filter is saved and could be reloaded. The image quality is better but still not very crisp when scaling down though.

[asturur]

which is a good filter? do you know.some? lanczos with 8 lobes? filters do
not do miracles anyway, it depends what do we expect.
Il 06/giu/2015 20:11, "Lionel Li" notifications@github.com ha scritto:

Hi @asturur https://github.com/asturur, yes it works good in terms of
now the filter is saved and could be reloaded. The image quality is better
but still not very crisp when scaling down though.

—
Reply to this email directly or view it on GitHub
#1741 (comment).

[jitendrapawar]

I tried this but it really affects performance, browser gets crashed.
I am using "lanczos 3". Can anyone help?

[asturur]

it is taxing yes. use a simpler one, bilinear or slice hack. are way faster.
On May 6, 2016 10:41 AM, "jitendrapawar" notifications@github.com wrote:

I tried this but it really affects performance, browser gets crashed.
I am using "lanczos 3". Can anyone help?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1741 (comment)

[jitendrapawar]

yes right, but that does not provide same accuracy as "lanczos 3".
Sorry to say, but I found another issue, if we use this filter, it gives error on using dispose function of canvas.

canvas.dispose();

[asturur]

can you create a jsfiddle with the problem?

2016-05-06 11:41 GMT+02:00 jitendrapawar notifications@github.com:

yes right, but that does not provide same accuracy as "lanczos 3".
Sorry to say, but I found another issue, if we use this filter, it gives
error on using dispose function of canvas.

canvas.dispose();

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1741 (comment)

[jitendrapawar]

sorry @asturur, I found that, I called canvas.dispose(); two times. Sorry for the trouble. But can we optimize the "image filtering". Or can you guide me to make it more optimized.

[asturur]

I'm pushing some control customization this weekend, polishing Itext regressions.From next week i'm gonna watch for general speedup with webgl filtering, i hope i can do something good/cool/performant.

[jitendrapawar]

Can anyone please explain me how image filter lanczos works ?

[vxrai]

@kangax To fix the pixelation of images on resize i used the sliceHack resize filter. After applying them, i resized the image to approx half of its original size and this time the image din’t pixelate. Everything worked perfect, life was awesome before I saved the canvas using Canvas.toJson and the next time when i reloaded the saved backup again on canvas using Canvas.loadFromJson the same resized image is now pixelated. It will be really great if you could help with this. If i resize the image again, anti aliasing will work as expected, but not sure why the anti-alised state was not retained when the images reloaded on canvas using LoadFromJson

[vxrai]

And here's the fiddle that replicates the issue - http://jsfiddle.net/varun598/jagfhh8d/22/

[Godweed]

I am facing this problem in dynamic patterns. I am attaching a high resolution image as pattern to object. And when trying to create a print version in node the pattern looks in small resolution.