How to handle OpenGL drivers that don't support off-screen antialising?

[deanmoriart]

I tried a lot of animation softwares and nearly all of them has a solid antialising system on preview while drawing. I can never get the same stroke quality in open toonz no matter what I do. I am trying to minimize the problem with soft brushes for now. I am on linux. Is this a known problem ?

Other than that this app is just wonderful. Don't know how to thank you.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[deanmoriart]

This line is vector and that's how it looks in 1080p..

[deanmoriart]

And this is a vector line from Tupi. (Again 1080p)
There is a huge difference sadly and I don't know what to do except going raster.

[RodneyBaker]

Just to check.... Is it possible you have 'Display vector as Raster' turned on?

…

On Tue, Apr 25, 2017 at 3:09 PM, deanmoriart ***@***.***> wrote: [image: test] <https://cloud.githubusercontent.com/assets/27736315/25405446/fe9b89b8-2a0b-11e7-8c69-5aa7c28b3f4f.png> And this is a vector line from Tupi. (Again 1080p) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1166 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACdQMPRW-EAOnXnqIkqLraamLotKBcNZks5rzlMHgaJpZM4NGfBZ> .

[deanmoriart]

@RodneyBaker Just checked it, it's off. Actually preview window is not that important to me but it effects output as well.

[artisteacher]

Do you have the camera preview toggled on or off? Try adjusting the view to actual pixel size (N) - if the line quality is still poor, then there is a problem.

Occasionally I've had issues where the vector lines all look extra jagged but it is very inconsistent. Often restarting the program returns the view of the vector line quality back to normal - on OS X.

[deanmoriart]

@artisteacher Thanks for the suggestions. But camera preview is off. It gets more jagged when I enable it. Bad thing is, it effects the output image not just a preview problem.

[artisteacher]

Have you added additional vector brushes? It might not be related but I believe that I first noticed this after adding new brushes. Drawing was pretty laggy too. I ended up removing the new brushes and I don't think I've had the problem since.

[morevnaproject]

Hello @deanmoriart! Yes, this is a known problem and it is related with videocard driver. I have no antialiasing on Intel card (native OS drivers), but with the same build antialiasing works fine on other machine with NVidia card (proprietary driver). As @blackwarthog suggests, installing official Intel drivers from Intel website could solve the problem, but I didn't tested that yet.

[deanmoriart]

@morevnaproject @blackwarthog Hey thanks for writing in. So I guess it's better go with raster for a while since ati graphic cards was already a huge pain in linux I better not to touch any driver again :) . Do you think if there is any hope to fix it in the next releases or it will stay as driver problem ?

Just for the record my graphics card is "[AMD/ATI] RV730/M96-XT [Mobility Radeon HD 4670]"

Thank you

[deanmoriart]

@artisteacher sorry I didn't see your last reply. I havent installed any vector brushes, using the regular one.

Can someone send a smooth vector line screenshot from Opentoonz please ?

[RodneyBaker]

Here's the requested screenshot of a smooth vector line: 

On Thu, Apr 27, 2017 at 11:20 AM, deanmoriart ***@***.***> wrote: @artisteacher <https://github.com/artisteacher> sorry I didn't see your last reply. I havent installed any vector brushes, using the regular one. Can someone send a smooth vector line screenshot from Opentoonz please ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1166 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACdQMFhXkedsE8F_jSegbRzmGrNd4qiqks5r0MBEgaJpZM4NGfBZ> .

I;m also adding a shot of that same line with thickness adjusted with the Pump Tool.  Note that this is all on Windows 10 not on Linux. Github appears to be treating my uploaded images differently today as they aren't showing directly. This might be because I responded via email.

[deanmoriart]

Thanks a lot @RodneyBaker .

So the question is Why it is effecting the output if it's about graphic card problem ? It should show it as jagged in the preview window but should render correctly as my noob logic saying to me since vector is a dot in space positioned as mathematical info right ? And why other apps doesn't have that problem ?

[RodneyBaker]

I'm not sure I fully understand the question because the output isn't being effected other than the fact that vector has to be rendered to raster in most image formats.

The same smooth line that is showing in the preview window is rendering as (relatively) smooth raster output on my end.

Two questions suggest themselves based on this observation:
What is different between your system and mine?
The obvious answer is that you are using Linux and I'm using Win10 but of course there are a lot more variables than that.

Attached is the output of a single three point vector line (thickened by the pump tool) to form a leaf-like line. This is directly rendered out of Opentoonz to my desktop.

The second question is similar to the first and that is; What settings in OpenToonz might be resulting in different display and output?

Regarding the second we likely need to see a video screen capture of your process and share some files.

[deanmoriart]

Thank you for your interest @RodneyBaker . This is the most smooth output I can get from vector level. No matter how high the resolution is it still looks so jagged.(and those jagged parts moves during animation so it becomes more problematic. I thought I was seeing the line like that because of my graphic card but I can see other people's export smooth and without any problems just like your leaf line.So I don't know what's happening.

It s like some kind of optimisation is missing during rendering to raster ?

[morevnaproject]

@deanmoriart

So the question is Why it is effecting the output if it's about graphic card problem ? It should show it as jagged in the preview window but should render correctly as my noob logic saying to me since vector is a dot in space positioned as mathematical info right ?

The rendering uses functions of graphic card to process the rendering output. This is why you get aliasing problem on both preview and render.

Do you think if there is any hope to fix it in the next releases or it will stay as driver problem ?

As far as I know, this can be fixed (at least on rendering level) by implementing antialiasing with software (for the cases when hardware support for necessary functions is missing).
But at the moment this is not on our plans.

[ideasman42]

Note that using rendering multiple times with jitter and accumulating can work well, Blender uses this calling it 'full samples'.
It has the advantage that it doesn't depend on the graphics drivers AA implementation. See docs

https://developer.blender.org/diffusion/B/browse/master/source/blender/editors/space_view3d/view3d_draw.c;a85f457195045dff3d31d4de739d9a98fba62cc2$3358-3418

[deanmoriart]

@morevnaproject I understand. Thank you.

[jpturcotte]

@ideasman42 Could something similar be easily implemented in OpenToonz?

[ideasman42]

@ideasman42 Could something similar be easily implemented in OpenToonz?

I don't see why not, though I don't know this code in OpenToonz well.

[morevnaproject]

@deanmoriart As a temporary workaround, you can render with double resolution and then scale your video down.

[deanmoriart]

@morevnaproject That worked ! So editing app fixes it while scaling down ? Thanks a lot I really needed that workaround.

[deanmoriart]

Hey, just for the info;

I managed to have perfect smooth lines on preview window with enabling Morphological anti-aliasing based on Jimenez'MLAA in Driconf settings. ..(I don't know what all of this about but it worked) It doesn't fix output tho.

[jpturcotte]

What is that? Could you tell us more, please?

[deanmoriart]

It's a configuration tool(applet) for open source graphic card drivers in ubuntu.(I guess) You can find more info in here.

https://dri.freedesktop.org/wiki/DriConf/

I changed this setting to 12 and it totally fixed the problem for me in preview window. Output still looks jagged.

[deanmoriart]

I installed latest mesa drivers to see if it can fix the output and now even that Jimenez antialising not working on preview. :) I really should stop doing random things.

[jpturcotte]

@morevnaproject & @ideasman42 If we started a bounty, could any of you or someone you know look into this? This is pretty ridiculous. Also, @shun-iwasawa.

[morevnaproject]

@blackwarthog what do you think about that?

[blackwarthog]

I need to investigate sources. I'm not sure that is proper way to evolve OpenToonz - hardware 3d acceleration is now the system requirement, and i don't now why we should introduce the software rendering. I've tried to be wise and objective :), actually i think that final rendering should be as accurate as possible: software or OpenCL (open computing language). Not OpenGL (open graphics library) - because it will always add some restrictions and simplifications.

In our lab we does not found good solution to render vectors at GPU. At hardware (GPU) we have almost same results as at software (CPU), but we uses only one CPU thread - so actually in our case CPU overtake videocard twice (we tested OpenGL with supersampling and HDR). So in some case it may be good solution to use software-renderer for vectors.

Also we have some artifacts with our renderer:

These artifacts seems solvable at HW if we'll use some kind of stencil test, but it will affect antialiasing. I'not sure. @ideasman42, @shun-iwasawa, am i right?

I need to investigate sources to undestand how SW solutions applicable to OpenToonz architecture, and listen the opinions of @ideasman42 and @shun-iwasawa.

[deanmoriart]

Some more questions; (apologies for being pain in the ass)

-Is there anything we can do to help as non coders ?

-Should I also report this to open source driver developers or they have nothing to do about this ?

-I don't know how to do this but switching to software rendering can fix the problem ?

-Does the vector antialiasing method Pencil 2d, Inskcape or Tupi uses can be useful to be implemented ? Those apps can view and export the vector lines perfectly. However I have the same problem in blender with grease pencil in viewport but it fixes the output with full sample as @ideasman42 said before.

[morevnaproject]

I was talking about the drivers, haha.

Ah, I see now! ^__^ In any case, it worth to submit a bugreport to maintainers of Intel and ATI drivers. You can reference this thread also.

By the way, I cannot access https://sources.morevnaproject.org/ws-software/OpenToonz.

Fixed by rebooting, sorry for inconvenience. We have weak servers, so they run out of memory sometimes because of overload. ^__^"

Does the fix take into account what drivers the user is using? If anti-aliasing can be done on the GPU with some drivers

The fix is implemented purely on GPU. No CPU rendering here.

With this fix the anti-aliasing should be driver-independent now. So, the tests with NVIdia cards are also welcome (just to make sure we don't have any regressions). ^__^

[ghost]

If you submit a PR, I can help confirm that nothing is broken on Windows.

[morevnaproject]

@turtleTooth Done.

[virr44]

@morevnaproject
I can confirm that this is working in Linux without
$ optirun ...
Very cool!

[ghost]

@virr44 Is your text antialiased? I'm trying to see if another issue is related to this.

[jpturcotte]

The fix is implemented purely on GPU. No CPU rendering here.

Just to make sure, it runs on embedded solutions like Intel HD Graphics and AMD APUs, though?

[virr44]

@turtleTooth nope it's jagged...

[morevnaproject]

Just to make sure, it runs on embedded solutions like Intel HD Graphics and AMD APUs, though?

Yes, it works on any videocard, even embed one. I have tested the fix on my Intel HD Graphics 3000 CPU-integrated card - now I have anti-aliasing in OT.

Is your text antialiased? I'm trying to see if another issue is related to this.

nope it's jagged...

It is possible to fix the problem with text in the same way as we did here for strokes. Since we are busy with finishing MyPaint brushes now, we will do that later.

[ideasman42]

Wouldn't it be simpler to do this by jittering the projection matrix. performing complete renders - and accumulating all buffers back together for the final result? - this way there is no need to add jitter support piece-meal to each primitive type.

This will likely make the drawing code less complicated as well.

(as mentioned in my previous post - with reference code)

[morevnaproject]

@ideasman42

Wouldn't it be simpler to do this by jittering the projection matrix. performing complete renders - and accumulating all buffers back together for the final result?

This will slow down the rendering, because that would require re-rendering same frame several times.

this way there is no need to add jitter support piece-meal to each primitive type.

We just fixed the implementation that already was in OpenToonz. It is actually possible to implement some other algorithm, which shouldn't cause slowdown (as full-frame jittering does) and @blackwarthog asked about any opinions on this here - #1166 (comment)
But we got no response, so we just fixed existing approach.

It's actually no problem to add anti-aliasing to shapes (and thus text), but at the moment we are busy with delivering MyPain brushes feature.

The disadvantage of current approach of anti-aliasing can be observed if you create a very thick line and set them semi-transparent. You will notice the line have artifacts on it. You can see that on illustration, provided by @blackwarthog (notice red lines) -

So, of course, it's best to re-implement current anti-aliasing algorithm. As an option, it is possible to do projection matrix jittering, as you suggested, but for final rendering only. I.e. keep current algorithm for workarea preview (to keep fast speed), but use projection matrix jittering for final render (to avoid artifacts). What do you think?

[ideasman42]

This will slow down the rendering, because that would require re-rendering same frame several times.

Sure, but this is what the graphics card does internally (in the case of full-sample AA).
Which isn't simply for smooth outlines, this over-samples images for higher quality output too.

Keep in mind this is a workaround for old/cheap hardware. And it can be used only for offline rendering, where OpenGL is quite fast already.

Probably in a year or two it can be removed, at some point we will drop support for old graphics cards that don't support AA.

My point is that having to support AA for each primitive type makes code more complicated, just to support rendering on hardware with missing OpenGL features is a net-loss in long term maintainability.

For Blender I added full sample AA. Renders may go from 1/50th seconds to 1/10th of a second (depending on sample settings), its still quite reasonable.
And for people who are seriously working on animations in a studio - they can get well supported hardware for their renderfarms.

[morevnaproject]

And it can be used only for offline rendering, where OpenGL is quite fast already.

This is what I have suggested in the last paragraph of my message - #1166 (comment)

Renders may go from 1/50th seconds to 1/10th of a second

If users are agree to have their real-time rendering 5 times slower, then sure we can do that. @deanmoriart, @jpturcotte do you want this?

Another question - how would you detect if full-sample AA should be enabled or not? Do you suggest to add an option or have some heuristic depending on OS/driver? In our current solution it works for every card without any additional conditions.

I understand the advantage of having full sampling for images, though.

[ideasman42]

If users are agree to have their real-time rendering 5 times slower, then sure we can do that.

Users won't like this of course, but this is a fallback for old hardware.
Maintainability is important, so it's a tradeoff.

how would you detect if full-sample AA should be enabled or not?

For off-screen AA render in Blender we check for:

GL_EXT_framebuffer_blit 
GL_ARB_texture_multisample
GL_EXT_framebuffer_multisample_blit_scaled
GL_EXT_framebuffer_multisample

Note that GL_EXT_framebuffer_multisample_blit_scaled is required when blitting from a multi-sampled buffers, even when you're not scaling.

I would guess most or all of these apply to OpenToonz.

Having said that, we found graphics cards can't be relied on to tell the truth (Intel / AMD, especially laptop GPU's), so this should probably be an made an option, "Software Anti-Aliasing".

[morevnaproject]

And for people who are seriously working on animations in a studio - they can get well supported hardware for their renderfarms.

There are serious people who work outside of the studio. I am an artist-on-the-go - I am often working on the road, drawing on my Lenovo X220t laptop. It have integrated drawing screen tablet, so it is very comfortable for drawing (including animation). It is not a cheap hardware. Still, it have Intel graphic card and I am on Linux, so I am affected by this issue. There are even more expensive analogues, including Wacom Cintiq Companion 2 (http://wacom.com/en-us/products/pen-computers/cintiq-companion-2) and they all use Intel cards. Your suggestion to make screen refreshing 5 times slower (which will affect not only vectors, but also raster drawing operations) doesn't sounds appealing.

[ideasman42]

I'm only suggesting to use this for offline rendering. Slower screen refreshing is of course unacceptable.

Any workaround for missing AA will have a performance trade-off, it's not as if rendering multiple passes causes offline rendering to suddenly become unusable.

And over-sampling per-primitive isn't really a great solution either... Im just suggesting if a fallback is needed for poor hardware support - add one which doesn't add too much maintenance overhead.

[morevnaproject]

I'm only suggesting to use this for offline rendering. Slower screen refreshing is of course unacceptable.

In this case I am completely support your proposal! ^__^ I think it would be nice to open separate issue/request for full-sample AA implementation at offline rendering.

[ideasman42]

@morevnaproject - sure, think its OK to close this report and open an issue for full-sample fallback/workaround.

Note that this feature could be communicated a bit differently. Full samples look noticeably better then the graphics cards over-sampling, if it doesn't use FSAA (especially for textures). And if you have different graphics cards you may notice a difference between the exact method of AA. So suggest to make this an option for offline rendering, so for example - you can do 32x samples ... even if the hardware doesn't support that.

[morevnaproject]

Thanks for the hints! Please keep this issue open until we add anti-aliasing for shapes.

[jpturcotte]

@morevnaproject & @blackwarthog Hi, guys! Are you still working on this, and can we have any news? Also, @deanmoriart, have you tested this solution? Thank you!

[morevnaproject]

If everyone is happy with the solution, then we are ready to submit it as pull request (for outlines only). ^__^ Anti-aliasing for regions will come up later.

[jpturcotte]

@morevnaproject Thanks a lot! I am eager to see your solution for shapes too.

[jpturcotte]

Sorry for the bump. @morevnaproject & @blackwarthog I see PR #1218 has been closed. Could we have an update on this, please?