New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WebGL] Reconcile the global matrix #637

Open
parasyte opened this Issue Jan 18, 2015 · 9 comments

Comments

Projects
3 participants
@parasyte
Member

parasyte commented Jan 18, 2015

This ticket will pick up where #591 leaves off. The goal here is to remove dependence on the global matrix, as much as possible. It will allow us to put almost every matrix transformation entirely within the GPU, where it can take advantage of highly parallelized vector ALUs that we don't have access to in JavaScript.

To start, recall that 3D rendering typically uses three matrices; called "model matrix", "view matrix", and "projection matrix". Working backwards, melonJS needs the projection matrix to transform our pixel coordinates to WebGL cube space coordinates using orthographic projection; and a view matrix is used for nested containers and other object that need relative drawing operations. The model matrix is unnecessary because the majority of entities drawn in melonJS are quads (exactly two triangles). A model matrix is more useful with large meshes, which have a greater number of vertices.

The scaling and rotation values can be streamed to the GPU as part of the vertex attribute array. These can be packed into 2 floats (a 3x3 matrix requires 6 floats for full precision) assuming 16 bits for each of the scaling coordinates. Additionally, the rotation angle and texture index can be packed into the same float. The anchor point will also be required, which is a 3rd float. These attributes can be applied within the vertex shader, since the order of operations is static; translation can be added to the destination coordinates before calling drawImage.

Entities that need a model matrix can implement one with a custom compositor and shader. Spine is an example that will benefit from a model matrix.

To reconcile the global matrix, we need to remove most of the matrix transformation usage. me.Sprite and me.Particle make heavy use of the global matrix right now. Here is the last proposal for a replacement: #591 (comment) The important part is that the first argument provides information about things like scaling and rotation for each drawImage call. The function signature will have to change to accept a texture region instead of coordinates, as an additional optimization. See: e785724#commitcomment-9329151

With that part done, each view matrix transformation will be treated as change in WebGL state; it will be sent to the GPU as a uniform variable, so changes to the matrix require flushing the compositor. This will be pretty efficient, assuming view matrix transformations only occur on me.Container boundaries.

Additional TODO items:

  • Implement fonts (hard), lines (easy), stroke (hard)
  • Create a packet inspector for debugging
  • Enable mipmapping on power-of-two textures
  • Pack multiple attributes into each float
  • Use a standard sampler2D-uniform on mobile

Attribute Packing: E.g. texture coordinates can be packed into a single float, since each axis will never be bigger than 65,536 pixels (fits into 16-bits). The packing algorithms will provide less than 16-bits of precision, probably 15-bits, which is still plenty for texture coordinates. Another candidate is colors, and much bigger win: pack a vec4 into a single float with almost no loss in precision!

@parasyte parasyte added this to the 2.2.0 milestone Jan 18, 2015

@parasyte parasyte modified the milestones: 3.0.0, 2.2.0 Jun 29, 2015

@parasyte parasyte modified the milestones: 3.0.0, Future Jul 26, 2015

@parasyte

This comment has been minimized.

Show comment
Hide comment
@parasyte

parasyte Sep 20, 2015

Member

I'll just add a note here: mobile platforms hate the fragment shader I wrote. The sampler2D array trick causes the GPU to run very poorly. (I've added a TODO item here to get this fixed.)

Member

parasyte commented Sep 20, 2015

I'll just add a note here: mobile platforms hate the fragment shader I wrote. The sampler2D array trick causes the GPU to run very poorly. (I've added a TODO item here to get this fixed.)

@obiot

This comment has been minimized.

Show comment
Hide comment
@obiot

obiot Nov 15, 2016

Member

@krojew i see that you have been working yourself on a webgl library, any chance you want to help us here optimize/complete our webgl renderer ? We still have a lots of improvements listed in this ticket but as well #720 and #593.

@parasyte can also provides further explanation on our implemetation :)

Member

obiot commented Nov 15, 2016

@krojew i see that you have been working yourself on a webgl library, any chance you want to help us here optimize/complete our webgl renderer ? We still have a lots of improvements listed in this ticket but as well #720 and #593.

@parasyte can also provides further explanation on our implemetation :)

@krojew

This comment has been minimized.

Show comment
Hide comment
@krojew

krojew Nov 15, 2016

@obiot I would be glad to help, but I have very limited time atm. I've not been able to even complete the typings yet.

But, I can give some quick tips for now from skimming over the description:

  • Offloading work to gpu is generally a good idea, but you need to know which operations actually benefit from that. For example: it's usually much faster to create a buffer on the gpu for transformations, create a transformation matrix on client-side, map it to the buffer with proper invalidation for updates and do one multiplication in the VS, rather than pass individual transformation elements to the VS (which I understand this issue is about, but I can be mistaken). You also gain more flexibility that way, e.g. skewing. This assumes WebGL can map buffers - I've been involved in multiplatform engines more than WebGL, and I can't remember that.
  • Unless you want to do distance-field font rendering, don't do that at all in WebGL. In fact, there's usually little reason to do that at all - use plain HTML. Fonts are a pain in the 3d world.
  • Instead of packing multiple values to single floats to save space, use half float types (if available - again can't remember about WebGL). Unpacking takes shader time and you need to handle interpolation yourself. In short - usually not worth it. Best uses are packing TBNs (which can drop one vector) or quats (which can drop one dimension and are usually pushed to GPU in large quantities due to animation/physics), but I don't think those are the things you need to care about.

krojew commented Nov 15, 2016

@obiot I would be glad to help, but I have very limited time atm. I've not been able to even complete the typings yet.

But, I can give some quick tips for now from skimming over the description:

  • Offloading work to gpu is generally a good idea, but you need to know which operations actually benefit from that. For example: it's usually much faster to create a buffer on the gpu for transformations, create a transformation matrix on client-side, map it to the buffer with proper invalidation for updates and do one multiplication in the VS, rather than pass individual transformation elements to the VS (which I understand this issue is about, but I can be mistaken). You also gain more flexibility that way, e.g. skewing. This assumes WebGL can map buffers - I've been involved in multiplatform engines more than WebGL, and I can't remember that.
  • Unless you want to do distance-field font rendering, don't do that at all in WebGL. In fact, there's usually little reason to do that at all - use plain HTML. Fonts are a pain in the 3d world.
  • Instead of packing multiple values to single floats to save space, use half float types (if available - again can't remember about WebGL). Unpacking takes shader time and you need to handle interpolation yourself. In short - usually not worth it. Best uses are packing TBNs (which can drop one vector) or quats (which can drop one dimension and are usually pushed to GPU in large quantities due to animation/physics), but I don't think those are the things you need to care about.
@obiot

This comment has been minimized.

Show comment
Hide comment
@obiot

obiot Nov 26, 2016

Member

@parasyte about this warning messages in the console, what about adding a debug option flag for the renderer, and only display theses messages when the flag is on ?

Member

obiot commented Nov 26, 2016

@parasyte about this warning messages in the console, what about adding a debug option flag for the renderer, and only display theses messages when the flag is on ?

obiot added a commit that referenced this issue Nov 27, 2016

[#637] add a `debug` flag to the WebGL renderer
Default to `off`. Once enabled, it will output warning messages in the
console to log any mi-usage of the renderer

obiot added a commit that referenced this issue Nov 27, 2016

[#637] rename the `debug` flag to `verbose`
it is a much more appropriate name for it. I also moved it to the base
renderer class, as it can be used as well to display more information
on the webgl or canvas renderer.
@obiot

This comment has been minimized.

Show comment
Hide comment
@obiot

obiot Dec 5, 2016

Member

now that the WebGL renderer is enabled by default with the platformer, I noticed that Firefox is also complaining a bit about it :

Error: WebGL: texImage2D: Incurred CPU pixel conversion, which is very slow.  melonJS.js:17367:17
Error: WebGL: texImage2D: Chosen format/type incurred an expensive reformat: 0x1908/0x1401  melonJS.js:17367:17
Error: WebGL: texImage2D: Incurred CPU-side conversion, which is very slow.  melonJS.js:17370:17
Error: WebGL: texImage2D: Incurred CPU pixel conversion, which is very slow.  melonJS.js:17370:17
Error: WebGL: texImage2D: Chosen format/type incurred an expensive reformat: 0x1908/0x1401  melonJS.js:17370:17

which leads to the two following lines :
https://github.com/melonjs/melonJS/blob/master/src/video/webgl/shader.js#L189
https://github.com/melonjs/melonJS/blob/master/src/video/webgl/shader.js#L192

@parasyte is this somehting you would maybe have the time/be willing to look at ?

Member

obiot commented Dec 5, 2016

now that the WebGL renderer is enabled by default with the platformer, I noticed that Firefox is also complaining a bit about it :

Error: WebGL: texImage2D: Incurred CPU pixel conversion, which is very slow.  melonJS.js:17367:17
Error: WebGL: texImage2D: Chosen format/type incurred an expensive reformat: 0x1908/0x1401  melonJS.js:17367:17
Error: WebGL: texImage2D: Incurred CPU-side conversion, which is very slow.  melonJS.js:17370:17
Error: WebGL: texImage2D: Incurred CPU pixel conversion, which is very slow.  melonJS.js:17370:17
Error: WebGL: texImage2D: Chosen format/type incurred an expensive reformat: 0x1908/0x1401  melonJS.js:17370:17

which leads to the two following lines :
https://github.com/melonjs/melonJS/blob/master/src/video/webgl/shader.js#L189
https://github.com/melonjs/melonJS/blob/master/src/video/webgl/shader.js#L192

@parasyte is this somehting you would maybe have the time/be willing to look at ?

@obiot

This comment has been minimized.

Show comment
Hide comment
@obiot

obiot Dec 5, 2016

Member

google'ing it, it seems like linked to a known issue with FF :
https://bugzilla.mozilla.org/show_bug.cgi?id=1246410

Member

obiot commented Dec 5, 2016

google'ing it, it seems like linked to a known issue with FF :
https://bugzilla.mozilla.org/show_bug.cgi?id=1246410

@parasyte

This comment has been minimized.

Show comment
Hide comment
@parasyte

parasyte Dec 5, 2016

Member

I'm not sure the WebGL renderer is fully "ready" to be default. It still has a ton of work needed.

Member

parasyte commented Dec 5, 2016

I'm not sure the WebGL renderer is fully "ready" to be default. It still has a ton of work needed.

@obiot

This comment has been minimized.

Show comment
Hide comment
@obiot

obiot Dec 5, 2016

Member

it is not, only in the platformer example

Member

obiot commented Dec 5, 2016

it is not, only in the platformer example

@parasyte

This comment has been minimized.

Show comment
Hide comment
@parasyte

parasyte Dec 5, 2016

Member

Ah, I see. It's for the website!

Member

parasyte commented Dec 5, 2016

Ah, I see. It's for the website!

@obiot obiot added this to To Do in Roadmap Dec 4, 2017

@obiot obiot changed the title from Reconcile the global matrix to [WebGL] Reconcile the global matrix Dec 13, 2017

@obiot obiot moved this from New Features (Core) to WebGL in Roadmap Dec 13, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment