[WebGL] Reconcile the global matrix

[parasyte]

This ticket will pick up where #591 leaves off. The goal here is to remove dependence on the global matrix, as much as possible. It will allow us to put almost every matrix transformation entirely within the GPU, where it can take advantage of highly parallelized vector ALUs that we don't have access to in JavaScript.

To start, recall that 3D rendering typically uses three matrices; called "model matrix", "view matrix", and "projection matrix". Working backwards, melonJS needs the projection matrix to transform our pixel coordinates to WebGL cube space coordinates using orthographic projection; and a view matrix is used for nested containers and other object that need relative drawing operations. The model matrix is unnecessary because the majority of entities drawn in melonJS are quads (exactly two triangles). A model matrix is more useful with large meshes, which have a greater number of vertices.

The scaling and rotation values can be streamed to the GPU as part of the vertex attribute array. These can be packed into 2 floats (a 3x3 matrix requires 6 floats for full precision) assuming 16 bits for each of the scaling coordinates. Additionally, the rotation angle and texture index can be packed into the same float. The anchor point will also be required, which is a 3rd float. These attributes can be applied within the vertex shader, since the order of operations is static; translation can be added to the destination coordinates before calling drawImage.

Entities that need a model matrix can implement one with a custom compositor and shader. Spine is an example that will benefit from a model matrix.

To reconcile the global matrix, we need to remove most of the matrix transformation usage. me.Sprite and me.Particle make heavy use of the global matrix right now. Here is the last proposal for a replacement: #591 (comment) The important part is that the first argument provides information about things like scaling and rotation for each drawImage call. The function signature will have to change to accept a texture region instead of coordinates, as an additional optimization. See: e785724#commitcomment-9329151

With that part done, each view matrix transformation will be treated as change in WebGL state; it will be sent to the GPU as a uniform variable, so changes to the matrix require flushing the compositor. This will be pretty efficient, assuming view matrix transformations only occur on me.Container boundaries.

Additional TODO items:

• Implement fonts (hard), lines (easy), stroke (hard)
• Create a packet inspector for debugging
• Enable mipmapping on power-of-two textures
• Pack multiple attributes into each float
• Use a standard sampler2D-uniform on mobile

Attribute Packing: E.g. texture coordinates can be packed into a single float, since each axis will never be bigger than 65,536 pixels (fits into 16-bits). The packing algorithms will provide less than 16-bits of precision, probably 15-bits, which is still plenty for texture coordinates. Another candidate is colors, and much bigger win: pack a vec4 into a single float with almost no loss in precision!

[parasyte]

I'll just add a note here: mobile platforms hate the fragment shader I wrote. The sampler2D array trick causes the GPU to run very poorly. (I've added a TODO item here to get this fixed.)

[obiot]

@krojew i see that you have been working yourself on a webgl library, any chance you want to help us here optimize/complete our webgl renderer ? We still have a lots of improvements listed in this ticket but as well #720 and #593.

@parasyte can also provides further explanation on our implemetation :)

[krojew]

@obiot I would be glad to help, but I have very limited time atm. I've not been able to even complete the typings yet.

But, I can give some quick tips for now from skimming over the description:

• Offloading work to gpu is generally a good idea, but you need to know which operations actually benefit from that. For example: it's usually much faster to create a buffer on the gpu for transformations, create a transformation matrix on client-side, map it to the buffer with proper invalidation for updates and do one multiplication in the VS, rather than pass individual transformation elements to the VS (which I understand this issue is about, but I can be mistaken). You also gain more flexibility that way, e.g. skewing. This assumes WebGL can map buffers - I've been involved in multiplatform engines more than WebGL, and I can't remember that.
• Unless you want to do distance-field font rendering, don't do that at all in WebGL. In fact, there's usually little reason to do that at all - use plain HTML. Fonts are a pain in the 3d world.
• Instead of packing multiple values to single floats to save space, use half float types (if available - again can't remember about WebGL). Unpacking takes shader time and you need to handle interpolation yourself. In short - usually not worth it. Best uses are packing TBNs (which can drop one vector) or quats (which can drop one dimension and are usually pushed to GPU in large quantities due to animation/physics), but I don't think those are the things you need to care about.

[obiot]

@parasyte about this warning messages in the console, what about adding a debug option flag for the renderer, and only display theses messages when the flag is on ?

[obiot]

now that the WebGL renderer is enabled by default with the platformer, I noticed that Firefox is also complaining a bit about it :

Error: WebGL: texImage2D: Incurred CPU pixel conversion, which is very slow.  melonJS.js:17367:17
Error: WebGL: texImage2D: Chosen format/type incurred an expensive reformat: 0x1908/0x1401  melonJS.js:17367:17
Error: WebGL: texImage2D: Incurred CPU-side conversion, which is very slow.  melonJS.js:17370:17
Error: WebGL: texImage2D: Incurred CPU pixel conversion, which is very slow.  melonJS.js:17370:17
Error: WebGL: texImage2D: Chosen format/type incurred an expensive reformat: 0x1908/0x1401  melonJS.js:17370:17

which leads to the two following lines :
https://github.com/melonjs/melonJS/blob/master/src/video/webgl/shader.js#L189
https://github.com/melonjs/melonJS/blob/master/src/video/webgl/shader.js#L192

@parasyte is this somehting you would maybe have the time/be willing to look at ?

[obiot]

google'ing it, it seems like linked to a known issue with FF :
https://bugzilla.mozilla.org/show_bug.cgi?id=1246410

[parasyte]

I'm not sure the WebGL renderer is fully "ready" to be default. It still has a ton of work needed.

[obiot]

it is not, only in the platformer example

[parasyte]

Ah, I see. It's for the website!

[obiot]

standard uniform sampler was implemented as part of #982

[obiot]

closing this one, lots of improvements were done during the last 2 majors version, at a point where we don't really need this ticket to track the here above left over points.