Draco mesh/point compression extension for glTF

[pjcozzi]

See KhronosGroup/glTF#874

@fanzhanggoogle if you are able to contribute this or start a prototype, it would be very much appreciated. Here's some tips:

• glTF loader/renderer is here: https://github.com/AnalyticalGraphicsInc/cesium/blob/master/Source/Scene/Model.js
• Unit tests are here: https://github.com/AnalyticalGraphicsInc/cesium/blob/master/Specs/Scene/ModelSpec.js
• glTF example app is here: https://github.com/AnalyticalGraphicsInc/cesium/blob/master/Apps/Sandcastle/gallery/3D%20Models.html
• General contributor guide is here: https://github.com/AnalyticalGraphicsInc/cesium/blob/master/CONTRIBUTING.md#opening-a-pull-request

Let us know if you need more help.

Part of #3241, #4678, and CesiumGS/3d-tiles#188

[pjcozzi]

Also, instead of branching off of master, use this branch that is being updated to support glTF 2.0 (and still 1.0 by converting to 2.0 on the fly): #4808

[fanzhanggoogle]

That sounds a good plan to validate Draco extension and a good way to promo it :)
I'm working COLLADATOGLTF2.0 + Draco extension now. Will start this after that's done.

[pjcozzi]

Sounds great @fanzhanggoogle! Ping @lasalvavida if you have any questions about the COLLADA2GLTF code.

[pjcozzi]

Draco sample models: KhronosGroup/glTF-Sample-Models#116

[pjcozzi]

@lilleyse can you please connect with @AnimatedRNG to come up with a detailed Draco roadmap and to show him the relevant parts of Cesium's glTF loading code? High-level roadmap is something like:

• Implement glTF Draco extension loading in Cesium using existing sample models (Add support for Draco glTF models #6191)
• Profile. Move Draco decompression to a web worker. Profile. Maybe improve web worker architecture. Profile. Unified time slicing tasks #2655, Improve asynchronous Primitive creation #1487
• Update some sample models used in Sandcastle to be Draco compressed
• Tweaks to 3D Tiles' b3dm and i3dm tile format specs for Draco extension. Add unit tests; 3D Tiles with Draco should just work once Cesium's glTF loader supports it
• Add Draco to 3D Tiles pnts tile format: update the Cesium loader and spec
• Perhaps help finish Draco in gltf-pipeline: https://github.com/fanzhanggoogle/gltf-pipeline/tree/2.0_mesh_compression
• To save GPU memory, evaluate recompressing with (or with something like) WEB3D_quantized_attributes and oct-encoded normals and textures
• Potentially update cesiumjs.org/convertmodel.html
• Potentially contribute to 3D Tiling Pipeline

Please open incremental pull requests into the draco-master branch.

Resources

• glTF extension spec: KHR_draco_mesh_compression KhronosGroup/glTF#874
• glTF sample models using Draco compression: Added sample models for Draco_mesh_compression_extension KhronosGroup/glTF-Sample-Models#116
• BabylonJS glTF Draco loader by @bghgary: babylon.dracoCompression.ts and KHR_draco_mesh_compression.ts
• Three.js glTF Draco loader by @ondys and @fanzhanggoogle: DRACOLoader.js
• Draco repo (use version 1.1.0): https://github.com/google/draco
• 3D Tiles spec
• gltf-pipeline 2.0 branch with Draco compression by @fanzhanggoogle: https://github.com/fanzhanggoogle/gltf-pipeline/tree/2.0_mesh_compression

[lilleyse]

@AnimatedRNG - here's some notes about glTF and some ideas for getting Draco support into Cesium:

The general structure for a glTF is:

scene -> node -> mesh -> primitive -> attributes -> accessors -> buffer view -> buffer

A scene contains a list of nodes. A node contain a model matrix and a reference to a mesh. A mesh contains a list of primitives, where each primitive is defined by a material, vertex attributes, and indices. Each attribute points to an accessor, which is an object that defines how to interpret buffer data for that vertex attribute (maps to the concept of glVertexAttribPointer). The accessor points to a buffer view which is some range within a buffer.

In Cesium, a DrawCommand is created for each primitive.

The Draco extension is also defined per-primitive. When the extension is present, ignore any data that might be pointed to by the primitive's attributes and indices. Instead the attribute and indices are contained within a chunk of data referenced in the extension's bufferView property. After decoding the compressed chunk we can get the final vertex attributes and indices for the mesh.

I think the easiest approach for getting this into Cesium will be:

Add a function after: processPbrMetallicRoughness(this.gltf, options); called parseDraco(this.gltf)

That function should check if the extension exists. If it does, loop over each mesh and each of its primitives (use ForEach.mesh and ForEach.meshPrimitive). When the extension is encountered in a primitive, run the third party Draco decoder on the bufferView. It will probably want a typed array of the data, so give it gltf.bufferViews[bufferView].extras._pipeline.source (this is where gltf-pipeline will have stored a typed array of the buffer view's contents). The decoder will probably return a typed array containing the decompressed index and vertex data. We are now done with the compressed bufferView, so it would be good to delete it since we don't want it parsed in parseBufferViews. Just remove the buffer view from the bufferViews section of the glTF.

Now edit the glTF so that everything points to the decoded data correctly. First add a new buffer for the decoded data to gltf.buffers. Set the buffer's extras._pipeline.source to the decoded data (this is a bit unclean, but it is the best approach so that addBuffersToLoadResources right below works). Then add a bufferView for each of the the decoded vertex attributes and for the decoded indices. Each of these will point to the buffer that was just created. Now for every accessor referenced by the primitive that used the Draco extension, set its bufferView property to the new buffer view created for it. With all this in place Cesium should be able to load everything correctly.

This approach could also be its own self-contained stage in gltf-pipeline. But for now keeping the code in Cesium is fine.

[pjcozzi]

Add to #5120 (comment):

• To save GPU memory, evaluate recompressing with (or with something like) WEB3D_quantized_attributes and oct-encoded normals and textures

[pjcozzi]

@lilleyse as you sure about your approach? It makes sense for gltf-pipeline, but we should not make a habit of doing a lot of glTF JSON object modification as part of our runtime loading. Instead, we should strive to get data into Cesium's data structures - even intermediate data structures that are used to aid loading - and then process. Don't you agree? Is that hard in this case? If so, why?

[lilleyse]

I see your point. It shouldn't be too hard to use intermediate data structures instead.

We are now done with the compressed bufferView, so it would be good to delete it since we don't want it parsed in parseBufferViews. Just remove the buffer view from the bufferViews section of the glTF.

After looking closer I noticed this step isn't needed. The draco buffer view target will not be marked as ARRAY_BUFFER so it will not be uploaded to the GPU like I thought. So ignore this.

Now edit the glTF so that everything points to the decoded data correctly.

Instead of this approach, we can probably circumvent most of the gltf asynchronous loading code and just create vertex and index buffers directly from the decoded data. Look at the createVertexBuffer/createIndexBuffer function. You will probably want something similar that takes a typedArray instead of a bufferViewId. The buffers should still be added to model._rendererResources.buffers, its key being Object.keys(model._rendererResources.buffers).length just so it doesn't overwrite anything. (Side note: buffers should probably be an array instead of an object, this seems like an area that wasn't changed during the 1.0 - 2.0 transition)

The last step is making sure createVertexArrays works correctly. If an accessor is one of the Draco accessors, it will not have a bufferView, byteOffset, or byteStride which are required. That information will need to be stored somewhere (not on the glTF like I had suggested) and fetched in createVertexArrays.

@AnimatedRNG let me know if anything is unclear. There may be some edge cases I haven't forseen.

[pjcozzi]

@ggetz @lilleyse here is the three.js implementation by @donmccurdy: donmccurdy/three.js#6

[pjcozzi]

@ggetz @lilleyse Draco sample models: KhronosGroup/glTF-Sample-Models#145

[ggetz]

Updated the task list. Extension loading in Cesium added in #6191

[pjcozzi]

@ggetz OK to close? Or is there anything left here: #5120 (comment)