Add lazy option to GLTFLoader.

[robertlong]

This PR adds the ability to lazily load or load parts of a glTF model. This can also be used to modify the glTF before calling parser.parse(). We intend to implement the MOZ_alt_materials extension in this way (see #14470).

This was previously discussed in: #11682

[donmccurdy]

Would you want to also return the json object here? I imagine users will want to use that to access extensions data, or to list the available materials.

[robertlong]

The json object is available via parser.json. I think this lines up well with the non-lazy method.

When specifying lazy:

onLoad( { parser: parser } )

When using the default non-lazy method:

onLoad( {
  scene: scene,
  scenes: scenes,
  cameras: cameras,
  animations: animations,
  asset: json.asset,
  parser: parser,
  userData: {}
} )

[donmccurdy]

There is also some pre-processing in parser.parse() that still needs to happen here — specifically parser.markDefs().

[takahirox]

Doesn't this work for MOZ_alt_materials even without lazy?

var fileLoader = new THREE.FileLoader();
var gltfLoader = new THREE.GLTFLoader();

fileLoader.setResponseType( 'json' ).load( url, function ( json ) {

    var materialDefs = json.materials;
    var altMaterialDefs = [];

    for ( var i = 0, il = materialDefs.length; i < il; i ++ ) {

        var materialDef = materialDefs[ i ];
        var materialIndex = i;

        if ( preferredTechnique !== undefined &&
            materialDef.extensions &&
            materialDef.extensions[ 'MOZ_alt_materials' ] &&
            materialDef.extensions[ 'MOZ_alt_materials' ][ preferredTechnique ] !== undefined ) {

            materialIndex = materialDef.extensions[ 'MOZ_alt_materials' ][ preferredTechnique ];

        }

        altMaterialDefs.push( materialDefs[ materialIndex ] );

    }

    json.materials = altMaterialDefs;

    gltfLoader.parse( json, THREE.LoaderUtils.extractUrlBase( url ), function ( gltf ) {

        scene.add( gltf.scene );

    } );

} );

Update: Oops, GLTFLoader.parse() doesn't accept json object now then we need to enable .parse() to accept json object.

Update2: Oops, we need to expose a method converting from binary to json for binary glTF.

[donmccurdy]

This extension could perhaps be implemented without the lazy option, yes, but per #11682 I think this is a good change all the same. We don't want to add too many vendor extensions into the loader, and having some basic hooks like this makes it easier for others to do custom things.

[robertlong]

I agree, I think the lazy option makes implementing this extension easier to implement with very minimal changes.

Another example of something we will use this for in Hubs:
https://github.com/mozilla/hubs/blob/master/src/vendor/GLTFLoader.js#L43

We need to rewrite URLs for our file proxy using an asynchronous API after downloading the glTF json but before calling parse(). We could use the method you specified above, but as you mentioned we would also need to provide methods to continue to support glb files.

[takahirox]

First, I agree with having hook points for handling user-defined extensions. My concern for .lazy is I guess it'd be a bit tricky for users...? My suggestion would be more friendly with Three.js users. (Yeah, there's a room for discussion.)

Before we go forward for further discussion, I have one question. How do you expect user uses parser before parser.parse() calling except for overriding json? I meant, is there any usecases where lazy can do while my suggestion (+ some fixes) can't?

[donmccurdy]

In addition to the case of implementing extensions, note that the glTF spec says:

The glTF asset contains zero or more scenes ... When scene is undefined, runtime is not required to render anything at load time.

Implementation Note: This allows applications to use glTF assets as libraries of individual entities such as materials or meshes.

For users who want to do this, manually downloading the JSON file and then "parsing" the entire thing doesn't seem like the right API. Even if you already had the JSON downloaded, I think you'd want a lazy option — or something like it — to access dependencies one by one, preferably without needing to know much about the structure of a glTF file.

Or maybe the file contains multiple scenes, one for each zone in the world, and you only want to download each zone as the user reaches it. This is preferable to having separate .gltf files for each zone, because materials and textures will automatically be recycled by the parser.

Out of scope for this PR, but we could make this even more user-friendly with a simple helper method that does lookups by name instead of by index: getDependencyByName( 'mesh', 'castle_tower' ).

---

tl;dr — I do think having a convenient way of getting dependencies out of a glTF file one by one is a good thing. But if there are better APIs for it than a .lazy option, I'm totally open to that. Maybe something like:

• loader.getParser( 'foo.gltf', ( parser ) => { ... } )
• parser = loader.getParser( json | arraybuffer )

[robertlong]

I agree with @donmccurdy, being able to get dependencies out of a glTF file is important and one of the nice things about this API.

I'd like to see the ability to modify the JSON after loading a GLTF/GLB's JSON. Ideally the loader handles both formats and hands me a JSON object so I don't have to check the response content type and magic integer and call parseGLB.

I'd also like to have the ability to pull images, bufferViews, specific nodes, etc out of the glTF asset. This will be useful for packing additional assets into a glTF file.

However, if there's a better API than .parse I'm also open to suggestions.

[takahirox]

Agreed with providing APIs to access the dependencies. Wondering which one do custom-extension developers want before parsing entire glTF, dependent Three.js object or dependent json content. I suppose json content rather than Three.js object?

I think one of the major use cases where to handle custom-extension before parsing entire glTF is to prevent downloading/loading/parsing unnecessary assets. For example, material's custom extension chooses lower resolution one for mobile use. In that case with parser's .getDependency, you get material's custom extension with var material = parser.getDependency( 'material', 0 ); material.userData.gltfExtensions...; but default textures for the material are already loaded inside. If user can get dependent json content, user can override textureIndex without downloading the default texture and then parse entire json.

[takahirox]

@robertlong

I'd also like to have the ability to pull images, bufferViews, specific nodes, etc out of the glTF asset. This will be useful for packing additional assets into a glTF file.

Adding assets? Can you give me some examples?

[donmccurdy]

In that case with parser's .getDependency, you get material's custom extension with var material = parser.getDependency( 'material', 0 ); material.userData.gltfExtensions...; but default textures for the material are already loaded inside.

That's OK I think — the material and its dependencies should be fully loaded, just as long as we're only loading the textures needed for that material and no more. There are a few places in GLTFLoader where we use getDependencies (plural), and we should try to remove those, but that can be another PR.

[takahirox]

I think one of the major use cases where to handle custom-extension before parsing entire glTF is to prevent downloading/loading/parsing unnecessary assets. For example, material's custom extension chooses lower resolution one for mobile use.

I meant, for example, a glTF material refers to large size textures and the material's custom extension refers to small size textures for mobile use. For mobile use, user doesn't want to download the large textures and instead wants to download the small textures. But var material = parser.getDependency( 'material', materialIndex ); loads the large textures because GLTFLoader doesn't know the custom extension and returns material after solving the dependencies.

I think the purpose of providing a hook point to custom extension developers before parsing entire glTF is providing a way to prevent unnecessary downloading/parsing data. But I don't think .getDependency/ies satisfies that demand, so I suppose what custom extension developers wants before parsing entire glTF is dependent json contents, they can override json.

[donmccurdy]

I see two use cases here:

1. Modify the JSON before it is parsed. Users already can do that by fetching the JSON first, as you mentioned earlier, modifying it, then calling parse(). Alt materials fits in this case.
2. Load unstructured or incremental data out of the asset, which isn't necessarily replacing anything. For example, a .gltf asset might contain just a list of materials. Or the application might load additional meshes and textures as they are needed during an experience.

Technically both cases work with lazy, because you can access the JSON via parser.json. I don't think (1) is as important, because it can already be accomplished by pre-loading JSON or by simply creating different (or dynamically generated) .gltf JSON files that reference some shared buffers and textures.

On the other hand, (2) is not possible without the lazy option, unless you do things like constructing dummy meshes to extract your materials from the "material library" asset, and that seems hacky. I think this lazy option is a useful thing to have, although it will need some additional refinement in the loader (to avoid fetching all available buffers at various times) to make it flexible.

[takahirox]

GLTFLoader.load/parse() returns parser as argument of onLoad callback (it's as temporal workaround tho). Can't it solve (2)?

[donmccurdy]

Except that all available scenes will be loaded eagerly. The content author would need to be sure nothing is referenced by any scene, or that there are no scenes in the asset at all. For most users I think that would involve hand edits to the file, so an API for lazy-loading does still seem simpler.

[takahirox]

Now I have some ideas/thoughts. Let me share after I'm back to home next week.

[takahirox]

Sorry I'm late.

I suggest to add .getParser() method to GLTFLoader (as experimental) and remove parser from onLoad's parameter of GLTFLoader's .parse(), rather than adding .lazy option.

I'll write the reason this weekend, time to sleep now.

[takahirox]

I think there're three use cases in glTF loader.

1. Load available scenes in glTF file
2. Lazily load unstructure/incremental (part of) glTF file
3. Process custom extension and/or some their own special processing (and then 1. or 2.)

For 1., users use .load(). I think 1. is the most popular use case and they don't need parser. Returning parser is unnecessarily exposing inside too much for this use case. It'd be better to remove parser from onLoad parameters.

And IIRC returning parser via onLoad is just a temporal workaround for custom extension handling in user side. I don't think we should add an option presupposing returning parser via onLoad.

For 2. and 3., users can use parser via .getParser() and process their own works as you suggested.

Parser is the core of GLTFLoader inside. We could still refactor and/or change APIs. Then I want .getParser() to be as experimental so far.

I speculate one of the reasons why you suggested .lazy flag is for small change inside. But I realized we still need a lot change including #14779. IMO we don't need to think about small change yet for now.

[donmccurdy]

And IIRC returning parser via onLoad is just a temporal workaround for custom extension handling in user side.

This use case is not temporary — users should be able to implement custom extensions without hacking the loader, at least for basic use cases.

How do you suggest getParser working? Could you write an example of how someone would use it? A GLTFLoader creates a new parser with every .load() call, you need to have loaded the JSON portion of the file to even create a parser... something like this?

var loader = new THREE.GLTFLoader();
loader.setDRACOLoader( dracoLoader );
loader.createParser( 'foo.gltf', async function ( parser ) { 

  var mesh = await parser.getDependency( 'mesh', 0 );

}, undefined, function ( e ) {

  console.error( e );

} )

[takahirox]

This use case is not temporary — users should be able to implement custom extensions without hacking the loader, at least for basic use cases.

I agree with providing a way to process their custom extension in user side without hacking (I specified it as 3. above) I meant, returning parser via onLoad is temporal workaround until we find better solutions.

How do you suggest getParser working? Could you write an example of how someone would use it? A GLTFLoader creates a new parser with every .load() call, you need to have loaded the JSON portion of the file to even create a parser... something like this?

Yes, something like that. I meant .getParser() as in #14492 (comment)

[donmccurdy]

With #14779 merged I think it would be a reasonable time to revisit this, if you are still interested. I'm still not sure of the right API here though.

[takahirox]

I still prefer separating parser return from .load(). Making .getParser() PR and I think I can share in this year.

[robertlong]

I think we can close this PR. I'm in favor in the .createParser() API instead.