New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for USD and USDZ formats #14219

Open
HenryB96 opened this Issue Jun 4, 2018 · 9 comments

Comments

Projects
None yet
7 participants
@HenryB96
Copy link

HenryB96 commented Jun 4, 2018

It would be great to get Three.js support for the Universal Scene Description (USD) file format and the upcoming USDZ format announced at WWDC today.

USDZ specification document: https://graphics.pixar.com/usd/files/USDZFileFormatSpecification.pdf (I assume that more details will come out in the coming months for this format)

@Mugen87

This comment has been minimized.

Copy link
Collaborator

Mugen87 commented Jun 5, 2018

It's a valid feature request but I personally don't understand Apple/Pixar. Why are they introducing a new file format and not just using glTF? Can somebody explain what USD /USDZ does better than glTF?

@Usnul

This comment has been minimized.

Copy link
Contributor

Usnul commented Jun 6, 2018

USD documentation: http://graphics.pixar.com/usd/docs/index.html

It's a valid feature request but I personally don't understand Apple/Pixar. Why are they introducing a new file format and not just using glTF? Can somebody explain what USD/USDZ does better than glTF?

I agree, part of it is politics me thinks. However, that's a separate discussion :)

@Mugen87

This comment has been minimized.

Copy link
Collaborator

Mugen87 commented Jun 6, 2018

Right now, I personally do not vote for a new loader. I hope for a USDZ->glTF converter...

@donmccurdy

This comment has been minimized.

Copy link
Collaborator

donmccurdy commented Jun 6, 2018

USD has been around for a couple years — it's commonly used in film, and not an obvious choice for runtime transmission. As far as I can tell USDZ is a wrapper around that. Apple hasn't given much detail at this point; maybe they have some plan to make it more runtime-appropriate, or maybe the messaging at WWDC was just unclear with respect to how they're actually using it. In any case I think we probably need to wait for more information.

If someone is interested in contributing a THREE.USDZLoader, then by all means. 🙂 But all of the documentation I've found so far suggests the format is very closely tied to the official (C++/Python) libraries for parsing it, similar to FBX and the FBX SDK. So unless those can be made web-friendly with WASM, implementing USDZ support in three.js and keeping up with changes in the format over time looks like a very time-consuming process, without the benefit of those libraries.

@mrdoob mrdoob referenced this issue Jun 16, 2018

Open

Loaders #5524

28 of 43 tasks complete
@richardmonette

This comment has been minimized.

Copy link
Contributor

richardmonette commented Jul 4, 2018

Out of curiosity, I tried opening the "Retro TV" file from https://developer.apple.com/arkit/gallery/ as if it was a simple zip. I 'unarchived' the .usdz by renaming it's extension to .zip, and then having my OS do its usual open procedure. Presumably, something similar could be achieved with https://gildas-lormeau.github.io/zip.js/

This is what I see when I 'unarchive' the usdz:

screen shot 2018-07-04 at 2 17 46 pm

So, the textures are just PNGs, which are probably referenced from within that .usdc, which is the next issue. According to https://graphics.pixar.com/usd/docs/Converting-Between-Layer-Formats.html, ".usdc files – binary format" which is in Crate (see Crate File Format in https://graphics.pixar.com/usd/docs/USD-Glossary.html) format.

There is also .usda, which is ASCII encoded, which would presumably be easier to write a loader around.

As a rough outline of some to-dos for this, minimally a Loader would be looking at extracting a Zip file in JS, and then also understanding the .usda (and potentially binary Crate .usdc as well.)

@TimvanScherpenzeel

This comment has been minimized.

Copy link

TimvanScherpenzeel commented Jul 4, 2018

Hi @richardmonette, I've done some research on USDZ and the USDA format and have written a proof-of-concept glTF to USDZ converter. Perhaps it is useful for your research: https://github.com/timvanscherpenzeel/gltf-to-usdz. You can see a demo here: https://twitter.com/domenicopanacea/status/1011292393801420800 or try it out yourself here: https://timvanscherpenzeel.github.io/gltf-to-usdz/ (requires iOS 12).

In short: geometry gets converted to OBJ, the metallic-roughness texture is split by channel and textures are Y-flipped. From this a USDA file is constructed dynamically and fed into usdz_converter to construct the final usdz.

I know some people are looking into writing a direct converter to USDZ but not a whole lot is known about the USDC format at this point in time (also in terms of decoding).

@richardmonette

This comment has been minimized.

Copy link
Contributor

richardmonette commented Jul 5, 2018

Hey @TimvanScherpenzeel, thanks for your message and work on this!

Have you considered using the USD Python API (https://graphics.pixar.com/usd/docs/Hello-World---Creating-Your-First-USD-Stage.html)? I wondered if it might be possible to skip the .obj stage and, using Python, read the glTF and go straight into .usda/c that way?

@TimvanScherpenzeel

This comment has been minimized.

Copy link

TimvanScherpenzeel commented Jul 5, 2018

I've had a lot of issues setting up USD and have not yet taken the time to fix the issue (I think it has something to do with having multiple versions of Python installed and during compilation it links to the wrong Python).

You could indeed write the OBJ file directly into the USDA file (that could actually be a good next step in my proof of concept). I know the file structure of the AR Gallery examples because a contributor to my project send me the USDA files after having the USDC files decoded it with usdcat.

@PixelPartner

This comment has been minimized.

Copy link

PixelPartner commented Jul 13, 2018

The whole point of Apple providing a USDZ QuickLook is to totally bypass using WebGL, DOM and Javascript and directly use the scene with ARkit2.0.
It's just a milestone to an rOS based on Apple/Pixar tech.
When they later add interactivity/scriptability, even less Web-UI will be needed.

Pixar USD, using highly efficient OpenSubDiv Patches for geometry, ported to mobile Metal will soon outperform HighEnd PC-VR WebVR.
Not as a file format, but as a whole Experience-Making/Presenting/(-Interaction) pipeline.

So IMHO it's more important to export USDZ from AR/VR Editors and for Android and Windows to catch the USD train, than to add an importer to the WebGL2 pipeline.

An important issue is that WebXR would have to also support SubDiv patches instead of legacy polymeshes to fully support USD as used by Apple.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment