Declarative XR Language #44
Comments
|
I think most people in the group wanted to see this as something that is implemented with web components and not by extending the HTML language. |
|
Has anyone started reviewing existing declarative specs? Things that pop to mind include
@rcabanier I think we all would agree that we don't necessarily need to official extend HTML; in fact, it's possible a future XR "browser" might want to read and display declarative XR even in the absence of the DOM. One could even imagine current systems (things like Sumerian) some day providing support to read and display Declarative XR content during authoring, or on the fly. The Important part is that there is a well defined language, such that content created for it can reasonably be expected to work in many places. A DOM-based implementation is clearly a great starting point. (Disclaimer: I worked on ARML 2.0 and a related KARML variation of KML before that) |
|
I totally agree. I want to make sure that we're all on the same page on where the work needs to happen. |
|
One of the problems I've had with AFrame is that it drops you off a cliff once you start creating custom components for more complex use cases. You have to learn all of A-Frame, all of the ThreeJS, and all of how they interact. This wouldn't be as much of a problem if A-Frame's component model was shared by ThreeJS (or vice versa). Then you could drop to Three without losing the nice built-in AFrame components. And you could use some of the great ThreeJS components within AFrame. Any new declarative system we come up with will have a similar challenge. I don't see the syntax as the problem. It's finding a shared component model that is the challenge. |
|
@blairmacintyre: I can comment on X3D. The working group that is overseeing development of X3D is not working on compatibility with DOM. There are several things in specific that they are not addressing. I can go into more detail, but I don't think it is the right time. I agree with @joshmarinacci that A-Frame is not easy. For pretty much anything beyond the basics you need to write custom code (or find someone else that you trust to do it). I started working on XSeen to see what can be done. |
|
I would agree that neither X3D nor AFrame are good, as is. But they do represent existing work. At some level, the issue with AFrame is that it relies on extensibility (and thus, requires it, and is incredibly fragile), and that systems like X3D are attempting to do enough that they don't need extensibility (making it huge and heavy). What I think the most valuable part of declarative XR, like HTML, is that it can reasonably be executed everywhere and content created with it can be portable, and (more importantly) long lived. The question, then, is what level of abstraction do we want to support? One possibility is to get all interaction out of the declarative XR, and focus on scenes and a Javascript API to interact with it. |
|
Once tactic that's possible with no new UA tech would be to create a library of custom elements that directly mirror the node types in a Three.js* scene graph (Scene, Group, Camera, etc). Then the library could update Three's scene with each mutation of the elements. That provides current HTML writers the smooth on-ramp that is so valuable from a-frame, eliminates the mismatch between elements and Three.js that causes confusion in a-frame, and prevents rat-holing on how to build logic into the declaration syntax. The answer to the question of how to do anything dynamic is to use Javascript to interact with the elements (and thus the scene) as it is with HTML.
(On a second read, this may be exactly what Blair suggested in their latest comment) |
|
I wouldn’t think we want to implement declarative 3D in code, necessarily, so figuring out what the declarative language is, and what the purpose is, is key. For example, I’ve got a student playing around with using OGC’s ARML2.0 as a way to encapsulate streaming POI’s (ARML 2.0 supports attaching POI’s to both geospatial and feature-based “anchors”, and could be expanded in interesting ways). Using some standard formats (glTF, SVG, etc) for the 2D/3D elements inside the POI makes much of this “declarative”. We’re implementing it inside a JavaScript/WebXR page, but the long term idea is that some of this might end up being pushed into different parts of the system. But, ARML is pretty weak in terms of “interesting” content. A simple scene graph that focuses on the scene structure, and uses standard elements for the leaf nodes (glTFs, for example), could be implemented in an ECS on top of three.js (something more well defined that AFrame, so it can be stable and well-defined) or (again, if desired) natively. This might allow even more generality than ARML2.0, but also has limitations. In both cases, we aren’t going to be able to implement sophisticated apps without having a very complex declarative language. If we require that there is non-trivial JS, then we will need to also have a well defined JS interface. Think about HTML: simple pages require very little or no JS, and complex pages only work because the interfaces are well defined so they can be implemented everywhere. If there is no reasonable expectation on the part of a content creator that the declarative language will work everywhere, and is going to be long lived, then it’s of limited value. I wonder if it’s worth pulling in more people from the 3dformats working group (might be a Khronos thing?) to see what their thoughts are? |
@blairmacintyre This is how Magic Leap implemented Prismatic. |
@blairmacintyre: I asked Neil Trevett about this multiple times over many years. He stated that Khronos was not in the business of defining a declarative XR language. They were operating below that level with models (glTF) and lower. That said, there are a lot of people in Khronos interested in the topic. The Web3D Consortium is in that business, but the current version of X3D and their publicly released info for the next version do not seem to align/integrate completely with DOM. @rcabanier & @blairmacintyre That is also how I developed and defined XSeen. I tried to extend HTML concepts to XSeen. Static stuff goes in nodes. Fully dynamic stuff goes into JavaScript. Things in the middle (like some animation) goes into nodes, others are left for JavaScript. |
|
May I suggest that we also draw inspiration from prior art in native engines (eg, Unity 3d, Unreal, Xenko, IDTech..), as they have evolved to meet challenges set by shipping realtime experiences. In particular, any discussion on realtime rendering would not be complete without considering runtime formats for statically "baked" calculations and combining them with dynamic / run-time rendering, physics, and audio. Essential assets include GI probes (SH ambient light probes, reflection probes), acceleration structures (BVH, octrees, signed distance fields), texture compression, ambient occlusion, baked diffuse lighting, procedural nav meshes, and hierarchical LOD models. Most of these "baked" calculations apply only to objects marked as "static". If all objects are "dynamic", the scale and quality of environments that can be created is greatly limited, even on modern hardware. |
I should also be noted that there are emerging, but not yet widely accessible, hardware capabilities such as DXR realtime raytracing that will eventually enable more assets to be "dynamic" and rely less on baked calculations applied to "static" objects. For the near-term future; however, such capabilities are still limited to being applied judiciously for objects that must be dynamic or for post-processing passes (eg, replacing screen-space raytracing). It will be several years before we can expect average web users to have the capability of rendering with "hack-free" full-scene raytracing or GI. |
|
@kearwood In reference to LODs, do you mean that the language should handle creation and display of LODs, or just that the language should provide the facilities to display LOD of the provided model? |
The language should be able to describe how objects are changed at various distances such as swapping models and being combined with others (hierarchical LOD). There are variations on how such distances are determined, such as by distance to origin of the model and degrees of coverage within the view frustum for AABBs. The language may also need to describe how the artist intends materials and baked lighting to be rendered in LODs. To assist in creating of these structures, the language could win by being easy to parse by offline tool chains. |
|
I describe one way to express hierarchical LOD as part of the scene graph node structure, with minimal attributes and a simple processing model: https://kearwood.com/content/hierarchical-lod Such a thing would apply primarily to static geometry or larger dynamic objects with many sub-parts. Simple dynamic objects (eg, an avatar model) would only need LOD meshes to swap at the model level, potentially embedded in the same GLTF file. |
Some of these may be reasonable low-level additions directly to glTF – in particular ambient occlusion (supported), texture compression (in progress), and LODs (proposed). Others, like navigation meshes and perhaps collision volumes, I'd consider too high-level for glTF and better suited for a declarative and composable language like this. |
At the January F2F meeting we held an unconference session on declarative languages for XR. This would be an HTML-like language that would reside in the web page that would present to the page developer a declarative mechanism for defining and interacting with XR environments. This language would support web page designers and developers who didn't want to build an XR environment from the API provided by the Immersive Web specification.
The discussion focused on the capabilities that were present in existing languages, what people might want to see in such a language, and potential issues beyond those covered by the IW specification (e.g., security, standardization, etc.).
The take-away from the meeting was to create this issue in the Proposals Repository and notify the public list that it is present and available for discussion. Future direction will be determined by participation.
Please post use cases as comments to this issue. At this time it is way to early to discuss implementations from this conversation; however, discussion of existing implementations is encouraged.
The text was updated successfully, but these errors were encountered: