Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Geospatial data, Geolocation and WebXR #5

Closed
blairmacintyre opened this issue Jan 17, 2018 · 29 comments

Comments

@blairmacintyre
Copy link

commented Jan 17, 2018

What will the relationship between WebXR and geospatial data be.

It seems that WebXR cannot entirely ignore geospatial positioning, as geospatial content will be a major use case for mobile AR (at least eventually).

The web already has a geolocation API, but it is not sufficient for these purposes: it gives position but not orientation, is of very poor quality and not synchronized with the WebXR frame data. The deviceorientation API cannot be relied on for orientation: it is of very poor quality, was never standardizes (and is potentially going to be removed from existing browsers) and is also not synchronized with the WebXR frame data.

ARKit offers the option to have it's local coordinate system be aligned with geospatial orientation (e.g., Y up, Z south, X east). This provides a possible direction for how geospatial might be handled: have the WebXR API expose a property that says if the coordinate frame can be aligned with geospatial EUS coordinates, and provide a way for the developer to request this. Crude/simple geospatial positional alignment (between the user and the local coordinates) is easier, if you are guaranteed to have the local device coordinates aligned with EUS: each time a geospatial value is received from the local coordinates, an estimate of the geospatial location of the origin of the local coordinate frame can be updated (based on error values, etc). It won't be any better than the error of the geolocation API, but can be stable (because the local coordinates are used for locating and rendering content, not the very-slowly-changing geolocation values).

In theory, if a platform also provided better estimates of geoposition (somehow), it could simply provide those improved values through the geolocation API, benefiting both geospatial AR and other applications that use location. Permission to access location would always be done through the current web permissions approach.

@anssiko

This comment has been minimized.

Copy link

commented Jan 29, 2018

We're incubating a next generation Geolocation API called Geolocation Sensor (based on the Generic Sensor API) and would love to get your WebXR-inspired geospatial positioning use cases and requirements considered in the design of this new API.

@blairmacintyre, I opened an issue to capture your use cases and requirements: https://github.com/WICG/geolocation-sensor/issues/12 Please chime in :-)

The Geolocation Sensor joins the growing family of Generic Sensor-based APIs that bring consistency to and modernize the API surface of the web platform. We have two Geolocation Sensor polyfill implementations (example) and are planning to start Chromium experimentation in the near future after we've nailed down the feature set for the first experimentation. The spec is currently in incubation in W3C Web Incubator CG and is expected to advance to formal standards track in Device and Sensors WG in April 2018 (new charter).

@blairmacintyre

This comment has been minimized.

Copy link
Author

commented Jan 29, 2018

@anssiko I'll chime in over there, thanks!

@blairmacintyre

This comment has been minimized.

Copy link
Author

commented Mar 6, 2018

I would like to "bump" this issue again. As we start to think about requesting capabilities during session creation in WebXR, I would like to propose that we can request that the platform geoaligns the local coordinate system if it can. After the session is created, the app should be able to check if the coordinate frame is geoaligned.

Platforms that don't support geoaligning their coordinate frames can simply ignore the capability request, and signal that the frame isn't aligned.

@janerivi

This comment has been minimized.

Copy link

commented May 15, 2018

Hi I am the founder of the https://open-arcloud.org/ . The TL DR AR-cloud description is a 1:1 digital map/twin of the physical world stored in the cloud that enables a shared programmable space attached directly to our physical suroundings that enables multiuser AR and persistent and universally consistent placement of virtual assets in the real world

One of the things we hope can help bring this to reality is a standard definition of geographical position and orientation that can be understood across platforms and applications. What we call "GeoPose". Each AR-goggle, AR-smartphone or AR-content could have a GeoPose at any given moment.

Obtaining GeoPose of an XR-device could be achieved by matching sensor data from the device with the 1:1 map in the cloud through something like a "GeoPose" cloud service. Once the device has its GeoPose it can display geospatial data and assets that are anchored to a GeoPose.

A bit more about that here:
https://open-arcloud.org/standards

please join our slack channel and chime in on how you think a GeoPose should be defined.
https://join.slack.com/t/open-arcloud/shared_invite/enQtMzE4MTc0MTY2NjYwLWIyN2E4YmYxOTA4MWNkZmI5OGQ4Mjg2MGYzNTc4OTRkN2RjZGUxOTc4YjJhOTQ0Nzc3OWMxYTA3ZDMxNGEzMGE

My hope of course is that WebXR will support using device GeoPose and do the proper transforms for assets that are ancored to GeoPoses or assets that are described by geospatial coordinates.

@blairmacintyre

This comment has been minimized.

Copy link
Author

commented May 21, 2018

Thanks @janerivi. I agree that we need geospatial support.

I'm curious why you are trying to invent new standards. What's wrong with all the work being done in the geospatial community (XYZECEF, LLA, etc), and using approximations like WGS84 (for example)? It would seem if you create new ways of representing geospatial data, all existing data becomes problematic.

@janerivi

This comment has been minimized.

Copy link

commented Jun 14, 2018

@blairmacintyre I would really like this to be in sync with what the geospatial community is developing. I have no intention of creating a new standard if one is already in existance that meets the requirements.

However only the 3d-position Latitude, Longitude & Atlitude is not sufficient for AR, Robotics & self driving cars - the spatial orientation (yaw,pitch,roll or quaternion ) relative to a global coordinate system is also needed. That is what is the intention of the GeoPose proposal. If such a thing already exists I'd be happy to learn about it and see if it is functional for AR. If it is then I would be very happy to promote the use of that standard, if not I hope to reqruit help in developing the standard.

@boblaublaw

This comment has been minimized.

Copy link

commented Jun 15, 2018

Just chiming in with the result of some searching, but unfortunately I see no notion of orientation/pose in the following standards:

  • GeoJSon
  • OGC Simple Feature access. Part 2 of this standard formalizes the text and binary representations of these primitives, which makes standards-based SQL implementations possible.

Unclear to me if ISO 19107:2003 includes pose.

@blairmacintyre

This comment has been minimized.

Copy link
Author

commented Jun 15, 2018

@janerivi you are right, LLA is wholly inadequate, which is why geospatial systems don't generally use LLA, except to convert to other formats like ECEF.

If you take a look at cesiumjs.org (for example) you can see a full-fledged geospatial system. geospatial poses are represented internally as ECEF (FIXED in their terminology) matrices: essentially, meters relative to the center of the earth and using the principle axes and equator as the coordinate system. Converting to/from (many) other formats is relatively straightforward. (They also support an INERTIAL frame that is used to represent satellites around the earth)

The simplest way to manage geospatial orientation in an AR system is essentially what Apple has done with ARKit; orient the coordinate frame around some point on the earth, such that XYZ == EUS (east, up, south). This means the local coordinate system is a tangent plane to the earth (or, more likely, the WGS 84 approximation to the globe that is used in LLA).

The essence of my request is to do this: allow a programmer to request that the local coordinates are oriented to align with EUS at the origin of the local system. Once that is done, geospatial position is easy to do using the web APIs. If other systems (such as an AR Cloud like thing) is used by an application, aligning the coordinates out of the cloud with the AR system is straightforward.

I do not think that the web standards can possibly standardize on any given AR cloud system. I also have huge issues with the whole idea of a global AR cloud; it seems to be a horrific privacy nightmare. So, while a given application might want to use that, it probably isn't going to make sense to assume something like that in the web.

@boblaublaw

This comment has been minimized.

Copy link

commented Jun 15, 2018

I do not think that the web standards can possibly standardize on any given AR cloud system. I also have huge issues with the whole idea of a global AR cloud; it seems to be a horrific privacy nightmare. So, while a given application might want to use that, it probably isn't going to make sense to assume something like that in the web.

I can't speak for @janerivi, but as someone who has taken to using the term "AR Cloud" quite a bit lately, I can offer a suggested interpretation: "AR Cloud" is like regular "cloud" in that it describes a variety of protocols, services, organizations, etc. It's not a singular service, app, company or protocol - just a handy catchall for all new services and technology specific to providing interoperable AR. Viewed in this light, I think we must assume this will take place on the web.

@blairmacintyre

This comment has been minimized.

Copy link
Author

commented Jun 15, 2018

I’m all-in on cloud-based AR services; but I’ve never heard anyone use “AR Cloud” to refer to that; I’ve always heard it be used to refer to the “massive model of the world used to do localization, persistence, etc”.

@boblaublaw

This comment has been minimized.

Copy link

commented Jun 15, 2018

You're not crazy, people are certainly focused on relocalization, but I think that has more to do with the immediate availability of some apparent solutions there than a holistic view of what is needed. It isn't like the name "cloud" refers to localization solutions. (In fact, it does refer broadly to global internet services.)

I completely agree that not enough attention is being paid to the privacy and security implications surrounding AR. It is my hope that airing these issues in the same venues will attract interested parties to the problem.

@janerivi

This comment has been minimized.

Copy link

commented Jun 15, 2018

@janerivi

This comment has been minimized.

Copy link

commented Jun 15, 2018

@blairmacintyre

This comment has been minimized.

Copy link
Author

commented Aug 28, 2018

I'd like to revisit this issue and move it to a repository to discuss. A more specific proposal is this.

In recent discussions of session creation and flow immersive-web/webxr#330 there is a notion of requesting features on a session. Assuming a flow like that will be exposed, I would like to propose we add a geoAlignment feature:

xrSession.requestGeoAlignment()

This feature would request that the underlying session aligns it's coordinate system such that XYZ = EastUpSouth (a standard geospatial frame of reference).

Devices that can do this would succeed, others would fail.

There are two reasons to fail:

  • the underlying platform doesn't support this, and the browser is unable to simulate it. ARKit supports this directly for example, so supporting it on such a platform is as simple as setting a flag. Other platforms could use the built in motion APIs to determine an offset between the underlying geospatial frame and the platform frames, and attempt to provide this capability. Or, they could choose not to.
  • the hardware is incapable. To support this, a device must have an electronic compass, or must use some sort of global tracking system that is aligned with a geospatial frame of reference. For example, if a virtual positioning system (using pre-set visual mapping) was used by the platform, the map could be aligned with a geospatial without the device needing to do the alignment.

The feature can expose geospatially aligned coordinates without provide geospatial position to the web application. This mean that we can request this without asking the user for location permission (assuming they have already given the browser location permission).

If the web page also requests geospatial location access, it will be trivial to create geospatial AR applications. (Crudely) aligning the origin of the WebXR coordinate frame (or the coordinates of an anchor) with a geospatial pose from the web geolocation API is trivial for a web framework to do, and if the WebXR coordinate frame is aligned with the earth, applications can easily display geospatially aligned content. Math libraries (such as the underlying libraries in cesium.js) exist to make this easy (we used these libraries for this very purpose in argon.js).

@TrevorFSmith

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

@blairmacintyre A few questions:

What should happen when the compass value rapidly changes, such as when the device moves near a magnet or large metal structure?

Would there be compass change events to let authors know that they should take measures to prevent nausea?

Would the UA change the orientation of the coordinate system or is it one-and-done at session start?

@blairmacintyre

This comment has been minimized.

Copy link
Author

commented Aug 28, 2018

What should happen when the compass value rapidly changes, such as when the device moves near a magnet or large metal structure?

The same thing that happens now: the OS will do the best it can to ameliorate it. But, by integrating with the XR platform (e.g., WebXR) more can be done that can be done by reading the compass independently: the platform has compass, accelerometer AND visual data, so it can distinguish between more than can be done separately.

Would there be compass change events to let authors know that they should take measures to prevent nausea?

There won't be nausea. Even with ARKit, the anchors don't move when the compass is disturbed, the underlying system updates the coordinate frame and anchors AT THE SAME TIME. So, the Z axis may rotate around Y (for example), but the visual scene remains stable because the TOTAL system can distinguish between different sensors: the visual features and inertial sensors aren't changing so there is no swim.

That's the whole reason I keep pushing for this. Without leveraging the platform, and trying to use the sensors at the web level, this is much harder.

Would the UA change the orientation of the coordinate system or is it one-and-done at session start?

It would change over time, but is essentially indistinguishable from other changes to the underlying platform. Each frame, the platform hands the the system the head pose and anchors in the current coordinate system. If the coordinate system changes it's relationship to the world, it doesn't matter. It's no different if that is caused by the system re-orienting w.r.t. the compass, etc., or because it changed it's understanding of the planes or structure of the world.

That's the beauty of it all.

@TrevorFSmith

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

@blairmacintyre Ok, thank you for the summary.

@johnpallett

This comment has been minimized.

Copy link

commented Aug 28, 2018

+1 to investigating geoAlignment, we think this is worth creating a repo for further discussion.

@cwilso

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

+1 to investigating, but I'd keep this in incubation for now.

@NellWaliczek

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

@TrevorFSmith Can we get a repo opened for this work?

@cwilso

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

We should indeed open a repo. Also, the README.md for that repo (and any new proposal, maybe?) should clearly define the scope it's trying to cover. I think it's more than just requestGeoAlignment() but it would be good to scope the involvement with the other efforts above?

@blairmacintyre

This comment has been minimized.

Copy link
Author

commented Sep 12, 2018

I'd like to have a repo, although honestly the first step is just "requestGeoAlignment()" ... there are a bunch of other things related to geo that we can talk there.

In fact, as we develop libraries to help devs create geospatial applications on top of webxr, we could discuss them (or even put them) there ...

@TrevorFSmith

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

I agree that we should create a repo, but got caught up on some meta work about process.

I lean toward creating a geo-orientation repo specifically for talking about just that feature and (hopefully) moving it toward acceptance by the working group. Then we can retire that repo and the work can continue in the WebXR spec.

For the other aspects of geo like global location we should start a new, separate Issue for each aspect and work it through the same process: Issue -> repo -> submit to appropriate WG (which may or may not be WebRX) -> the WG's process for acceptance.

So, how about I create a repo named "geo-alignment" and tag Blair to write a scope limiting README?

Please thumbs up if that works for you, or comment if you'd like something different.

@blairmacintyre

This comment has been minimized.

Copy link
Author

commented Sep 12, 2018

I don't see a reason to limit the discussion. Sure, ignore my suggestion about long term storage of libraries, that was silly.

But as the discussion above shows, separating this feature from the broader question about how to effectively create geo-spatial AR systems doesn't seem plausible. How can we know the feature is adequate or correct if we can't demonstrate it in use?

I don't think we need to make any decisions or recommendations about things like how WebXR might align with the current geoposition API; all of that "work" can be done in a JS library that builds on those two APIs, with the only difference being that apps that only use the geolocation API wouldn't see added precision without knowing about and managing WebXR. But, that's not really important here, it's really a discussion for the folks working on the geolocation spec. Independent of that, we should work through how this would work, and can even test it, to make sure this simple is sufficient.

After all, the feature itself is trivial; the only real question in and of itself is how to expose it, which is really bound up in the current "session creation" discussions where we're talking about how to request and check for specific capabilities.

@cwilso

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

I'm not thinking this has to only be the narrowest possible slice, but I would like to see some delineation of scope in the readme, even if it grows over time.

@TrevorFSmith

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

@blairmacintyre

This comment has been minimized.

Copy link
Author

commented Sep 12, 2018

We are probably saying the same thing, frankly. I'm also not interested in engaging in wide ranging discussions about geo, since I've already "done that" (we have extensive geospatial AR support in argon.js, I've been down this path multiple times).

My view (which I suspect you implicitly agree with) is that we'll end up discussing some of this, as it relates to "should we do this geo-alignment thing."

So, calling it geo-alignment, and then restricting the discussion to "geo things related to the geo-alignment feature" is what we want.

@TrevorFSmith

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

@TrevorFSmith

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

I have created a new repo to discuss geo-alignment: https://github.com/immersive-web/geo-alignment/

New conversation about that feature should happen in that repo's Issues: https://github.com/immersive-web/geo-alignment/issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.