Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is Markerless AR possible with AR.js? #190

Open
MP619212 opened this issue Sep 18, 2017 · 51 comments

Comments

@MP619212
Copy link

commented Sep 18, 2017

Apple ARKit and Google ARCore allow apps to generate AR Content anchored to the real world using Plane Detection instead of Markers.

Is this possible with AR.js on iOS and Android devices?

GOAL
To view AR Content on a Mobile Web Browser without using a Marker.

Thanks!

@jarodium

This comment has been minimized.

Copy link

commented Sep 18, 2017

https://en.wikipedia.org/wiki/LLA_Marker
Different approach from what you want but I think it could be a great addon for this lib

@ajsutrave

This comment has been minimized.

Copy link

commented Sep 24, 2017

Hey has anyone made any progress on this? If not, I'd like to volunteer

@MP619212

This comment has been minimized.

Copy link
Author

commented Sep 24, 2017

Please contact me at: mattpowell1985@gmail.com and let's discuss via Skype.

@larsgaardsoe

This comment has been minimized.

Copy link

commented Sep 29, 2017

I'm also very interested in this. How can I follow og maybe contribute?

@MP619212

This comment has been minimized.

Copy link
Author

commented Sep 29, 2017

@ajsutrave

This comment has been minimized.

Copy link

commented Sep 29, 2017

OK so I took a little bit of a look. This might be involved change because the Daqri ARKit package that is compiled by ASM.js is what is supplying the movement information to AR.js. We'd have to replace this or add more functionality to make it marker less. We could add something like OpenCV to do the feature point detection.

TLD;DR Difficult but maybe possible

@1d10t

This comment has been minimized.

Copy link

commented Sep 30, 2017

hi2all!

i wrote three basic aframe components for gps- and compass-based markerless ar
(see code at https://github.com/1d10t/test/blob/16b9a59ec9fcacfaa7b5c485622f21c7f4191fbb/a-frame-webcam-dae.html)

'gps-place' component is for object entity
'gps-position' and 'compass-rotation' components is for camera

in result is not too smooth, cause of compass and gps sensor is laggy. maybe need some algorithmic workaround

@jeromeetienne

This comment has been minimized.

Copy link
Owner

commented Oct 25, 2017

hi @1d10t

i love what you did there. and im finnaly got the time to integrate it :) unfortunatly i can't make your example to work. i run it but nothing appears on the screen beside the webcam... any hint on how to make it work ? :)

What i am doing wrong ?

@jeromeetienne

This comment has been minimized.

Copy link
Owner

commented Oct 26, 2017

@1d10t i know what i was doing wrong, i wasnt updating the longitude/lattitude :)

@1d10t

This comment has been minimized.

Copy link

commented Oct 26, 2017

there was a bug in calculation of move direction, latest commits of this file is better.
also u can test simplified example with sphere primitives https://1d10t.github.io/test/phills-sphere.html

@1d10t

This comment has been minimized.

Copy link

commented Oct 26, 2017

i expect that we can use some sort of LLA-markers to stabilize position in highly noisy locations, where GPS like a crazy bunny

@jeromeetienne

This comment has been minimized.

Copy link
Owner

commented Oct 26, 2017

@1d10t i would love to talk about how to improve the accuracy

@jeromeetienne

This comment has been minimized.

Copy link
Owner

commented Oct 26, 2017

@1d10t i noticed you dont handle altitude in your code. is that just small shortcut or is there an issue with handling altitude ?

@1d10t

This comment has been minimized.

Copy link

commented Oct 26, 2017

just didnt try

@raynt

This comment has been minimized.

Copy link

commented Oct 27, 2017

I am very interested in testing this out. From the example:
https://github.com/1d10t/test/blob/16b9a59ec9fcacfaa7b5c485622f21c7f4191fbb/a-frame-webcam-dae.html

What would need to be edited in this file to test? I adjusted the latitude and longitude but don't see any AR object appearing at these coordinates. Would be very thankful for a quick run through how to get this example running.

@AndreaMonzini

This comment has been minimized.

Copy link

commented Oct 27, 2017

Hello, i just tested it and it works on iPad Air® with iOs® 11 !

@1d10t great work! are your components under MIT license like AR.js?

@raynt 3D .dae object are commented, just delete comments ;)

Testing: http://amvr.it/test/agps.html

@raynt

This comment has been minimized.

Copy link

commented Oct 27, 2017

Thank you for the reply Andrea. When I delete the comments I just get a white screen.

`		<a-asset-item id="fence_50_31_0040507_9_asset" src="fence_50_31_0040507_9.dae"></a-asset-item>
		<a-asset-item id="fence_50_31_0040507_6_asset" src="fence_50_31_0040507_6.dae"></a-asset-item>
		<a-asset-item id="fence_50_31_0040507_16_asset" src="fence_50_31_0040507_16.dae"></a-asset-item>
		<a-asset-item id="fence_50_31_0040507_25_asset" src="fence_50_31_0040507_25.dae"></a-asset-item>

`

If I set a long/lat coordinate for across the street will I be able to see the 3d object looking across from the opposite side of the street or does the mobile device need to be at that specific long/lat for the 3d object to appear?

@AndreaMonzini

This comment has been minimized.

Copy link

commented Oct 27, 2017

@raynt i keep testing on Android® and GPS this time and it works ( a bit laggy but ok).
Have you changed Longitude and latitude coordinates near to your actual position?
The object appears on the gps-place="longitude: xx.xxxxxxx; latitude: xx.xxxxxxx" from the perspective of the mobile device coordinates.
Remember to delete comments from <a-entity> too.

@1d10t

This comment has been minimized.

Copy link

commented Oct 27, 2017

@AndreaMonzini

This comment has been minimized.

Copy link

commented Oct 29, 2017

@1d10t Thank you :)

I'm testing it with different 3D Object, trying to move around the position and it's working but the position is quite trembling probably for the jitter of the sensor data.

@raynt

This comment has been minimized.

Copy link

commented Oct 30, 2017

Thanks @AndreaMonzini and @1d10t I got it working as well with your assistance. It was very unstable but I was testing indoors so maybe the GPS signal is weak. Interesting to see if it works better outside.

@AndreaMonzini

This comment has been minimized.

Copy link

commented Nov 5, 2017

Hello i found this very interesting project by @andygup , with Apache License Version 2.0, and it could be useful to reduce the jitter:

https://github.com/Esri/html5-geolocation-tool-js

In particular:

https://github.com/Esri/html5-geolocation-tool-js/blob/master/js/GeolocationHelper.js

@andygup

This comment has been minimized.

Copy link

commented Nov 6, 2017

@AndreaMonzini Thanks for the mention! There are three ways to reduce location fluctuations on the current generation of smartphones: location averaging (GelocationHelper.js is one example), have user turn off all data access to Cellular and Wifi (not always possible or ideal) to force the device to use GPS-only, and/or use a Cordova plugin that separates NETWORK locations from GPS locations.

Good news is that high-accuracy GNSS-enabled phones are coming (hopefully) next year.

@javismiles

This comment has been minimized.

Copy link

commented Nov 7, 2017

Im super interested in this as well, thank you @1d10t @AndreaMonzini MIT for your great work, question, I tried your example https://1d10t.github.io/test/phills-sphere.html , on android, and I can see the white sphere right on the place where the mobile phone is. Now, if I wanted to put the sphere x meters in front of me, what and where would I have to change the code? I see that at the moment as I move the phone, the sphere follows the position of the phone, I change the gps-place coordinates, but object remains in same place always, how can I station the object so that its always either in the same global coordinates independently of the mobile phone position or better in the same relative position x distance away in relation to the mobile phone position?

Update: I wasn't testing right yesterday, now I have put the right latitude and longitude coordinates and now the testing works well, I can see and move around the entity but of course no precision, it jumps all over the place, but I can see the potential indeed

thanks a lot again ;)

@raynt

This comment has been minimized.

Copy link

commented Nov 7, 2017

@javismiles this example requires Unity and Kudan SDK ( https://www.youtube.com/watch?v=7-SiR_qJwaU ) but I think it helps explain what you are looking for (scroll to end of video). Basically the 3d model stays in an anchored position and you can walk closer, further away or around the object. This would be an amazing addition to AR.JS

@1d10t

This comment has been minimized.

Copy link

commented Nov 7, 2017

@javismiles try to go further and you will see, that spheres moves with large jumps by several meters. if you've ever played in ingress (by niantic-google), you might have noticed similar accuracy.

@javismiles

This comment has been minimized.

Copy link

commented Nov 7, 2017

@raynt, @1d10t thank you for your messages, so @raynt yes that's exactly what I mean, and this is something that Jerome showed recently in his first experiment combining Google ArCore and ar.js https://www.youtube.com/watch?v=I1foLzF8slc , I wonder how these ar.js - arcore experiments tie in with this other markerless gps based approach, in any case I can see that the final effect is similar except that here we have at the moment a lack of precision, the positioned object jumps all over the place because of the gps accuracy issues I guess,

and @1d10t so you mean that if i move around a lot the sphere won't stay in the center but will jump all around, ok so you are right yes, I just made new tests, I wasn't testing correctly yesterday, now I get it all, I input my latitude and longitude and now yes, I can see the box entity I added to the scene and I can move around it but yes it jumps all around and it's not precise at all, I get you all now yes, so it works but it's not precise at all, we can see the huge potential of it, but it has a total lack of precision right now

I wonder if, as some have suggested above, if LLA markers used indoors would help to make this experiment more precise?

btw, speaking of markerless AR and a bit off topic, another type of markerless ar is feature based detection and last week totally by chance i discovered these people:
https://github.com/Tastenkunst/brfv4_javascript_examples
https://tastenkunst.github.io/brfv4_javascript_examples/
I don't know if you are familiar with them, but it's pretty amazing...

thank u everybody, this is all fascinating stuff, Ive been playing with ar.js and aframe for a while now and really enjoying all of this

@AndreaMonzini

This comment has been minimized.

Copy link

commented Nov 7, 2017

@andygup thank you for the explanation!
@javismiles of course the system works better outdoor and only if you do not need precise position accuracy, i'm implementing a project in large squares for example.

@raynt

This comment has been minimized.

Copy link

commented Nov 7, 2017

Thanks for sharing those links @javismiles.
@jeromeetienne will you be releasing your implementation of ar.js/arcore?

@enjinstudios

This comment has been minimized.

Copy link

commented Jan 2, 2018

Any plans to roll this feature out in the near future? Happy 2018!

@jucall

This comment has been minimized.

Copy link

commented Mar 17, 2018

@1d10t I tried your code (https://1d10t.github.io/test/phills-sphere.html) but I can not see anythings. Maybe because I'm in southern hemisphere, with negatives coordinates (-33.0519424,-71.3836586).
Did you tested it before with negatives coordinates?.
Greetings.

@tomasdev

This comment has been minimized.

Copy link

commented May 4, 2018

anybody has a good plane detection paper?

@florianmaxim

This comment has been minimized.

Copy link

commented Jul 26, 2018

So it seems like this is not being developed anymore?
That's too bad :(

@jack0088

This comment has been minimized.

Copy link

commented Aug 10, 2018

I googled an ~hour and this seemed to be the best framework for AR over the web ... but the markers make it pretty odd to use.

@ajmaurya99

This comment has been minimized.

Copy link

commented Aug 20, 2018

Still my question is Markerless object placement is possible using AR.js
I still haven't found any solution to this.
Please share if you know on how to achieve this.
Or any other library that I can use to achieve the same on web

@kageori

This comment has been minimized.

Copy link

commented Sep 22, 2018

I really need information about developing markerless AR.
Is this function not developed in AR.js?

@aboutqx

This comment has been minimized.

Copy link

commented Oct 29, 2018

Seems it's not using gps like arkit or arcore,but an algorithm called slam.
link: https://www.kudan.eu/kudan-news/b2-evaluate-slam-accuracy/

@mnutsch

This comment has been minimized.

Copy link

commented Oct 29, 2018

@aboutqx The current iteration of AR.js is best described as a mashup of ARToolkit (marker detection) and AFrame.js (3D scene rendering).

@kageori Standard mobile web browsers don't expose ARKit/ARCore functionality (yet). If you want to experiment with markerless AR through the web right now, you might want to check out Mozilla's Project WebXR Viewer: https://github.com/mozilla-mobile/webxr-ios

@jlcasasg

This comment has been minimized.

Copy link

commented Nov 7, 2018

@1d10t I tried your code (https://1d10t.github.io/test/phills-sphere.html) but I can not see anythings. Maybe because I'm in southern hemisphere, with negatives coordinates (-33.0519424,-71.3836586).
Did you tested it before with negatives coordinates?.
Greetings.

@jucall
Did you manage to run the example?, I'm having the same issue :/

@1d10t

This comment has been minimized.

Copy link

commented Nov 9, 2018

hi ppl
now u can test 'gps-place' and 'gps-position' components on another example with your PC

  1. open https://1d10t.github.io/test/a-frame-dae-gmap-ground.html

  2. run code in console
    camera.components['gps-position'].zeroCrd = {latitude: -33.0519424, longitude: -71.3836586};
    camera.components['gps-position'].crd = {latitude: -33.0519424, longitude: -71.3836586};
    camera.setAttribute('position', camera.getAttribute('position'));
    document.querySelector('a-scene').append(ce('a-sphere', {'gps-place': "longitude: -71.3836586; latitude: -33.0519424"}));

  3. sphere will appears under your virtual location

  4. use wasd controls to move camera

@jucall

This comment has been minimized.

Copy link

commented Nov 9, 2018

@1d10t great work !!! Thanks.
Let's try it @jlcasasg

@hrobar hrobar referenced this issue Dec 5, 2018
@evaristoc

This comment has been minimized.

Copy link

commented Jan 3, 2019

Hi,

I confirm I saw this example on Samsung Galaxy S7, Android Oreo: https://1d10t.github.io/test/a-frame-dae-gmap-ground.html

This one could see updating parameters but no sphere: https://1d10t.github.io/test/phills-sphere.html. The size of the scene didn't fit the size of screen.

Both tested indoors.

@evaristoc

This comment has been minimized.

Copy link

commented Jan 3, 2019

Is this part of the project developed somewhere I could watch to see if I could help/test? It seems you are using some ML for image stabilization? I probably could help.

@evaristoc

This comment has been minimized.

Copy link

commented Jan 3, 2019

@1d10t @jeromeetienne and all:

Have you decided which technologies / solutions you would like to work for the markerless example?

  • LLA (something like what @1d10t has been working so far)?
  • SLAM?
  • other(s)?
@evaristoc

This comment has been minimized.

Copy link

commented Jan 22, 2019

@1d10t for the many (excellent) examples you made using mobile location sensors, argon.js appears to do more or less what your code does.

If I understand correctly what argon.js tries to do differently is making you the 0,0 coordinate and re-calculating all the coordinates to any point in the world from you as reference. In that way you would not be mapped incorrectly if the precision fails: no matter your GPS coordinates, you are always 0,0.

The problem in both cases resides in the lack of precision and the variability of the GPS data. In your case the positioning is as accurate and precise as the GPS data. In the case of argon.js the positioning is more accurate but still unstable, ranging between 2 and 100mts away from my actual position. If I understand correctly the argon.js approach, I think its problem is the speed at which the GPS data changes versus the time it takes to recalculate the reference points. It is possible that by the time I am setup as 0,0 the GPS data changed so all the relations become wrongly estimated.

I actually don't know.

Here an explanation of how argon.js works:
https://docs.argonjs.io/tutorial/part1/

They recommend the following (which now make sense...):

Since the local reference frame may change at any time, a programmer should not save and use the values in this frame for more than a single update and render step. If the values need to be saved and used over multiple frames, it is possible to be notified when the local frame of reference changes.

The argon.js repository is now deprecated: its authors are putting their efforts in what it seems the main focus of several organizations nowadays: XR browsers - I think @jeromeetienne is also involved in one of those initiatives? Despite of being deprecated I would recommend you to check it out! They have been using another library (cesium.js) to estimate positioning.

@droid001

This comment has been minimized.

Copy link

commented Jan 24, 2019

I'm guessing one needs a tango phone to run the markerless tracking method in ar.js which Jerome was writing about in this Medium article?

@FangerZero

This comment has been minimized.

Copy link

commented Feb 6, 2019

I found this but can't seem to get any WebAR to work on my Pixel XL regardless of what I try... I can't even get the Google Samples to work...
https://medium.com/@urish/web-powered-augmented-reality-a-hands-on-tutorial-9e6a882e323e

@jonnyscholes

This comment has been minimized.

Copy link

commented Feb 6, 2019

@commentatorboy

This comment has been minimized.

Copy link
Contributor

commented Feb 6, 2019

I found this but can't seem to get any WebAR to work on my Pixel XL regardless of what I try... I can't even get the Google Samples to work...
https://medium.com/@urish/web-powered-augmented-reality-a-hands-on-tutorial-9e6a882e323e

It is because you have to download the chrome-like browser for it work.
https://github.com/google-ar/WebARonARCore.

I'm guessing one needs a tango phone to run the markerless tracking method in ar.js which Jerome was writing about in this Medium article?

Yes you do.

So it seems like it is not possible to do markerless AR in ar.js in common browsers.
And just to notify people. Tango is now called ARCore. So I do not know if Tango is supported in any way.

So for anything to work you have to install:
https://play.google.com/store/apps/details?id=com.google.ar.core&e=-EnableAppDetailsPageRedesign

Also you can see here how it should be done.
https://developers.google.com/web/updates/2018/06/ar-for-the-web

@tomasdev

This comment has been minimized.

Copy link

commented Feb 14, 2019

For those needing plane detection I highly recommend 8thwall.com

@droid001

This comment has been minimized.

Copy link

commented Jun 5, 2019

A comment about 8thwall, I think they don't have true SLAM detection but rather use the accelerometers and other sensors to place a model in 3D space. It works on close flat surfaces but if for instance you've placed the model on the floor and then move backwards where there's table surface then the model isn't obscured by the table top. Expensive tech for something which doesn't do the job 100% in my opinion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.