New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Cardboard mode viewing of Spatial -> 3D scenes #369
Comments
Welcome @bilalsattar24, @juanv911, and @leo831! It will be great to have this feature in STOQS. I'm glad that you chose this issue for your capstone project. Please use this thread to ask any questions and report any progress. |
Checkout https://developer.oculus.com/webvr/ |
I see that @juanv911 submitted Pull Request #420. That's great! Can @bilalsattar24 or @leo831 provide any comments here on how this works? (It will be a few weeks before I can test the proposed code changes.) |
This pull request was a mistake. We aren't quite ready for a pull request yet. But the changes we've made so far includes some code to generate a 3d human model we found online in x3d format. It generates the image and we can move it around on desktop, but it does't render properly on mobile. The current Oculus example includes the 3d images in .bin format and we're working towards using those .bin graphics. Sorry about the accidental pull request. |
Hey Mike, We have been submitting our code to this branch. |
Hey Mike, |
I tested your PR and it kind of works! I think we can iterate together a few times over the next few weeks to help make it ready for merging into master and be available for easier testing on http://stoqs.mbari.org:8000/. I'll make some comments on your code in #422. |
Hey Mike, |
Hi Juan, I think you are really close. When you assign to CARDBOARD.viewpoint use |
I suggest renaming the Enter VR/Leave VR buttons. Instead of:
make it:
|
Hi Mike, I changed the id of CARDBOARD.viewpoint to We will keep working on it and we will synchronize with upstream and submit our commit tomorrow after our group meeting. |
Here's how I modified your code to bind to the
Also, if you don't see anything in the scene try pressing Page Down to cycle through the viewpoints. (I suspect that there is a bug where near and far clipping planes are not being properly set if the viewpoints are switched.) |
With those new changes, it shows the center of the scene, but only on Safari and Firefox. In Chrome it shows the scene very close, but not properly and yeah when Leaving Cardboard, it doesn't reset back to the initial size. By near and far clipping planes, are you referring to the zFar and zNear attributes? |
Yes. zFar and zNear define the clipping planes. They are changed in order to see objects that are close to the viewpoint. I think #422 is almost ready for merging. Please take a look at https://github.com/stoqs/stoqs/pull/422/files and clean up any white space differences from the original. We want to see only the relevant changes in the diff. |
The formatting/whitespaces have been taken care of. The diff looks much cleaner now. |
X3DOM Google Cardboard headset rotation in support of #369
Congratulations! I just merged #422 and did a quick test on MBARI's production server. The viewpoint orientation was driven by my phone's orientation when I was in Cardboard mode. Good job! There was one problem though: I could not rotate the normal mode scene with the mouse. It seems that the sensor update of the viewpoint orientation happens all the time. Please look at your code and see if you can put the sensor update portion in a place so that it gets executed only when |
Thanks |
Hi Mike, I submitted #428. I created a function called enable_cardboard() and put the Javascript code there. I removed the |
I pulled what you have now in #428 to http://stoqs.mbari.org:8000/default where you should be able to test it with your phones. I did notice that the 'Leave Cardboard' button doesn't work. |
I was able to test it on my phone and it works on Firefox, but not in Chrome. When I add this line |
#428 has been pulled to http://stoqs.mbari.org:8000/default. The 'Leave Cardboard' button now switches back to the original Viewpoint, but orientation data from the IMU is still being sent to the Viewpoint's orientation. |
According to http://caniuse.com/fullscreen the 'RequestFullScreen' call does not work for iOS Safari or the Android Browser. I think instead we can make use of the full screen button already implemented in the JavaScript (geo-zoom-lc-button). Also, some parent <div>s will need to have their padding set to 0px for the scene to fully fill the phone's screen. |
We edited the file to make the landscape mode work. |
Hi Mike, I removed |
I have pushed the latest changes. |
Current #428 has been pulled to http://stoqs.mbari.org:8000/default. Looking pretty good! I suggest removing ' auto' from this line so that extra white space isn't added below the scene:
|
I removed 'auto' from the X3D html tag and removed a submodule and a .swp file that were created and pushed by accident. |
I think the same X3D element can be used for Rift and Cardboard. The HTML for |
I tested on my development system with the following diff and the Enter Cardboard / Leave Cardboard functionality worked fine. This diff removes the Rift functionality, which is O.K. as that was even more experimental than this new Cardboard functionality. It also removes the experiment with the Leap Motion hand gesture rendering, which is also O.K. as that never worked. Perhaps you can test with these changes and commit your file. With that I think #428 can be merged. Here's the diff: $ git diff stoqs/stoqs/templates/stoqs/stoqsquery.html
diff --git a/stoqs/stoqs/templates/stoqs/stoqsquery.html b/stoqs/stoqs/templates/stoqs/stoqsquery.html
index 75df235..420c55c 100644
--- a/stoqs/stoqs/templates/stoqs/stoqsquery.html
+++ b/stoqs/stoqs/templates/stoqs/stoqsquery.html
@@ -314,7 +314,7 @@
</div> <!-- geo-3dplot-control -->
<div style="height:100%;">
<!-- Elements for Google Cardboard -->
- <X3D id="spatial-3d-x3d-cardboard" style="width:100%;height:100%;margin:0px;display:block;border:none;" ondownloadsfinished="downloadsFinished(event)">
+ <X3D id="spatial-3d-x3d" style="width:100%;height:100%;margin:0px;display:block;border:none;" ondownloadsfinished="downloadsFinished(event)">
<scene>
<Environment frustumCulling="false"></Environment>
<navigationInfo></navigationInfo>
@@ -421,163 +421,6 @@
</X3D>
<!-- End of Cardboard Elements -->
- <X3D id="spatial-3d-x3d" style="width:100%;height:100%;margin:0px auto;display:block;border:none;" ondownloadsfinished="downloadsF
- <Scene>
- <Environment frustumCulling="false"></Environment>
- <NavigationInfo></NavigationInfo>
- <Viewpoint id="mp-x3d-viewpoint1" def="vp" onviewpointChanged="setScale(event)"></Viewpoint>
- <!-- Test GeoViewpoint for Monterey Bay area, user must page-down to bind to it -->
- <GeoViewpoint id="mp-x3d-geoviewpoint1" def="gvp" position="36.8 -122.1 100000" orientation="1 0 0 -1.57" centerofrotation
- </GeoViewpoint>
- <Background DEF='bgnd' skyColor=".9608 .9608 .9608"></Background>
- <Group id='spatial-3d-root' render='true'>
- <Group id='spatial-3d-scene' def="spatial-3d-scene">
- <GeoOriginTransform containerField='children'>
- <Group id="mp-x3d-terrain">
- </Group>
- </GeoOriginTransform>
- <Group id="platform-animations"></Group>
- <Group id="platform-models"></Group>
- </Group>
- </Group>
-
- <!-- Elements for Rift - Orientations routed to GeoViewpoint, so must page down to that Viewpoint -->
- <group id='spatial-3d-left' DEF='left' render='false'>
- <shape>
- <appearance>
- <renderedTexture id="rtLeft" stereoMode="LEFT_EYE" update='ALWAYS'
- dimensions='640 800 4' repeatS='false' repeatT='false'>
- <!--<viewpoint USE='vp' containerField='viewpoint'></viewpoint>-->
- <geoviewpoint USE='gvp' containerField='GeoViewpoint'></geoviewpoint>
- <background USE='bgnd' containerField='background'></background>
- <!--group USE='left' containerField="excludeNodes"></group-->
- <group USE='spatial-3d-scene' containerField="scene"></group>
- </renderedTexture>
-
- <composedShader>
- <field name='tex' type='SFInt32' value='0'></field>
- <field name='leftEye' type='SFFloat' value='1'></field>
- <shaderPart type='VERTEX'>
- attribute vec3 position;
- attribute vec2 texcoord;
-
- uniform mat4 modelViewProjectionMatrix;
- varying vec2 fragTexCoord;
-
- void main()
- {
- vec2 pos = sign(position.xy);
- fragTexCoord = texcoord;
-
- gl_Position = vec4((pos.x - 1.0) / 2.0, pos.y, 0.0, 1.0);
- //gl_Position = vec4(pos.xy / 4.0 + vec2(-0.75,0.75), 0.0, 1.0);
- }
- </shaderPart>
- <shaderPart DEF="frag" type='FRAGMENT'>
- #ifdef GL_ES
- precision highp float;
- #endif
-
- uniform sampler2D tex;
- uniform float leftEye;
- varying vec2 fragTexCoord;
-
- void main()
- {
- float distortionScale = 0.7; //0.685;
- //float distortionK[4] = float[](1.0, 0.22, 0.24, 0.0);
- vec2 lensCenter = vec2(0.151976495726, 0.0);
- if (leftEye == 0.0) {
- lensCenter.x *= -1.0;
- }
-
- vec2 theta = (fragTexCoord * 2.0) - 1.0;
- float rSq = theta.x * theta.x + theta.y * theta.y;
- vec2 rvec = theta * (1.0 + 0.22 * rSq + 0.24 * rSq * rSq); //+ 0.0 * rSq * rSq * rSq);
- vec2 texCoord = (distortionScale*rvec+(1.0-distortionScale)*lensCenter + 1.0) / 2.0;
-
- if (any(notEqual(clamp(texCoord, vec2(0.0, 0.0), vec2(1.0, 1.0)) - texCoord,
- vec2(0.0, 0.0)))) {
- //if (leftEye == 0.0) gl_FragColor = vec4(1.0,1.0,0.0,1.0);
- //else gl_FragColor = vec4(1.0,0.0,0.0,1.0);
- discard;
- }
- else {
- vec3 col = texture2D(tex, texCoord).rgb;
- gl_FragColor = vec4(col, 1.0);
- }
- //gl_FragColor = texture2D(tex, fragTexCoord);
- }
- </shaderPart>
- </composedShader>
-
- </appearance>
- <plane solid="false"></plane>
- </shape>
- </group>
-
- <group id='spatial-3d-right' DEF='right' render='false'>
- <shape>
- <appearance>
- <renderedTexture id="rtRight" stereoMode="RIGHT_EYE" update='ALWAYS'
- dimensions='640 800 4' repeatS='false' repeatT='false'>
- <!--<viewpoint USE='vp' containerField='viewpoint'></viewpoint>-->
- <geoviewpoint USE='gvp' containerField='GeoViewpoint'></geoviewpoint>
- <background USE='bgnd' containerField='background'></background>
- <group USE='spatial-3d-scene' containerField="scene"></group>
- </renderedTexture>
- <composedShader>
- <field name='tex' type='SFInt32' value='0'></field>
- <field name='leftEye' type='SFFloat' value='0'></field>
- <shaderPart type='VERTEX'>
- attribute vec3 position;
- attribute vec2 texcoord;
-
- uniform mat4 modelViewProjectionMatrix;
- varying vec2 fragTexCoord;
-
- void main()
- {
- vec2 pos = sign(position.xy);
- fragTexCoord = texcoord;
-
- gl_Position = vec4((pos.x + 1.0) / 2.0, pos.y, 0.0, 1.0);
- }
- </shaderPart>
- <shaderPart USE="frag" type='FRAGMENT'>
- </shaderPart>
- </composedShader>
- </appearance>
- <plane solid="false"></plane>
- </shape>
- </group>
-
- <!-- Elements for Leap Motion -->
- <transform id="handtrafo1" translation="3.8 1.00905 -5.97228">
- <transform id="handtrafo" scale="0.0015 0.0015 0.0015" translation="0.1 -0.3 -0.5">
- <transform id="hand">
- <transform>
- <transform>
- <transform>
- <transform scale="40 10 40" center="0 0 -2">
- <shape>
- <appearance>
- <material diffuseColor="0.603 0.894 0.909"></material>
- </appearance>
- <box></box>
- </shape>
- </transform>
- </transform>
- </transform>
- </transform>
-
- <transform id="bones"></transform>
- </transform>
- </transform>
- </transform>
-
- </Scene>
- </X3D>
</div>
</div>
</div>
@@ -2876,7 +2719,7 @@ function enable_cardboard(){
function deg2rad(deg){return deg * degreeToRadiansFactor;}
- runtime = $('#spatial-3d-x3d-cardboard')[0].runtime;
+ runtime = $('#spatial-3d-x3d')[0].runtime;
rtLeft = document.getElementById('rtLeft');
rtRight =document.getElementById('rtRight');
//rtLeft._x3domNode.lensCenter = 0;
@@ -2889,7 +2732,7 @@ function enable_cardboard(){
rtLeft.setAttribute('dimensions', hw + ' ' + lastH + ' 4');
rtRight.setAttribute('dimensions', hw + ' ' + lastH + ' 4');
- var element = document.getElementById("spatial-3d-x3d-cardboard");
+ var element = document.getElementById("spatial-3d-x3d");
CARDBOARD.viewpoint = document.getElementById('mp-x3d-geoviewpoint1');
CARDBOARD.viewpoint.setAttribute('set_bind','true');
element.runtime.exitFrame = function() {
@@ -3230,23 +3073,23 @@ function enable_cardboard(){
$(".cardboard-button").click(function(element) {
if(cardboarded){
- $('#spatial-3d-x3d-cardboard #spatial-3d-root').attr('render', 'true');
- $('#spatial-3d-x3d-cardboard #spatial-3d-left').attr('render', 'false');
- $('#spatial-3d-x3d-cardboard #spatial-3d-right').attr('render','false');
+ $('#spatial-3d-x3d #spatial-3d-root').attr('render', 'true');
+ $('#spatial-3d-x3d #spatial-3d-left').attr('render', 'false');
+ $('#spatial-3d-x3d #spatial-3d-right').attr('render','false');
$('#mp-x3d-geoviewpoint1').attr('position', $('body').data()['extent'][2] + ' ' + $('body').data()['extent'][1] + ' 1000');
$('#mp-x3d-geoviewpoint1').attr('orientation', '1 0 0 -1.57');
$(this).html('Enter Cardboard');
$('#mp-x3d-geoviewpoint1').attr("set_bind", "false");
- $('#spatial-3d-x3d-cardboard')[0].runtime.exitFrame = function() {}
+ $('#spatial-3d-x3d')[0].runtime.exitFrame = function() {}
// Reset padding to its initial padding
$('#left-column').parent().parent().css( { 'padding': '0px 20px' } );
$('#accordion1').css( { 'padding': '19px' } );
$('#collapseSpatial > div').css( { 'padding': '9px 15px' } );
}
else {
- $('#spatial-3d-x3d-cardboard #spatial-3d-root').attr('render', 'true');
- $('#spatial-3d-x3d-cardboard #spatial-3d-left').attr('render', 'true');
- $('#spatial-3d-x3d-cardboard #spatial-3d-right').attr('render', 'true');
+ $('#spatial-3d-x3d #spatial-3d-root').attr('render', 'true');
+ $('#spatial-3d-x3d #spatial-3d-left').attr('render', 'true');
+ $('#spatial-3d-x3d #spatial-3d-right').attr('render', 'true');
$('#mp-x3d-geoviewpoint1').attr('position', $('body').data()['extent'][2] + ' ' + $('body').data()['extent'][1] + ' 100');
$('#mp-x3d-geoviewpoint1').attr('orientation', '0 0 1 0');
$(this).html('Leave Cardboard'); |
Hi Mike, |
Current #428 has been pulled to http://stoqs.mbari.org:8000/default. Reports of proper functioning on various VR devices (smart phones) are welcomed. |
I have an LG G4 Android phone and it loads the scene in fullscreen and it rotates, but only on Firefox. When zooming in on Firefox, it doesn't move to a new position, it resets back to the initial position. In Chrome it doesn't go into fullscreen and it doesn't rotate. In Chrome, before clicking Enter Cardboard you can rotate the 3D object with your fingers, but after clicking the Enter Cardboard button the rotation stops working |
According to the Mobile tab under Browser compatibility at https://developer.mozilla.org/en-US/docs/Web/API/Window/ondeviceorientation Chrome 50 should work. |
Hi Mike,
I have the Samsung Galaxy S7 and I have the same issues that Juan mention.
It only works for Firefox and not for Chrome.
|
Congratulations, #428 has just been merged into master! Would you like to do some extra credit work to enable Cardboard mode with a button like this between the 'i' and 'full screen' buttons? It'd make it easier for people to use and would be a nice thing to put on your poster. |
The existing three buttons are using this sprite |
Yes. Modifying the sprite image would give the best results. |
I'm looking at the GeoViewpoint setting at line: stoqs/stoqs/stoqs/templates/stoqs/stoqsquery.html Line 3092 in e7d66f9
as I don't think it's working correctly. |
We submitted #439. It includes the Cardboard icon and it goes into fullscreen. Changing the background color from white to black like the Rift seems better, but the text and icons seem unreadable. I don't know if that's something you want us to do. |
I merged #439 and pulled to http://stoqs.mbari.org:8000/default. Interestingly, when I tested the PR on my dev machine before merging I was able to see the Cardboard icon, but on our production server there is just a blank space: My dev machine does show it: |
I think it is because the old image is still in the cache. It is not showing the icon on my computer either. In my localhost it is showing it. |
I forgot that on a production Django server we need to copy static files to the web server directory with:
The icon is now visible on http://stoqs.mbari.org:8000/default. |
I'd like to minimize all the menus in when going into Cardboard mode. I'll work on that. |
Here is a good "Share this View" link for testing and demoing the Cardboard feature: Open this on your phone (iPhone or Android/Firefox), wait a few minutes for all the models to load then click the Cardboard button. |
Here's a screen shot from the link mentioned in #369 (comment) and after platform scaling events are fired from the GeoViewpoint: |
What do you guys think, should I close this issue? |
Yeah, I think its time. It's been a good journey. |
Here is the entry for the VR Hackathon held in San Francisco last weekend: https://devpost.com/software/stoqs-cardboard and the YouTube video showing how to use the Cardboard feature in STOQS. The video doesn't show it, but it's helpful to put your phone into one of these: |
The STOQS UI has an experimental "Enter VR" mode that has been tested with the Oculus Rift DK2. Much of this code (Javascript in the main template page) can be reused for implementing a "Cardboard" mode that will work on smart phones and other VR headsets.
See http://examples.x3dom.org/Demos/ClassroomVR/classroom-cardboard.html for a working example.
Also see https://webvr.info/, whose goal is to have one Javascript API for all VR devices.
The text was updated successfully, but these errors were encountered: