Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Adding ARCore #26221

Open
wants to merge 4 commits into
base: master
from

Conversation

Projects
None yet
6 participants
@BastiaanOlij
Copy link
Contributor

BastiaanOlij commented Feb 24, 2019

This is only a beginning but just in case people want to follow the development, provide suggestions, etc, submitting my changes so far.

Right now it almost runs, it downloads the libraries, requests permissions (well with a little help from GDScript at this time) and has all the code that should allow us to do positional tracking. No camera image yet or CameraServer integration. Unfortunately ARCore refuses to initialize and that is currently preventing me from continueing.

For those who want to help out you can best clone the ARCore branch on my fork and just compile it,
I'm maintaining a test project here:
https://github.com/BastiaanOlij/godot3_test_projects/tree/ARCore/ARCore

Note also that ARCore seems to only work on 64bit android phones, so compile accordingly!

@BastiaanOlij BastiaanOlij added this to the 3.2 milestone Feb 24, 2019

@BastiaanOlij BastiaanOlij self-assigned this Feb 24, 2019

@BastiaanOlij BastiaanOlij requested a review from akien-mga as a code owner Feb 24, 2019

@BastiaanOlij BastiaanOlij force-pushed the BastiaanOlij:ARCore branch from ab860a5 to df28bd2 Feb 24, 2019

@BastiaanOlij

This comment has been minimized.

Copy link
Contributor Author

BastiaanOlij commented Feb 24, 2019

One thing I could use some help with is including ARCore libraries itself. I've reserved a place for these within thirdparty/arcore, they are originally sourced from here: https://github.com/google-ar/arcore-android-sdk

Right now you need to download the Android SDK, compile the hello ar example, this will cause gradlew to extract a number of support libraries, and copy those support libraries into thirdparty/arcore/jni

The license around these .so files may not be compatible with MIT so I don't know if we can just include them in the thirdparty/arcore folder. If not we may need to look into scons or gradlew downloading them as components somehow.

The other issue I'm trying to solve is that while we're compiling these .so files into our project (see the detect.py changes) Android is still complaining that the files can't be loaded. It looks like they have dynamic counter parts.

@aaronfranke

This comment has been minimized.

Copy link
Contributor

aaronfranke commented Feb 24, 2019

Is this only for Android, or does it also work with other AR platforms like Windows MR?

@HeadClot

This comment has been minimized.

Copy link

HeadClot commented Feb 24, 2019

Is this only for Android, or does it also work with other AR platforms like Windows MR?

It is only for android as far as I know.

@BastiaanOlij

This comment has been minimized.

Copy link
Contributor Author

BastiaanOlij commented Feb 25, 2019

@BastiaanOlij BastiaanOlij force-pushed the BastiaanOlij:ARCore branch from df28bd2 to 7d2d2ee Feb 28, 2019

@BastiaanOlij

This comment has been minimized.

Copy link
Contributor Author

BastiaanOlij commented Feb 28, 2019

Ok, it now pulls in the AR libraries from Google through gradlew. Stores them in thirdparty/godot, then copies the files it needs into platform/android/java/jni

Now compiles and starts up without crashing, but on my pocophone it fails on ArCoreApk_requestInstall
I have working ARCore apps so I'm guessing it's a rights issue.

Anyway, getting closer to a version that does tracking :)

@@ -169,3 +169,15 @@ if lib_arch_dir != '':
if ndk_version != None and LooseVersion(ndk_version) >= LooseVersion("15.0.4075724"):
stl_lib_path = str(env['ANDROID_NDK_ROOT']) + '/sources/cxx-stl/llvm-libc++/libs/' + lib_arch_dir + '/libc++_shared.so'
env_android.Command(out_dir + '/libc++_shared.so', stl_lib_path, Copy("$TARGET", "$SOURCE"))

if 'module_arcore_enabled' in env and env['module_arcore_enabled']:

This comment has been minimized.

Copy link
@BastiaanOlij

BastiaanOlij Feb 28, 2019

Author Contributor

Need to see if we can move this into modules\arcore\SCSub somehow

@@ -298,6 +298,17 @@ def mySpawn(sh, escape, cmd, args, env):
env.Append(CPPFLAGS=['-DANDROID_ENABLED', '-DUNIX_ENABLED', '-DNO_FCNTL'])
env.Append(LIBS=['OpenSLES', 'EGL', 'GLESv3', 'android', 'log', 'z', 'dl'])

## arcore, should see about moving this into our module

This comment has been minimized.

Copy link
@BastiaanOlij

BastiaanOlij Feb 28, 2019

Author Contributor

Same with this, need to see if we can move it into modules\arcore somehow, not sure if its possible

@BastiaanOlij BastiaanOlij force-pushed the BastiaanOlij:ARCore branch from 7d2d2ee to d1e4231 Mar 3, 2019

@@ -12,10 +12,12 @@
android:xlargeScreens="true"/>

<uses-feature android:glEsVersion="0x00020000" android:required="true" />
<uses-feature android:name="android.hardware.camera.ar" android:required="true"/>

This comment has been minimized.

Copy link
@BastiaanOlij

BastiaanOlij Mar 3, 2019

Author Contributor

Features should become part of our export settings


$$ADD_PERMISSION_CHUNKS$$

<application android:label="@string/godot_project_name_string" android:icon="@drawable/icon" android:allowBackup="false" tools:ignore="GoogleAppIndexingWarning" $$ADD_APPATTRIBUTE_CHUNKS$$ >
<meta-data android:name="com.google.ar.core" android:value="required" />

This comment has been minimized.

Copy link
@BastiaanOlij

BastiaanOlij Mar 3, 2019

Author Contributor

This too needs to be redone as an export setting

@BastiaanOlij BastiaanOlij force-pushed the BastiaanOlij:ARCore branch 2 times, most recently from 5ab0d42 to 985ef98 Mar 6, 2019

@BastiaanOlij

This comment has been minimized.

Copy link
Contributor Author

BastiaanOlij commented Mar 7, 2019

I've added the AndroidPermissions singleton code in this for now, with this and some GDScript code the camera permissions now work properly. This needs to be redone and properly implemented outside of this PR (see #26553 and #26506 for related discussions) so it's just in there temporarily.

The proper implementation here would be for the ARCore module to interact with the permissions module, ask it if camera permissions have been granted and request permissions if they have not. This way there is no code needed on the GDScript side.

ARCore still fails to load with an non-descriptive error. I think the problem lies at the start of _resume where I'm retrieving the activity.

@whplh

This comment has been minimized.

Copy link

whplh commented Mar 8, 2019

Hey there. To keep it concise. I'm using a debug build of this commit with the ARCore sample project from BastiaanOlij 👍 :

  • From the template manager I get the message: "Error getting the list of mirrors"

    • Template manger in 3.06 works fine
  • I manually installed/tried the export templates beta1 and beta11 from https://downloads.tuxfamily.org/godotengine/3.1/

  • When I try to export the sample project to Android I get:

    Failed to export the project for platform 'Android' Export templates seem to be missing or invalid

    • Other export platforms work
    • I can export with Godot 3.0.6 to Android -no problems

Which export templates should I use?

Note: I could not find ARCore's license in Godots UI : Help->About->Thirdparty License or is it "The Android Open Source Project" ?

Anyway - Great work! Any hints on how to resolve my issues would be great. Happy weekend.

@akien-mga

This comment has been minimized.

Copy link
Member

akien-mga commented Mar 8, 2019

Which export templates should I use?

You should build export templates yourself from the same commit you used to build the editor. There are no export templates prebuilt for that commit, which is why you can't use the export template manager.
The simplest is to build your own APK and add it as Custom Debug Package in the Android preset.

Note: I could not find ARCore's license in Godots UI : Help->About->Thirdparty License or is it "The Android Open Source Project" ?

Those licenses are extracted from the COPYRIGHT.txt file. ARCore is not in there yet, and being a module it should likely not be hardcoded but added conditionally - which is not possible at the moment, see #23283.

@BastiaanOlij

This comment has been minimized.

Copy link
Contributor Author

BastiaanOlij commented Mar 9, 2019

@whplh what akien said :)
Also this currently is far from ready for any testing, it doesn't run yet. Still trying to figure out a bunch of build issues.

@BastiaanOlij BastiaanOlij force-pushed the BastiaanOlij:ARCore branch 2 times, most recently from 288aaf6 to 41221b2 Mar 11, 2019

@BastiaanOlij

This comment has been minimized.

Copy link
Contributor Author

BastiaanOlij commented Mar 12, 2019

Ok, this is starting to do something.

First, notifications aren't being sent to ARVRInterfaces yet, so when you give camera permission we don't get our signal to do our next step. Give camera permission and restart the app.
Then, request install doesn't work yet, the code is commented out for now. Make sure ARCore is installed on your phone!
You can't visualise the camera yet, I hope to work on that soon.
Finally compiling has proven to have some interesting issues. The following works for me on windows:

scons platform=android target=release_debug -j8
scons platform=android target=release_debug android_arch=arm64v8 -j8
cd platform\android\java
gradlew clean
gradlew build

This builds the debug build ONLY, if you need the release build add that in too.
The gradlew clean is very important

@reduz

This comment has been minimized.

Copy link
Member

reduz commented Apr 4, 2019

can't wait to merge once it's working!

@BastiaanOlij
Copy link
Contributor Author

BastiaanOlij left a comment

can't wait to merge once it's working!

Still a lot to do :) I'll round off that restructure PR first and then I need to add support for an Android specific texture type to the visual server. And then finish the camera server so we can actually display the camera image, and then I can finally start making this do something that makes sense :)

@@ -103,6 +103,7 @@

static final int MAX_SINGLETONS = 64;
static final int REQUEST_RECORD_AUDIO_PERMISSION = 1;
static final int REQUEST_CAMERA_PERMISSION = 2;
private IStub mDownloaderClientStub;

This comment has been minimized.

Copy link
@BastiaanOlij

BastiaanOlij Apr 5, 2019

Author Contributor

I'll be moving this into a separate PR

@BastiaanOlij BastiaanOlij force-pushed the BastiaanOlij:ARCore branch 2 times, most recently from 3a278a6 to e0d7806 Apr 7, 2019

@BastiaanOlij

This comment has been minimized.

Copy link
Contributor Author

BastiaanOlij commented Apr 7, 2019

Ok, just rebased and fixed a few small things. The initialization of ARCore now works properly. The view and projection matrices are still wrong, I think its got to do with providing the incorrect orientation data to ARCore so it will only be correct if you hold your phone in one orientation. Not 100% sure.

I'm going to work on the camera next.

@BastiaanOlij BastiaanOlij force-pushed the BastiaanOlij:ARCore branch from e0d7806 to 87dd2d9 Apr 16, 2019

@BastiaanOlij BastiaanOlij requested review from eska014, reduz, vnen and godotengine/documentation as code owners Apr 16, 2019

@BastiaanOlij

This comment has been minimized.

Copy link
Contributor Author

BastiaanOlij commented Apr 16, 2019

Nearly got the camera server to work with ARCore, getting late so checked in all the changes so far.

Note that:

  • we're not getting notifications yet so we don't react on gaining camera permissions. You need to restart the app after providing camera permissions before tracking will work
  • background image is not being shown yet, nearly there
  • the camera is reversed, so we're moving the wrong way, need to rotate it 180 degrees, haven't had time to look at that yet.
  • orientation has not yet been fixed.

I'm also going to move some changes into the camera server PR and move a few of the visual server changes into its own PR

@BastiaanOlij BastiaanOlij force-pushed the BastiaanOlij:ARCore branch from 87dd2d9 to f6a8b2d Apr 17, 2019

@BastiaanOlij BastiaanOlij force-pushed the BastiaanOlij:ARCore branch from f6a8b2d to 1acd8fb Apr 18, 2019

@BastiaanOlij BastiaanOlij requested a review from karroffel as a code owner Apr 18, 2019

@BastiaanOlij

This comment has been minimized.

Copy link
Contributor Author

BastiaanOlij commented Apr 18, 2019

Ok, work for today was mostly restructuring after putting a bunch of changes into the PRs they belong in.

I'm nearly at a point where the camera image renders. In order to access the camera images we need to use a special extention on Android. Turns out this also needs a special sampler in the shader called samplerExternalOES, I've rigged it in for now.

Problem is that that results in a compilation error:
ERROR: 0:48: 'samplerExternalOES' : requires extension GL_OES_EGL_image_external_essl3 to be enabled

@reduz any ideas how I enable that extension?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.