Skip to content

Conversation

IvanKobzarev
Copy link
Contributor

@IvanKobzarev IvanKobzarev commented Jun 5, 2020

Stack from ghstack:

Example of using direct linking to pytorch_jni library from aar and updating android/README.md with the tutorial how to do it.

Adding nativeBuild dimension to test_app, using direct aar dependencies, as headers packaging is not landed yet, excluding nativeBuild from building by default for CI.

Additional change to scripts/build_pytorch_android.sh:

Skipping clean task here as android gradle plugin 3.3.2 exteralNativeBuild has problems with it when abiFilters are specified.

Will be returned back in the following diffs with upgrading of gradle and android gradle plugin versions.

Differential Revision: D22118945

@dr-ci
Copy link

dr-ci bot commented Jun 5, 2020

💊 CI failures summary and remediations

As of commit 5808499 (more details on the Dr. CI page):


  • 2/2 failures introduced in this PR

🕵️ 2 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_macos_10_13_py3_test (1/2)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

Jun 18 17:12:13 [E request_callback_impl.cpp:153] Received error while processing request type 2: PickleError: ScriptModules cannot be deepcopied using copy.deepcopy or saved using torch.save. Mixed serialization of script and non-script modules is not supported. For purely script modules use my_script_module.save() instead.
Jun 18 17:12:13   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(90): serialize 
Jun 18 17:12:13   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(142): serialize 
Jun 18 17:12:13  
Jun 18 17:12:13 [E request_callback_impl.cpp:153] Received error while processing request type 2: PickleError: ScriptModules cannot be deepcopied using copy.deepcopy or saved using torch.save. Mixed serialization of script and non-script modules is not supported. For purely script modules use my_script_module.save(<filename>) instead. 
Jun 18 17:12:13  
Jun 18 17:12:13 At: 
Jun 18 17:12:13   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/jit/__init__.py(1895): __getstate__ 
Jun 18 17:12:13   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(90): serialize 
Jun 18 17:12:13   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(142): serialize 
Jun 18 17:12:13  
Jun 18 17:12:13 [E request_callback_impl.cpp:153] Received error while processing request type 2: PickleError: ScriptModules cannot be deepcopied using copy.deepcopy or saved using torch.save. Mixed serialization of script and non-script modules is not supported. For purely script modules use my_script_module.save(<filename>) instead. 
Jun 18 17:12:13  
Jun 18 17:12:13 At: 
Jun 18 17:12:13   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/jit/__init__.py(1895): __getstate__ 
Jun 18 17:12:13   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(90): serialize 
Jun 18 17:12:13   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(142): serialize 
Jun 18 17:12:13  
Jun 18 17:12:13 ok (1.389s) 
Jun 18 17:12:15   test_unexepected_kwarg_is_specified (__main__.JitRpcTestWithSpawn) ... ok (1.243s) 
Jun 18 17:12:16   test_user_rrefs_confirmed (__main__.JitRpcTestWithSpawn) ... ok (1.374s) 
Jun 18 17:12:17   test_user_rrefs_confirmed_remote (__main__.JitRpcTestWithSpawn) ... ok (1.439s) 

See CircleCI build pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build-x86_32 (2/2)

Step: "pytorch android gradle build only x86_32 (for PR)" (full log | diagnosis details | 🔁 rerun)

Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:99) 
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 	... 31 more 
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Caused by: com.android.ide.common.process.ProcessException: Error configuring CMake server (/usr/local/bin).  
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Check for working CXX compiler: /opt/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ 
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Check for working CXX compiler: /opt/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -- works 
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Detecting CXX compiler ABI info 
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Detecting CXX compiler ABI info - done 
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Detecting CXX compile features 
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Detecting CXX compile features - done 
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] ANDROID_STL:c++_shared 
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] CMake Error: The following variables are used in this project, but they are set to NOTFOUND. 
Jun 18 23:44:20 23:44:19.537 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Please set them or make sure they are set and tested correctly in the CMake files: 
Jun 18 23:44:20 23:44:19.538 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] PYTORCH_LIBRARY 
Jun 18 23:44:20 23:44:19.538 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]     linked by target "pytorch_testapp_jni" in directory /var/lib/jenkins/workspace/android/test_app/app 
Jun 18 23:44:20 23:44:19.538 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]  
Jun 18 23:44:20 23:44:19.538 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Configuring incomplete, errors occurred! 
Jun 18 23:44:20 23:44:19.538 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] See also "/var/lib/jenkins/workspace/android/test_app/app/.externalNativeBuild/cmake/mbqLocalBaseRelease/x86/CMakeFiles/CMakeOutput.log". 
Jun 18 23:44:20 23:44:19.538 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]  
Jun 18 23:44:20 23:44:19.538 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 	at com.android.build.gradle.tasks.CmakeServerExternalNativeJsonGenerator.executeProcessAndGetOutput(CmakeServerExternalNativeJsonGenerator.java:180) 
Jun 18 23:44:20 23:44:19.538 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 	at com.android.build.gradle.tasks.CmakeExternalNativeJsonGenerator.executeProcess(CmakeExternalNativeJsonGenerator.java:116) 
Jun 18 23:44:20 23:44:19.538 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 	at com.android.build.gradle.tasks.External

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 52 times.

IvanKobzarev added a commit that referenced this pull request Jun 5, 2020
@IvanKobzarev IvanKobzarev changed the title [WIP][android] test_app example linking to pytorch_android aar content [android] test_app example linking to pytorch_android aar content Jun 5, 2020
IvanKobzarev added a commit that referenced this pull request Jun 6, 2020
IvanKobzarev added a commit that referenced this pull request Jun 8, 2020
@IvanKobzarev IvanKobzarev requested review from ljk53 and dreiss June 8, 2020 18:03
IvanKobzarev added a commit that referenced this pull request Jun 8, 2020
IvanKobzarev added a commit that referenced this pull request Jun 13, 2020

#include "jni.h"

#include <ATen/NativeFunctions.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this header? I think we should still encourage people to use <torch/script.h>

}
signingConfigs {
config {
storeFile file("testapp.keystore")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious: what's the common practice to handle keystore for demo/test app? I found this but not sure whether it's worth the trouble for test app: https://stackoverflow.com/questions/44911547/where-to-store-android-keystore-file-for-cirlceci-build

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can do it more securely for CircleCI. That keystore is completely dummy, there is no 'PyTorch' and 'Facebook' inside it.
I think if somebody will start signing his app with this keystore there should not be any problems.

ALOGI("%s %s", m, os.str().c_str());
}

struct JITCallGuard {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel it's probably helpful to copy the comment in your README here as well as people might copy/paste the code snippet without reading readme, e.g.:

To load torchscript model for mobile we need some special setup which is placed in `struct JITCallGuard` in this example. It may change in future, you can track the latest changes keeping an eye in our [pytorch android jni code]([https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/cpp/pytorch_jni_jit.cpp#L28)

also, maybe change "need some special setup" to a bit more context, e.g. "need set these guards because mobile build doesn't support features like autograd for smaller build size"

…content"


Example of using direct linking to pytorch_jni library from aar and updating android/README.md with the tutorial how to do it.

Adding `nativeBuild` dimension to `test_app`, using direct aar dependencies, as headers packaging is not landed yet, excluding `nativeBuild` from building by default for CI.

Additional change to `scripts/build_pytorch_android.sh`:

Skipping clean task here as android gradle plugin 3.3.2 exteralNativeBuild has problems with it when abiFilters are specified.

Will be returned back in the following diffs with upgrading of gradle and android gradle plugin versions.

[ghstack-poisoned]
…content"


Example of using direct linking to pytorch_jni library from aar and updating android/README.md with the tutorial how to do it.

Adding `nativeBuild` dimension to `test_app`, using direct aar dependencies, as headers packaging is not landed yet, excluding `nativeBuild` from building by default for CI.

Additional change to `scripts/build_pytorch_android.sh`:

Skipping clean task here as android gradle plugin 3.3.2 exteralNativeBuild has problems with it when abiFilters are specified.

Will be returned back in the following diffs with upgrading of gradle and android gradle plugin versions.

[ghstack-poisoned]
@IvanKobzarev IvanKobzarev requested a review from ljk53 June 18, 2020 06:10
…content"


Example of using direct linking to pytorch_jni library from aar and updating android/README.md with the tutorial how to do it.

Adding `nativeBuild` dimension to `test_app`, using direct aar dependencies, as headers packaging is not landed yet, excluding `nativeBuild` from building by default for CI.

Additional change to `scripts/build_pytorch_android.sh`:

Skipping clean task here as android gradle plugin 3.3.2 exteralNativeBuild has problems with it when abiFilters are specified.

Will be returned back in the following diffs with upgrading of gradle and android gradle plugin versions.

[ghstack-poisoned]
@IvanKobzarev IvanKobzarev mentioned this pull request Jun 18, 2020
…content"


Example of using direct linking to pytorch_jni library from aar and updating android/README.md with the tutorial how to do it.

Adding `nativeBuild` dimension to `test_app`, using direct aar dependencies, as headers packaging is not landed yet, excluding `nativeBuild` from building by default for CI.

Additional change to `scripts/build_pytorch_android.sh`:

Skipping clean task here as android gradle plugin 3.3.2 exteralNativeBuild has problems with it when abiFilters are specified.

Will be returned back in the following diffs with upgrading of gradle and android gradle plugin versions.

[ghstack-poisoned]
…content"


Example of using direct linking to pytorch_jni library from aar and updating android/README.md with the tutorial how to do it.

Adding `nativeBuild` dimension to `test_app`, using direct aar dependencies, as headers packaging is not landed yet, excluding `nativeBuild` from building by default for CI.

Additional change to `scripts/build_pytorch_android.sh`:

Skipping clean task here as android gradle plugin 3.3.2 exteralNativeBuild has problems with it when abiFilters are specified.

Will be returned back in the following diffs with upgrading of gradle and android gradle plugin versions.

Differential Revision: [D22118945](https://our.internmc.facebook.com/intern/diff/D22118945)

[ghstack-poisoned]
…content"


Example of using direct linking to pytorch_jni library from aar and updating android/README.md with the tutorial how to do it.

Adding `nativeBuild` dimension to `test_app`, using direct aar dependencies, as headers packaging is not landed yet, excluding `nativeBuild` from building by default for CI.

Additional change to `scripts/build_pytorch_android.sh`:

Skipping clean task here as android gradle plugin 3.3.2 exteralNativeBuild has problems with it when abiFilters are specified.

Will be returned back in the following diffs with upgrading of gradle and android gradle plugin versions.

Differential Revision: [D22118945](https://our.internmc.facebook.com/intern/diff/D22118945)

[ghstack-poisoned]
…content"


Example of using direct linking to pytorch_jni library from aar and updating android/README.md with the tutorial how to do it.

Adding `nativeBuild` dimension to `test_app`, using direct aar dependencies, as headers packaging is not landed yet, excluding `nativeBuild` from building by default for CI.

Additional change to `scripts/build_pytorch_android.sh`:

Skipping clean task here as android gradle plugin 3.3.2 exteralNativeBuild has problems with it when abiFilters are specified.

Will be returned back in the following diffs with upgrading of gradle and android gradle plugin versions.

Differential Revision: [D22118945](https://our.internmc.facebook.com/intern/diff/D22118945)

[ghstack-poisoned]
@facebook-github-bot
Copy link
Contributor

@IvanKobzarev merged this pull request in 52a2adb.

xwang233 pushed a commit to xwang233/pytorch that referenced this pull request Jun 20, 2020
…torch#39587)

Summary:
Pull Request resolved: pytorch#39587

Example of using direct linking to pytorch_jni library from aar and updating android/README.md with the tutorial how to do it.

Adding `nativeBuild` dimension to `test_app`, using direct aar dependencies, as headers packaging is not landed yet, excluding `nativeBuild` from building by default for CI.

Additional change to `scripts/build_pytorch_android.sh`:

Skipping clean task here as android gradle plugin 3.3.2 exteralNativeBuild has problems with it when abiFilters are specified.

Will be returned back in the following diffs with upgrading of gradle and android gradle plugin versions.

Test Plan: Imported from OSS

Differential Revision: D22118945

Pulled By: IvanKobzarev

fbshipit-source-id: 31c54b49b1f262cbe5f540461d3406f74851db6c
@facebook-github-bot facebook-github-bot deleted the gh/IvanKobzarev/33/head branch June 22, 2020 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants