Skip to content

Commit

Permalink
Restructure the native module
Browse files Browse the repository at this point in the history
Consolidate all code into the src folder
  • Loading branch information
topjohnwu committed Jul 23, 2022
1 parent c7c9fb9 commit b9e89a1
Show file tree
Hide file tree
Showing 198 changed files with 51 additions and 44 deletions.
34 changes: 17 additions & 17 deletions .gitmodules
@@ -1,45 +1,45 @@
[submodule "selinux"]
path = native/jni/external/selinux
path = native/src/external/selinux
url = https://github.com/topjohnwu/selinux.git
[submodule "busybox"]
path = native/jni/external/busybox
path = native/src/external/busybox
url = https://github.com/topjohnwu/ndk-busybox.git
[submodule "dtc"]
path = native/jni/external/dtc
path = native/src/external/dtc
url = https://github.com/dgibson/dtc.git
[submodule "lz4"]
path = native/jni/external/lz4
path = native/src/external/lz4
url = https://github.com/lz4/lz4.git
[submodule "bzip2"]
path = native/jni/external/bzip2
path = native/src/external/bzip2
url = https://github.com/nemequ/bzip2.git
[submodule "xz"]
path = native/jni/external/xz
path = native/src/external/xz
url = https://github.com/xz-mirror/xz.git
[submodule "nanopb"]
path = native/jni/external/nanopb
path = native/src/external/nanopb
url = https://github.com/nanopb/nanopb.git
[submodule "mincrypt"]
path = native/jni/external/mincrypt
path = native/src/external/mincrypt
url = https://github.com/topjohnwu/mincrypt.git
[submodule "pcre"]
path = native/jni/external/pcre
path = native/src/external/pcre
url = https://android.googlesource.com/platform/external/pcre
[submodule "libcxx"]
path = native/jni/external/libcxx
path = native/src/external/libcxx
url = https://github.com/topjohnwu/libcxx.git
[submodule "zlib"]
path = native/jni/external/zlib
path = native/src/external/zlib
url = https://android.googlesource.com/platform/external/zlib
[submodule "parallel-hashmap"]
path = native/jni/external/parallel-hashmap
path = native/src/external/parallel-hashmap
url = https://github.com/greg7mdp/parallel-hashmap.git
[submodule "termux-elf-cleaner"]
path = tools/termux-elf-cleaner
url = https://github.com/termux/termux-elf-cleaner.git
[submodule "zopfli"]
path = native/jni/external/zopfli
path = native/src/external/zopfli
url = https://github.com/google/zopfli.git
[submodule "cxx-rs"]
path = native/jni/external/cxx-rs
path = native/src/external/cxx-rs
url = https://github.com/topjohnwu/cxx.git
[submodule "termux-elf-cleaner"]
path = tools/termux-elf-cleaner
url = https://github.com/termux/termux-elf-cleaner.git
7 changes: 4 additions & 3 deletions build.py
Expand Up @@ -211,6 +211,7 @@ def binary_dump(src, var_name):

def run_ndk_build(flags):
os.chdir('native')
flags = 'NDK_PROJECT_PATH=. NDK_APPLICATION_MK=src/Application.mk ' + flags
proc = system(f'{ndk_build} {flags} -j{cpu_count}')
if proc.returncode != 0:
error('Build binary failed!')
Expand All @@ -223,7 +224,7 @@ def run_ndk_build(flags):


def run_cargo_build(args):
os.chdir(op.join('native', 'rust'))
os.chdir(op.join('native', 'src'))
targets = set(args.target) & set(rust_targets)

env = os.environ.copy()
Expand All @@ -242,9 +243,9 @@ def run_cargo_build(args):
cxxbridge = op.join(local_cargo_root, 'bin', 'cxxbridge' + EXE_EXT)
mkdir(native_gen_path)
for p in ['base', 'boot', 'core', 'init', 'sepolicy']:
text = cmd_out([cxxbridge, op.join(p, 'src', 'lib.rs')])
text = cmd_out([cxxbridge, op.join(p, 'lib.rs')])
write_if_diff(op.join(native_gen_path, f'{p}-rs.cpp'), text)
text = cmd_out([cxxbridge, '--header', op.join(p, 'src', 'lib.rs')])
text = cmd_out([cxxbridge, '--header', op.join(p, 'lib.rs')])
write_if_diff(op.join(native_gen_path, f'{p}-rs.hpp'), text)

# Start building the actual build commands
Expand Down
15 changes: 4 additions & 11 deletions native/README.md
Expand Up @@ -4,31 +4,24 @@

Install the NDK required to build and develop Magisk with `./build.py ndk`. The NDK will be installed to `$ANDROID_SDK_ROOT/ndk/magisk`. You don't need to manually install a Rust toolchain with `rustup`, as the NDK installed already has a Rust toolchain bundled.

## Code Paths

- `jni`: Magisk's code in C++
- `jni/external`: external dependencies, mostly submodules
- `rust`: Magisk's code in Rust
- `src`: irrelevant, only exists to setup a native Android Studio project

## Build Configs

All C/C++ code and its dependencies are built with [`ndk-build`](https://developer.android.com/ndk/guides/ndk-build) and configured with several `*.mk` files scatterred in many places.
All C/C++ code and its dependencies are built with [`ndk-build`](https://developer.android.com/ndk/guides/ndk-build) and configured with several `*.mk` files scattered in many places.

The `rust` folder is a proper Cargo workspace, and all Rust code is built with `cargo` just like any other Rust projects.
The `src` folder is also a proper Cargo workspace, and all Rust code is built with `cargo` just like normal Rust projects.

## Rust + C/C++

To reduce complexity involved in linking, all Rust code is built as `staticlib` and linked to C++ targets to ensure our final product is built with an officially supported NDK build system. Each C++ target can at most link to **one** Rust `staticlib` or else multiple definitions error will occur.

We use the [`cxx`](https://cxx.rs) project for interop between Rust and C++. Although cxx supports interop in both directions, for Magisk, it is strongly advised to avoid calling C++ functions in Rust; if some functionality required in Rust is already implemented in C++, the desired solution is to port the C++ implementation into Rust and export the migrated function back to C++.
We use the [`cxx`](https://cxx.rs) project for Rust and C++ interop.

## Development / IDE

All C++ code should be recognized and properly indexed by Android Studio out of the box. For Rust:

- Install the [Rust plugin](https://www.jetbrains.com/rust/) in Android Studio
- In Preferences > Languages & Frameworks > Rust, set `$ANDROID_SDK_ROOT/ndk/magisk/toolchains/rust/bin` as the toolchain location
- Open `native/rust/Cargo.toml`, and select "Attach" in the "No Cargo projects found" banner
- Open `native/src/Cargo.toml`, and select "Attach" in the "No Cargo projects found" banner

Note: run `./build.py binary` before developing to make sure generated code is created.
6 changes: 5 additions & 1 deletion native/build.gradle.kts
Expand Up @@ -9,10 +9,14 @@ android {

externalNativeBuild {
ndkBuild {
path("jni/Android.mk")
path("src/Android.mk")
}
}

sourceSets.getByName("main") {
manifest.srcFile("src/AndroidManifest.xml")
}

defaultConfig {
externalNativeBuild {
ndkBuild {
Expand Down
1 change: 0 additions & 1 deletion native/rust/.gitignore

This file was deleted.

1 change: 1 addition & 0 deletions native/jni/.gitignore → native/src/.gitignore
@@ -1 +1,2 @@
test.cpp
target/
File renamed without changes.
12 changes: 6 additions & 6 deletions native/jni/Android.mk → native/src/Android.mk
Expand Up @@ -153,7 +153,7 @@ include $(BUILD_EXECUTABLE)
endif

ifdef B_TEST
ifneq (,$(wildcard jni/test.cpp))
ifneq (,$(wildcard src/test.cpp))

include $(CLEAR_VARS)
LOCAL_MODULE := test
Expand All @@ -176,7 +176,7 @@ LOCAL_MODULE := libpolicy
LOCAL_STATIC_LIBRARIES := \
libbase \
libsepol
LOCAL_C_INCLUDES := jni/sepolicy/include
LOCAL_C_INCLUDES := src/sepolicy/include
LOCAL_EXPORT_C_INCLUDES := $(LOCAL_C_INCLUDES)
LOCAL_SRC_FILES := \
sepolicy/api.cpp \
Expand All @@ -186,12 +186,12 @@ LOCAL_SRC_FILES := \
sepolicy/statement.cpp
include $(BUILD_STATIC_LIBRARY)

include jni/Android-rs.mk
include jni/base/Android.mk
include jni/external/Android.mk
include src/Android-rs.mk
include src/base/Android.mk
include src/external/Android.mk

ifdef B_BB

include jni/external/busybox/Android.mk
include src/external/busybox/Android.mk

endif
2 changes: 2 additions & 0 deletions native/src/AndroidManifest.xml
@@ -0,0 +1,2 @@
<!-- This file exists only to make Android Studio happy -->
<manifest/>
3 changes: 2 additions & 1 deletion native/jni/Application.mk → native/src/Application.mk
@@ -1,3 +1,4 @@
APP_BUILD_SCRIPT := src/Android.mk
APP_ABI := armeabi-v7a arm64-v8a x86 x86_64
APP_CFLAGS := -Wall -Oz -fomit-frame-pointer -flto
APP_LDFLAGS := -flto
Expand All @@ -10,7 +11,7 @@ APP_STRIP_MODE := --strip-all
# Busybox should use stock libc.a
ifdef B_BB
APP_PLATFORM := android-24
APP_LDFLAGS += -T jni/lto_fix.lds
APP_LDFLAGS += -T src/lto_fix.lds
ifeq ($(OS),Windows_NT)
APP_SHORT_COMMANDS := true
endif
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion native/rust/Cargo.toml → native/src/Cargo.toml
Expand Up @@ -23,4 +23,4 @@ panic = "abort"
strip = true

[patch.crates-io]
cxx = { path = "../jni/external/cxx-rs" }
cxx = { path = "../src/external/cxx-rs" }
4 changes: 2 additions & 2 deletions native/jni/base/Android.mk → native/src/base/Android.mk
Expand Up @@ -4,7 +4,7 @@ LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)
LOCAL_MODULE := libbase
LOCAL_C_INCLUDES := jni/include $(LOCAL_PATH)/include out/generated
LOCAL_C_INCLUDES := src/include $(LOCAL_PATH)/include out/generated
LOCAL_EXPORT_C_INCLUDES := $(LOCAL_C_INCLUDES)
LOCAL_EXPORT_STATIC_LIBRARIES := libcxx
LOCAL_STATIC_LIBRARIES := libcxx
Expand All @@ -28,5 +28,5 @@ LOCAL_MODULE := libcompat
LOCAL_SRC_FILES := compat/compat.cpp
# Fix static variables' ctor/dtor when using LTO
# See: https://github.com/android/ndk/issues/1461
LOCAL_EXPORT_LDLIBS := -static -T jni/lto_fix.lds
LOCAL_EXPORT_LDLIBS := -static -T src/lto_fix.lds
include $(BUILD_STATIC_LIBRARY)
3 changes: 3 additions & 0 deletions native/rust/base/Cargo.toml → native/src/base/Cargo.toml
Expand Up @@ -3,5 +3,8 @@ name = "base"
version = "0.1.0"
edition = "2021"

[lib]
path = "lib.rs"

[dependencies]
cxx = "1.0.69"
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
1 change: 1 addition & 0 deletions native/rust/boot/Cargo.toml → native/src/boot/Cargo.toml
Expand Up @@ -5,6 +5,7 @@ edition = "2021"

[lib]
crate-type = ["staticlib"]
path = "lib.rs"

[dependencies]
base = { path = "../base" }
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
1 change: 1 addition & 0 deletions native/rust/core/Cargo.toml → native/src/core/Cargo.toml
Expand Up @@ -5,6 +5,7 @@ edition = "2021"

[lib]
crate-type = ["staticlib"]
path = "lib.rs"

[dependencies]
base = { path = "../base" }
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Submodule pcre updated from 000000 to 8e1268
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Submodule zlib updated from 000000 to 3a0aa2
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
1 change: 1 addition & 0 deletions native/rust/init/Cargo.toml → native/src/init/Cargo.toml
Expand Up @@ -5,6 +5,7 @@ edition = "2021"

[lib]
crate-type = ["staticlib"]
path = "lib.rs"

[dependencies]
base = { path = "../base" }
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
1 change: 0 additions & 1 deletion native/src/main/AndroidManifest.xml

This file was deleted.

File renamed without changes.
File renamed without changes.
File renamed without changes.
Expand Up @@ -5,6 +5,7 @@ edition = "2021"

[lib]
crate-type = ["staticlib", "rlib"]
path = "lib.rs"

[dependencies]
base = { path = "../base" }
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.

0 comments on commit b9e89a1

Please sign in to comment.