Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Android package management for the NDK #916

Open
DanAlbert opened this issue Feb 22, 2019 · 22 comments
Assignees

Comments

@DanAlbert
Copy link
Collaborator

@DanAlbert DanAlbert commented Feb 22, 2019

This bug tracks the Easier access to common open-source libraries item on our roadmap.

There are many other commonly-used libraries (such as Curl and BoringSSL) that are currently difficult to build/package, let alone keep updated. We should offer (a) a tool to build open source projects, (b) a repository of prebuilts, (c) a command-line tool to add prebuilts to an ndk-build/cmake project, and (d) Studio integration to add prebuilts via a GUI.

We currently have cdep, but it doesn't have a very large corpus and could use some build system integration polish. If not cdep, adding support for exposing native libraries from AARs would be an alternative.

Some questions about this came up on another bug, specifically in what we were planning to use and why, so I wanted to publish the requirements that we had for this, show what we'd considered, and explain the current plan.

Note that this isn't the full design document. I've spared the nitty gritty details since I don't think they're particularly interesting at this time, but if anyone has questions just ask and I can elaborate :)

Requirements

  1. Works on Linux, Darwin, and Windows build hosts.
  2. Supports arbitrary build systems.
  3. Able to deliver projects employing a variety of architectures:
    • Header only libraries.
    • Single library projects.
    • Multi library packages.
  4. Able to describe complex library use:
    • Export include paths to library users.
    • Export cflags and ldflags to library users.
    • Complex include/link ordering requirements such as those found the C++ standard library.
  5. Supports binary distributions.
  6. Able to fetch from arbitrary sources.
  7. Support IDE integration.
  8. Supports cross package dependencies.
  9. Support for non-Android targets.
  10. Support for multi-language projects.

Options

With these requirements, none of the existing options we know of will work as-is. We considered cdep, AAR (Android packages via Maven), vcpkg, and hunter. cdep is C/C++ only, which means packages like GTestJNI can't be distributed with it. AAR doesn't support C/C++ well (it supports C/C++ implementation, but the interface to the library must be Java), which obviously won't work for the NDK. vcpkg doesn't seem to support more complex build system features such as exporting required flags to dependents. Hunter is CMake only.

Of these options, AAR seems like our best bet. It doesn't meet the single most obvious need (being able to actually expose a C++ interface), but unlike the other options it's the only one that could be reasonably modified to fit these requirements. It also is the extant method for distributing Java dependencies for Android, and reusing existing infrastructure is a big plus.

Design

The plan is to build some low level tooling (which I'm currently calling Prefab) for generating build system integration for prebuilt packages and some metadata describing usage requirements and then extending gradle and AAR to make use of that.

When I say low level, what I mean is that Prefab only solves the build system integration part of the problem. For an Android gradle project, gradle is still responsible for resolving and fetching dependencies. When gradle fetches an AAR that contains Prefab artifacts, it extracts those to a directory and calls prefab to have it generate build system integration. The generated output is then passed to ndk-build or CMake to be used as you would any other dependency. This means that if your organization uses some other mechanism for distributing dependencies, Prefab artifacts can be distributed that way as well (including something as simple as a git submodule).

For example, your build.gradle would contain something like the following:

dependencies {
    implementation 'com.google.gtest:gtest:1.0'
    implementation 'com.google.gtestjni:gtestjni:1.0'
}

and your CMakeLists.txt would contain:

find_package(gtest 1.0 REQUIRED MODULE)
find_package(gtestjni 1.0 REQUIRED MODULE)

# Additional targets for building the actual app library...

add_library(apptest
  SHARED
    src/test/cpp/app_test.cpp
)

target_link_libraries(apptest
  PRIVATE
    app
    gtest::gtest
    gtestjni::gtestjni
)

And that's it. Gradle will fetch the dependencies, call Prefab to generate the CMake package definitions, and find_package will find the dependencies.

Build system integration

Build system integration is provided by plugins for Prefab, and we'll include CMake and ndk-build plugins by default. If you're using a custom build system, you can provide a path to a plugin jar on the command line to teach Prefab how to generate what you need.

Non-Android targets

Support for non-Android targets will be simple to add (basically just a class recognizing and perhaps disambiguating libraries for the platform, such as Android needs to do with multiple ABIs, minSdkVersion targets, etc). We're only planning on implementing support for Android, but patches adding other platforms are welcome.

So, that's the rough outline. If I've missed any major requirements that make this approach not viable for your project, please speak up. We're still in the very early phases of this so it's not too late for large changes.

@Zingam

This comment has been minimized.

Copy link

@Zingam Zingam commented Feb 22, 2019

Could you post a link to AAR? A simple google search for "aar package manager" doesn't reveal anything.

@DanAlbert

This comment has been minimized.

Copy link
Collaborator Author

@DanAlbert DanAlbert commented Feb 22, 2019

https://developer.android.com/studio/projects/android-library

tl;dr it's like a jar, but has Android specific pieces. It's distributed via maven like a jar would be.

@alexcohn

This comment has been minimized.

Copy link

@alexcohn alexcohn commented Feb 24, 2019

@DanAlbert, I believe that the androidNativeBundle gradle plugin already implements this for AAR, at least partially.

@Zingam

This comment has been minimized.

Copy link

@Zingam Zingam commented Mar 8, 2019

So this is basically an Android only solution? It's not meant to provide packages targeting Linux, Mac or Windows? I hoped that AAR were something different than Android Archive (AAR) that I didn't know about.

@DanAlbert

This comment has been minimized.

Copy link
Collaborator Author

@DanAlbert DanAlbert commented Mar 8, 2019

tl;dr: No, it isn't Android specific (see that "Non-Android targets" section of the original post).

To elaborate and hopefully clear up the confusion, it depends on what you mean by "this". There are two separate pieces here:

  1. Distribution mechanism (not the topic of this bug, could be anything)
  2. Build system integration (the topic of this bug, called Prefab, not Android specific)

There are a large number of good distribution mechanisms out there, so I figure we're better of leaning on those to do what they do best.

Build system integration in the form that we want it, however, doesn't have many good options. Requirements 2, 4, and 10 in particular ruled out the existing options.

This bug describes a new tool (Prefab) that generates build scripts for a set of prebuilt libraries and headers. It plays no part in dependency resolution or distribution.

So, no, Prefab is not Android specific. As the original post states, Android support will be the first implemented, and I personally don't have plans to add other platforms (though Linux might be added just so I can prove to myself that what I have is easily extended to support additional platforms), but we'll accept patches for other platforms (within reason; if you want VAX support you can maintain your own fork).

@Firefly35

This comment has been minimized.

Copy link

@Firefly35 Firefly35 commented Mar 20, 2019

Considering native builds, did you have a look to conan ?

@DanAlbert

This comment has been minimized.

Copy link
Collaborator Author

@DanAlbert DanAlbert commented Mar 20, 2019

From a quick read, that seems to check must of the boxes. I'll have a more thorough look later today to see if that can work for us. My initial concern is that it doesn't support multi language packages, but that may be something we can add.

Thanks for the pointer. Hadn't seen that one before.

@DanAlbert

This comment has been minimized.

Copy link
Collaborator Author

@DanAlbert DanAlbert commented Mar 20, 2019

Huh. Having now read further, this actually looks a lot like what I'd designed, but even includes some of the stretch goals I haven't discussed here (source distribution, namely). The only requirement we have that I don't think this covers is the ability to play well with mixed Java/C++ packages, but at a glance that might still be doable?

Conan looks incredibly well thought out and executed, so I'm going to play with this a bit and see if it can be made to fit all our requirements.

@DanAlbert

This comment has been minimized.

Copy link
Collaborator Author

@DanAlbert DanAlbert commented Mar 21, 2019

It looks like Conan can also be made to package a jar quite easily. Seems like the only thing we would need to do would be to build a gradle plugin that can handle Conan (it might be a part of the Android gradle plugin, if that integrates better).

@Firefly35

This comment has been minimized.

Copy link

@Firefly35 Firefly35 commented Mar 21, 2019

On my side, I started the development of native dependencies management some years ago, and started to define coherent compile/link rules for cross-platform development, before conan gets stable.
The build rules handle for instance static/dynamic runtime issues on windows.
The next release will also provide a simple macro to switch to brew llvm on Mac OSX platforms.

Nowadays, the system (recently renamed remaken) has evolved (a release will come soon - the main tool is in the finalization step) to a meta-packaging system.

I integrated conan, vcpkg, but also system's dependent packagers (apt, yum, brew, chocolatey) and the binary format defined at the beginning of the project.

Conan was easy to integrate to remaken, and nowadays remaken offers the versatility to choose dependencies among the various binaries available, and depending on the project specific needs.

If you 're interested, I will provide the link to the (future) github repository hosting the tool.

For the moment, the qmake build rules are available from https://github.com/b-com-software-basis/builddefs-qmake.
Cmake rules are on the go.

The remaken tool roadmap will include cmake compile and link flags generation, remaken format packager from a build folder ...

@Zingam

This comment has been minimized.

Copy link

@Zingam Zingam commented Mar 21, 2019

Whatever solution you decide on, please, don't require external dependencies - like Python, Java...

@DanAlbert

This comment has been minimized.

Copy link
Collaborator Author

@DanAlbert DanAlbert commented Mar 21, 2019

The top two choices at this time will require one of those. The NDK/SDK already depend on and include Java and Python, so this isn't anything new.

@android android deleted a comment from Oleh-Kravchenko Aug 13, 2019
@DanAlbert

This comment has been minimized.

Copy link
Collaborator Author

@DanAlbert DanAlbert commented Aug 21, 2019

Wanted to give a quick update since it's been a while:

Spent some time evaluating other options and came to the conclusion that the original plan (discussed in the first post of this thread) was going to offer the best UX for Android.

The tooling to make this work and the gradle plugin changes to make it a transparent part of an AAR have been written. There's still a fair amount of polish that needs to be done on both before it's shippable, but we have a working proof of concept.

The current hope is that this will be available alongside the Studio 3.7 release cycle, which will likely go to stable in the first half of next year (I'm not sure when the canaries will first be available, or which stage will be the first one with these features available). Hoping to publish the source for the tooling far ahead of that, which will include design info and usage docs.

@Zingam

This comment has been minimized.

Copy link

@Zingam Zingam commented Aug 22, 2019

What I find worrisome about creating a completely new delivery system is who is going to add and maintain new packages? I doubt that the audience of NDK is that huge. Would google add a few popular libraries and that would be it.
Unless you also plan to extend it to other systems (not just Android) like Windows, MacOS, IOS, Linux and thus competing with the already established package manager it will also add more maintenance burden to cross-platform projects.
I am always fond of new shiny stuff and project maintenance is also the most annoying part.

How will this plan fit with this: #1011
What do you think about all that? Is this Android only project?

@DanAlbert

This comment has been minimized.

Copy link
Collaborator Author

@DanAlbert DanAlbert commented Aug 22, 2019

Is this Android only project?

That's covered in the original post. Also #916 (comment)

How will this plan fit with this: #1011

Orthogonal.

disigma pushed a commit to wimal-build/ndk that referenced this issue Sep 25, 2019
Gradle needs to be able to specify a dependency import path for Prefab
modules. Add NDK_GRADLE_INJECTED_IMPORT_PATH to support this.

This could instead be a more generic NDK_ADDITIONAL_IMPORT_PATHS, but
that's probably of limited use since Gradle will clobber the user's
flags with its own.

Test: ./run_tests.py --rebuild
Bug: android/ndk#916
Change-Id: I18f9ff2830846d9ea63130d6949c9d199b72db60
@NewProggie

This comment has been minimized.

Copy link

@NewProggie NewProggie commented Oct 17, 2019

This is very good news, as all dependencies in a CMake-based project should always use ˋfind_packageˋ for resolving dependencies. I have written about this topic before as well and demonstrated a straight forward approach for Android- as well as external dependencies here in case you are interested. [1]

[1] http://kai-wolf.me/cpp/android/cmake/gradle/2019/02/18/working-with-cpp-in-android-effectively/

@sinhpn92

This comment has been minimized.

Copy link

@sinhpn92 sinhpn92 commented Nov 5, 2019

Many commonly-used libraries is very need to use (cURL, openSSL, boringSSl, etc,... ). But hard to build for use in ndk.

@DanAlbert

This comment has been minimized.

Copy link
Collaborator Author

@DanAlbert DanAlbert commented Nov 5, 2019

BoringSSL is particularly annoying (you need a go toolchain to build it...) It's not particularly useful for apps either, so that's not one we intend to publish (but anyone that wants to do so can of course do so themselves). cURL and OpenSSL are ones that we've already prepped for this, along with jsoncpp, since it was easy and helped me make a demo with the other two.

@DanAlbert

This comment has been minimized.

Copy link
Collaborator Author

@DanAlbert DanAlbert commented Nov 5, 2019

The underlying implementation of this is now up on GitHub: https://github.com/google/prefab (expect a lot of broken URLs; I only just published this so I haven't setup the matching github.io page yet). While it's a big step closer to finishing off this bug, it's just one piece. We still need to:

  1. Finish up enough of the TODOs that we can publish something to Maven for the Android Gradle Plugin to pick up.
  2. Submit the AGP support.
  3. Fix AGP to automatically package CMake import targets (this is an pre-existing issue, but we need it to be fixed for this to have good UX in Studio with CMake).
  4. Build up a corpus of packages to make this worth using at launch (currently have OpenSSL, cURL, and jsoncpp ready, but will probably add things like libpng, libjpeg, zlib, etc).
  5. Ship it!

Should still be on track to have this available in AGP 4.0.

@sonicdebris

This comment has been minimized.

Copy link

@sonicdebris sonicdebris commented Nov 10, 2019

Out of curiosity, which are the reasons that made you discard Conan in the end?

@H-G-Hristov

This comment has been minimized.

Copy link

@H-G-Hristov H-G-Hristov commented Nov 11, 2019

Is there a wishlist/roadmap of which libraries you (Google) will support?

As a part of the NDK users are multimedia developers I'd expect that at least the most popular multimedia/graphics/audio libraries to be included: freetype2, SDL2, OpenAL Soft, opus, vorbis, ffmpeg, some VP9, av1 libraries, etc

@DanAlbert

This comment has been minimized.

Copy link
Collaborator Author

@DanAlbert DanAlbert commented Nov 11, 2019

Out of curiosity, which are the reasons that made you discard Conan in the end?

Seamless integration with Gradle was one reason.

The largest concern though is that with conan the user's machine has to build the packages, not just fetch them. The building part is one of the things we're trying to avoid doing. It adds unnecessary time to the user's build and almost certainly will result in "works on my machine". With how much code out there still uses autoconf and similar tools, that would likely be a large problem for Windows users. I don't think this has been much of a problem with Conan so far because primarily the people building things on Windows are also the people building things for Windows. For Android development there would likely be many Linux-only packages that would not build on Windows.

We also are able to do a bit better with ABI matching than Conan can. Conan requires an exact match of the build settings across modules, so the common case for Android where a library is built for android-16 but the app is targeting 21 would not work. With Prefab we can currently match that, and by the end of the week it should be improved to allow finding a best match if there's more than one valid match (google/prefab#8).

Is there a wishlist/roadmap of which libraries you (Google) will support?

I've already taken care of OpenSSL, cURL, and jsoncpp. The cURL/OpenSSL use case was our benchmark for "have we done this right at all", and jsoncpp tied those together into a presentable sample.

Someone I spoke with at ADS requested LuaJIT, and @alexcohn has a big list for us over at #456 (comment).

The Prefab bug tracker is going to be the home for package requests. I haven't filed any of those I just mentioned yet, but feel free to start populating it: https://github.com/google/prefab/issues/new/choose (use the "Package request" template).

We're going to have to think about which packages we commit to, since each one will have a recurring cost that comes with it. We fortunately do have some tooling that helps us keep things up to date in AOSP that I'm hoping we can extend to cover this use case, but I'm particularly worried about testing these.

That is to say, please file those requests, but be patient while we figure out what we can and can't afford to support right away.

Based on what I've seen people struggle with on SO, I think that from your list SDL2 and ffmpeg are definitely two that we want to support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.