Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
make all toolchains standalone toolchains #780
This has been discussed several times, but we haven't had a bug filed for this yet. Rather lengthy since I wanted to explain our thinking on some of these decisions since I suspect there will be many questions. Don't hesitate to chime in if something in here isn't clear and speak up if you have specific concerns. I have a prototype of this that passes all of the NDK's tests so it has at least proven feasible, but I'd be very interested in hearing about any specific issues people think need to be addressed that I've missed.
Properly compiling and linking code for Android is much harder than it needs to be. Compilers must be invoked with a specific set of flags that may change over time, defaults change, and if not using a build system that supports the NDK you need to use a standalone toolchain, which many people don't seem to discover.
Instead, we can hoist most of this logic into the compiler driver (now that we have only one compiler driver to support). The driver can predict the location of the sysroot relative to its own location, and we can install binutils alongside clang so it is automatically picked up as well.
We can also do the same for the libc++ headers and libraries, though since the compiler will always link the STL after all of the user's specified libraries it is not possible to avoid issues like #379 when the dependencies are broken (if the dependencies are all correct everything works fine, but there are a lot of broken prebuilts out there that ndk-build currently has no problems with, though cmake and standalone toolchains do). There is not much that can be done about this without having the compiler driver (or the linker) re-order the user's inputs, which is not something any compiler/linker I know of does.
The various default build/linker flags Android uses ought to also be part of the clang driver. This includes things like
With all of the above taken care of, we (and more importantly, third-party build system maintainers) can remove a bunch of code the is duplicated throughout the various systems. The
Making these changes does necessitate making some changes to the layout of the NDK. Clang (and GCC, FWIW, not that we use that any more) already had logic to find binutils/sysroots/etc based on its own install directory. All we need to do to take advantage of it is to:
Getting Clang to use the various NDK tool/library paths as-is would have been a much more invasive change to Clang and prevented us from sharing as many code paths with the other Clang targets, making it much more fragile and less likely to be accepted (the NDK's layout is very inconsistent currently; just explaining things like "well the arm binutils is in arm-linux-androideabi-4.9, even though it's not GCC 4.9, and not even GCC for that matter, btw the x86 one is just x86-4.9 rather than being triple-prefixed for some reason" seems like it would make these changes difficult to upstream).
As such we have a few options to consider as to how we approach this problem logistically:
We're not going to do this. It's an option, but the only benefit is that it doesn't increase the size of the NDK, but it does make for a ton of migration pain.
This has the advantage of having the easier migration story. We'll keep both around until we can remove the old one without causing undue pain (in practice this means "until a gradle plugin supporting the new form is available in the stable channel", at which point most other build systems that try to keep up to date have also been fixed). The disadvantage is that it roughly doubles the size of the NDK. I don't have exact numbers right now, but the new layout doesn't need to have an identical libc.a copied into each API level (itself an affordance for old build systems) or duplicate things like make/python. It's still probably a significant size increase.
This saves us some space (
There's an additional downside here in that using scripts to save space comes at a build-time cost. On Linux/Darwin the cost of
I personally prefer option 2 since it has the fewest adoption speedbumps, but ~50% install size increase compared to option 3 is not insignificant. I don't think it's worth it, but I could be convinced otherwise if people feel strongly about it (though do remember that the increase is temporary; in a few releases it will be back to its pre-r19 size, plus or minus any unrelated changes).
Lifting more default flags (Android-wide requirements, not best practices, probably) into the driver is being tracked as #812.
An interesting idea, but in practice I think this is roughly equivalent to option 1. Instead of "do you have r18 or r19?", the question for each build system becomes, "do you have r18, r19 type 1, or r19 type 2?" (and then stretch that question out until the transition period closes and we go back to having only one distribution). It wouldn't decrease the disruption faced by users, but it would exacerbate the disruption for build system maintainers. I don't think that is worth saving 1GB of install size.
It also would require us maintaining duplicate copies of the CMake toolchain and ndk-build throughout the transition (ditto for any third-party build system). That means more bugs, as well as more unaddressed bugs.
Different team, but I believe they're working on it. To be clear, what I actually meant was update our CMake toolchain file, not the builtin cmake implementation. That is a different bug that will be dealt with after this one.
Why can't we just make stub binaries instead of stub batch scripts? We can have a dozen copies of the same executable that just executes the correct one. There's three architectures to support, but mac and linux are the same, and windows is not that different. It'd be like 5 lines of C++, so it's not like it'd be that bad even if they were completely alien.
The MSVCRT/Mingw emulation of
Shell scripts seem fine on Unix. I think most of the trouble from stubs is on Windows, right?
Issues with batch files I can think of:
FWIW: A strip batch file does break Gradle:
I think the binaries could be linker scripts pointing to the right place, and the headers are probably small enough to not matter. (or they could be "#include "../../../../", if the size of the headers is actually a problem)
Install a copy (we can't remove the old paths until gradle has adapted to the new layout) of the toolchain/sysroot/etc to $NDK/toolchain such that it can be invoked without the build system needing to juggle --sysroot/-gcc-toolchain/etc. This follows the second approach described on android/ndk#780. That is it installs a full copy of the toolchain rather than installing scripts to refer to the old toolchain during the transition period. Test: ./checkbuild.py && ./run_tests.py Bug: android/ndk#780 Change-Id: I8d8f10ea6c95378937ac54e440c5e1a9cb3bbeaa