[BUG] shell interpreter hardcode #!/bin/bash update to #!/usr/bin/env bash.

[cederom]

Description

• #!/bin/bash shell script interpreter is a Linux only syntax because Linux does not distinguish between /bin/sh and /bin/bash (/bin/sh is /bin/bash).
• Unix uses #!/usr/bin/env bash because bash may not be part of the system or it may be installed in the userland (i.e. /usr/local/bin/bash).
• All scripts using #!/bin/bash hardcode will work only on Linux, but not Unix (i.e. BSD).
• Please update the shell interpreter so scripts could also work on other OS than Linux (i.e. BSD).

Why I got here? Because Google does not release Android-NDK for FreeBSD. But FreeBSD can natively emulate Linux ELF, so tools for Linux can also work on FreeBSD.. unless they have Linux-only hardcodes. Please fix :-)

This problem is in the source code for a long time, and it is still in the master, for instance:
https://android.googlesource.com/a/platform/ndk/+/refs/heads/master/build/tools/ndk_bin_common.sh

Affected versions

r25, Canary

Canary version

270076ca936ed0dca16de84e302dde296aaf1ca4

Host OS

Linux

Host OS version

FreeBSD 13-STABLE

Affected ABIs

armeabi-v7a, arm64-v8a, x86, x86_64

Build system

ndk-build

Other build system

No response

minSdkVersion

20

Device API level

21

[jfgoog]

I agree we shouldn't be assuming /bin/bash. A potentially trickier problem is that we should also assume as little as possible about the capabilities of the shell -- no bash-specific features.

[cederom]

Thanks @jfgoog :-)

• The quick fix for now is to use #!/usr/bin/env bash in place of #!/bin/bash and this seems to be easy to automate :-)
• The safest common denominator is /bin/sh. But that may require additional work to review and rewrite the scripts.
• Although there are no NDK releases for FreeBSD, I can use Linux binaries on FreeBSD (yes FreeBSD can natively emulate Linux ELF), so some packages can be built that way, compiler binaries seems to work, but some components/libraries have hardcodes to work only on Linux (or macOS that adhered to Linux non-compatibilities).
• Linux is not Unix and vice-versa. Linux is Open-Source but Open-Source is not (only) Linux. There are other OS out there that are not only ommitted but also excluded by favorizing enforced non-standard Linux only solutions.
• Note that only on Linux /bin/sh = /bin/bash. On FreeBSD bash is not part of the base, but it can be easily installed to the userland. Even on macOS /bin/sh is not /bin/bash.

[jfgoog]

I did some work very related to this, which I haven't been able to finish:

https://android-review.googlesource.com/c/platform/ndk/+/2249853

As I recall, there weren't any major problems when I essentially replaced "bash" with "sh".

There's also this reference in external/shaderc/glslang that might need to be fixed upstream:

https://android.googlesource.com/platform/external/shaderc/glslang/+/refs/heads/master-ndk/CMakeLists.txt#368

[cederom]

• Thank you @jfgoog ! You rox :-)
• Is this NDK repo only available at Google? No GitHub location? How can I help testing? :-)
• Is it possible to build my own NDK natively on FreeBSD? We already have some android tools ported like adb, fastboot, etc :-)
• Why do I need this?
  • Because I got really sick and tired of completely unreliable JavaScript as the cross platform mobile development.
  • So I have switched to Python + Kivy :-) That works but requires NDK to bundle Python + Code + Libraries into a shared library and then to APK/AAB.
  • I can develop on FreeBSD but then I need to switch to macOS in order to release Android and iOS apps.
  • I would prefer to have at least Android build natively on the FreeBSD because this is my main workstation with far better hardware than MacBookPro (terribly slow 2015 hw with current 12.6 macOS and amazingly big XCode).
  • I have already cross compiled APK from my FreeBSD using Godot Game Engine, so this is possible.
  • Python + Kivy is more complex to build but far more versatile with possibility of creating and using python modules.
  • Some Kivy dependencies (and their builds named "recipes") already build but some does not yet build. When all recipes build the APK/AAB generation on FreeBSD should be complete (even when using Linux only provided binaries) :-)

[jfgoog]

The NDK repo is all open source. You can try building your own, but it's probably never been tried on FreeBSD, and I would be very surprised if it worked. Instructions are at https://android.googlesource.com/platform/ndk/+/refs/heads/master/docs/Building.md

[cederom]

The NDK repo is all open source. You can try building your own, but it's probably never been tried on FreeBSD, and I would be very surprised if it worked. Instructions are at https://android.googlesource.com/platform/ndk/+/refs/heads/master/docs/Building.md

Thanks, I saw that, but what is repo and gerrit? How this relates to git?

Update: Okay these are Google helper scripts [1], I will probably want to add them to FreeBSD ports or add them to a local user account :-)

[1] https://source.android.com/docs/setup/download#repo

[cederom]

In this repo script I would also suggest interpreter call:

#!/usr/bin/env python3

[cederom]

1. The method presented on [1] does not fetch antyhing, although repo seems to work correctly.
2. So I have git clone https://android.googlesource.com/platform/ndk as stated on [2].
3. python3 setup.py build seems to work with no problem.
4. After fixing the shell interpreter in ndk-build I have ERROR: Unknown host operating system: FreeBSD. That should be fixed easily too.
5. Then I would have to see the underlying compilers etc but we already have come cross compile tools for aarch64 on FreeBSD:

aarch64-binutils-2.37_4,1      GNU binary tools
aarch64-freebsd-sysroot-a2021.11.09 FreeBSD 12.2-RELEASE aarch64 sysroot
aarch64-gcc12-12.1.0           Cross GNU Compiler Collection for aarch64
aarch64-gcc6-6.5.0_3           Cross GNU Compiler Collection for aarch64
aarch64-gcc9-9.5.0             Cross GNU Compiler Collection for aarch64
aarch64-none-elf-binutils-2.37_4,1 GNU binary tools
aarch64-none-elf-gcc-11.3.0    GNU Compiler Collection for bare metal aarch64 cross-development
aarch64-rust-bootstrap-1.63.0  Create bootstrap compilers for building lang/rust
gnatcross-binutils-aarch64-2.27_1 Infrastructure for C/Ada FreeBSD cross-compiler (aarch64)
gnatcross-sysroot-aarch64-1_2  FreeBSD sysroots for C/Ada cross-compiler (amd64)

7. How could I provide patches to NDK? I have created a fork repo [3] this seems the easiest way? :-)

[1] https://android.googlesource.com/platform/ndk/+/refs/heads/master/docs/Building.md
[2] https://android.googlesource.com/platform/ndk/
[3] https://github.com/cederom/google-android-ndk

[cederom]

• Here goes the #!/bin/bash -> #!/usr/bin/env bash fix: https://github.com/cederom/google-android-ndk/tree/cederom-bash-scripts-interpreter-fix-20221017
• Please also verify the Docker part as this is Linux only solution and there may be no need to change of /bin/bash to /usr/bin/env bash.. I have no way to check :-)

[DanAlbert]

To be blunt (and I know you won't like this answer, but I think you'd like being led on even less): we will not be spending engineering time supporting BSD (unless you count macOS). We have staggeringly more users on Windows, macOS, and even Linux that we'd have to deprioritize to support (each) BSD. We'll accept patches for low-cost portability improvements (such as the one in the title of this bug), but won't be accepting anything that adds to our maintenance burden or misleads users (such as ndk-build claiming support for OSs which we don't support). If we're not able to fund BSD support (and we are not), anyone that's hoping to ship something is going to have a better time doing so from a supported platform than trying to muddle through the bugs that can and will arise from using an unsupported path.

With infinite resources, sure, we'd do this. We don't have that though so we have to pick our battles.

We are not the people that own repo. You'll need to raise those issues with the owner of that tool.

[cederom]

Hey there @DanAlbert :-)

1. Is this repo the official place to report bugs in Google Android NDK? If not where should I report and provide patches?
2. Are you Google employee assigned to handle issues reported here, including providing a fixes and accepting patches provided by the community? If not then I am sorry where should I send patches?
3. Issues that I have reported are trivial to fix, they just require proper use of applied tools to solve blocking problems and to make code portable, I have provided patches.
4. The recognition of FreeBSD is probably a result of my bug reported some time ago that was improperly implemented (FreeBsd -> FreeBSD, freebsd -> linux). Valid implementation enables using tools from a platform that you already support (Linux). No additional work from you is required just running the tests on provided patches. FreeBSD can natively emulate Linux ELF [1].
5. I can understand that Google may not be interested in providing binary releases to other OS than currently supported due to additional workload and costs, sure this is reasonable, but there should be no problem with the Open-Source community doing the work for you, right? :-)

[1] https://docs.freebsd.org/en/articles/linux-emulation/

[DanAlbert]

Is this repo the official place to report bugs in Google Android NDK? If not where should I report and provide patches?

Yes, this is the place to report bugs. No, it is not the place to send patches. The readme links our docs which will tell you how to send patches that we can merge.

Are you Google employee assigned to handle issues reported here, including providing a fixes and accepting patches provided by the community? If not then I am sorry where should I send patches?

For NDK bugs, yes, but like I said, we don't work on repo.

[cederom]

Thanks @DanAlbert for accepting one of my first patches at Google Gerrit, I will send further propositions directly over there.. I assume that would be the preferred way of propositions / discussions? :-)

[cederom]

@jfgoog are you also on the Google Gerrit? I can see some bash vs sh propositions are there as well.. if you need some testing on FreeBSD please let me know (and the preferred way of communication) :-)

[DanAlbert]

If you have a patch to send, yeah, just send patches. If you have a discussion that isn't about a specific code change, the discussions tab here is the right place. If you have bugs to report, that's this.

[cederom]

If you have a patch to send, yeah, just send patches. If you have a discussion that isn't about a specific code change, the discussions tab here is the right place. If you have bugs to report, that's this.

ACK! :-)

[cederom]

Thanks for accepting the patch into upstream @DanAlbert :-)