[CI] CI for i686-linux-android is broken

[JohnTitor]

Currently, aarch64-unknown-linux-android and i686-linux-android targets are broken.
The logs:

• aarch64-unknown-linux-android: it's now resolved but some it seems API version is old somehow.
• i686-linux-android

We should investigate the cause and land a fix.

[rupansh]

Consider using a newer toolchain for android.
Current CI uses API 21's toolchain. These functions were added in later versions of android to the TC.
According to headers in sysroot/usr/include of the latest ndk

API 23 Functions:
openpty, forkpty, login_tty, sethostname, sigtimedwait, fmemopen, open_memstream, open_wmemstream, clock_getcpuclockid

API 24 Functions:
lockf,preadv64,pwritev64, getifaddrs, freeifaddrs, getgrgid_r, getgrnam_r

[JohnTitor]

Current CI uses API 21's toolchain.

Hm, it's weird. IIUC we use API 24 for arm/aarch64 and API 28 for i686/x86_64, am I missing something?

[rupansh]

Current CI uses API 21's toolchain.

Hm, it's weird. IIUC we use API 24 for arm/aarch64 and API 28 for i686/x86_64, am I missing something?

Did you recently change the CI? It was using android21 with azure pipelines 12 days ago.

[JohnTitor]

No. Here's our config for Android NDK:

libc/ci/android-install-ndk.sh

Lines 5 to 34 in 2d044d9

	NDK=android-ndk-r21d
	wget --tries=20 https://dl.google.com/android/repository/${NDK}-linux-x86_64.zip
	unzip -q ${NDK}-linux-x86_64.zip
	case "$1" in
	arm)
	arch=arm
	api=24
	;;
	armv7)
	arch=arm
	api=24
	;;
	aarch64)
	arch=arm64
	api=24
	;;
	i686)
	arch=x86
	api=28
	;;
	x86_64)
	arch=x86_64
	api=28
	;;
	*)
	echo "invalid arch: $1"
	exit 1
	;;
	esac;

It was using android21 with azure pipelines 12 days ago.

You mean r21d is API 21?

[rupansh]

No. Here's our config for Android NDK:

libc/ci/android-install-ndk.sh

Lines 5 to 34 in 2d044d9

	NDK=android-ndk-r21d
	wget --tries=20 https://dl.google.com/android/repository/${NDK}-linux-x86_64.zip
	unzip -q ${NDK}-linux-x86_64.zip
	case "$1" in
	arm)
	arch=arm
	api=24
	;;
	armv7)
	arch=arm
	api=24
	;;
	aarch64)
	arch=arm64
	api=24
	;;
	i686)
	arch=x86
	api=28
	;;
	x86_64)
	arch=x86_64
	api=28
	;;
	*)
	echo "invalid arch: $1"
	exit 1
	;;
	esac;

It was using android21 with azure pipelines 12 days ago.

You mean r21d is API 21?

The old build is no longer opening on azure pipelines but I remember it was using aarch64-linux-android21-clang
Perhaps I am remembering wrong. My bad!

[qwandor-google]

I also noticed that it was using aarch64-linux-android21-clang, while the other Android architectures use x86_64-linux-android28-clang and armv7a-linux-androideabi24-clang. I couldn't see where these version numbers come from though, as the android-install-ndk.sh script does indeed seem to be installing a newer version.

[qwandor-google]

Where can I find instructions on how to run the CI tests locally, so I can try to debug this without having to push to a PR every time?

[JohnTitor]

You can run the scripts like CI on your local but I wouldn't recommend it.
I think we have two simple ways to debug CI:

1. Run GHA on your local with https://github.com/nektos/act
2. Run GHA on your fork repo with a tweak to run Android jobs only (I'm doing it on Fix aarch64-linux-android target CI #1927 with try builder)

[JohnTitor]

So, after some debugging, I think rust-lang/cc-rs#495 is the cause. If we set cc = "=1.0.52", the test passes: https://github.com/rust-lang/libc/runs/1266137760
I'm going to pin cc crate for now, but it'd be great if we could introduce an actual fix for it (I'll file an issue on that repository later).