make can't find utilities while compiling gem ffi

[gouravkhunger]

Problem description

I am compiling ruby and make using a custom prefix. The bootstraps are working correctly and I can see the binaries work

I can also install some gems without a problem. But when I try to install the ffi gem, make doesn't seems to be able to find core utilities like ls, sh, rm, etc.

Here's the output log:

current directory: /data/data/xyz.jekyllex/files/usr/lib/ruby/gems/3.1.0/gems/ffi-1.15.5/ext/ffi_c
/data/data/xyz.jekyllex/files/usr/bin/ruby -I /data/data/xyz.jekyllex/files/usr/lib/ruby/3.1.0 -r ./siteconf20220807-4690-m3xlro.rb extconf.rb
checking for ffi_prep_closure_loc() in -lffi... yes
checking for ffi_prep_cif_var()... yes
checking for ffi_raw_call()... yes
checking for ffi_prep_raw_closure()... yes
checking for whether -pthread is accepted as LDFLAGS... yes
creating extconf.h
creating Makefile

current directory: /data/data/xyz.jekyllex/files/usr/lib/ruby/gems/3.1.0/gems/ffi-1.15.5/ext/ffi_c
make DESTDIR\= clean
make: rm: No such file or directory
make: [Makefile:193: clean-static] Error 127 (ignored)
make: /data/data/xyz.jekyllex/files/usr/bin/sh: No such file or directory
make: [Makefile:176: clean] Error 127 (ignored)

current directory: /data/data/xyz.jekyllex/files/usr/lib/ruby/gems/3.1.0/gems/ffi-1.15.5/ext/ffi_c
make DESTDIR\=
make: echo: No such file or directory
make: *** [Makefile:245: AbstractMemory.o] Error 127

make failed, exit code 2

I found that ls, rm, mv, those basic features aren't symlinked:

Even when I have dash and coreutils in my custom build-bootstraps.sh file:

PACKAGES+=("dash")
PACKAGES+=("make")
PACKAGES+=("ruby")
PACKAGES+=("libllvm")
PACKAGES+=("binutils")
PACKAGES+=("coreutils")
PACKAGES+=("pkg-config")

The built SYMLINKS.txt also has them symlinked to coreutils:

(./bin/sh is also symlinked to dash)

What could be the potential problem?

What steps will reproduce the bug?

• Use custom $PREFIX for building bootstrap with the packages listed above.
• After, boostrap install process, coreutils won't work even when symlinks are there.

What is the expected behavior?

No response

System information

I don't use termux-info in my bootstrap. But I am testing on an aarch64 emulator if that matters.
I generated bootstraps through GitHub Actions.

[gouravkhunger]

Actually, installing ffi using gem install ffi works on the official Termux app. Just make sure to install binutils besides make and ruby.

I can't seem to get it working on custom $PREFIX even with required packages installed.

[Grimler91]

What's the output of file $PREFIX/bin/coreutils?

strace should be able to tell what file it is that is actually missing: strace -o ls.log ls

[gouravkhunger]

@Grimler91 The package that has file and strace binaries is probably not there in my bootstrap:

[Grimler91]

Well, would help if you build and install them then

[gouravkhunger]

@Grimler91 I tried building and installing a bootstrap for package file and strace, still doesn't work.

I do see strace in bin/

Am I compiling the wrong packages?

EDIT: The SYMLINKS.txt doesn't seem to include symlink for ./bin/strace probably that's the reason?

[licy183]

Hello, it seems that you are using the shell from /system/bin/sh. Have you set your PATH environment variable properly? Try setting PATH=YOUR_CUSTOM_PREFIX.

[gouravkhunger]

I think the coreutils binary was never moved to the bin folder during bootstrap installation process.

Screenshot of files in emulator:

But when I extract the zip I can see the coreutils binary. Some problem is there during inflating the file to desired location ig.

@licy183 where do I set the PATH? On the android app or during some build step 🤔

Thanks for the help 🙌

[truboxl]

Please run the command env in the app and paste the output here

[chyn0907]

How do you get this shell? Are you using adb shell? If you are using adb, to test the bootstrap, you need root permission and some command like the following.

BASE_DIR="/data/data/xyz.jekyllex/files"
su $(stat -c %u $BASE_DIR)
export PREFIX="$BASE_DIR/usr"
export HOME="$BASE_DIR/home"
export TMPDIR="$BASE_DIR/usr/tmp"
export LD_LIBRARY_PATH="$PREFIX/lib:$LD_LIBRARY_PATH"
export PATH="$PREFIX/bin:$PATH"
export TZ="UTC"
export LANG="en_US.UTF-8"
export SHELL="$PREFIX/bin/bash"
exec "$SHELL"

Then you will get a shell like Termux.

[gouravkhunger]

@truboxl As I told, my bootstrap only has specific binaries needed for my use case. The env binary is not there, could you please tell the package to be included to have the env binary setup?

@chyn0907 I am using an emulator from Android studio. I cloned the termux app repository and changed the package instances as per my need (just for testing my bootstraps, won't use termux app in the final app as it is).

[licy183]

env is a sym-link to coreutils in Termux, but you could also use the one from toybox like /system/bin/env.
I think it is abnormal that the shell from the Android application starts with /system/bin/sh rather than $PREFIX/bin/sh, which is the default behavior of the failsafe session, but I have no idea why this happens. So, if it is convenient, would you please provide the source code or provide a patch file between the origin termux source code and your fork?

[gouravkhunger]

@licy183 I just changed com.termux instances to xyz.jekyllex across the source files and the package folder names. But I did this some 3-4 months ago so it might not be up to date as per the new app. The patch would become so large, but basically there's change of package name only. I haven't stored it on any remote repo.

Regarding the env symlink - that's the problem. I have coreutils there in my bootstrap. But it is not being installed correctly when I do a fresh install on the emulator. When I extract the files from bootstrap zip, I see coreutils binary as well as symlinks in SYMLINKS.txt file. But when I explore the content of emulator, all other binaries are there (the reason why ruby and make work) except coreutils and dash.

[gouravkhunger]

I very very sorry guys 😔

It was all due to Gradle using cached older bootstrap zip (which didn't have dash and coreutils) rather than ones with latest packages. When I first got the error from make, about not being able to find utilities like ls, rm and sh, I knew that I needed dash and coreutils.

I rebuilt the bootstrap, tried different things but still got the same errors. But then I tried building bootstrap for just coreutils and after exploring device file manager I still round ruby, and older binaries which made me skeptical of the caches.

I force deleted app from the emulator, deleted all the caches, .gradle, .idea folder. Re-added bootstraps to cpp folder. Now things worked!

Again, I am very sorry for wasting your time. I was stuck in this for 2+ days so thought of getting some expert help 😓

Thank you for your precious time, everyone.