build-bootstraps.sh: adding a way to generate bootstrap based on pacman #10540

Maxython · 2022-05-07T20:26:59Z

Note: all packages that have been compiled are archived (including those that are compilation dependencies). It is not right.

@agnostic-apollo

stale · 2022-06-23T11:38:03Z

This issue/PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale · 2022-08-07T13:42:50Z

This issue/PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

diyism · 2022-09-29T15:35:51Z

@Maxython , error occured when I build:

$ git clone -b bootstraps --depth 1 https://github.com/termux-pacman/termux-packages.git
$ cd termux-packages
$ ./scripts/run-docker.sh
$ export NDK="/home/builder/lib/android-ndk-r23b"
$ export TERMUX_NDK_VERSION="23b"
$ ./scripts/setup-android-sdk.sh
$ ./scripts/build-bootstraps.sh --architectures aarch64
[*] Building 'apt'...
termux - building apt for arch aarch64...
Building dependency libandroid-support if necessary...
libandroid-support@28-2 built - skipping (rm /data/data/.built-packages/libandroid-support to force rebuild)
...
patching file cmath
patching file math.h
cp: cannot stat '/home/builder/.termux-build/_cache/android-r23b-api-24-v9/lib64/clang/12.0.8/lib/linux/aarch64/libunwind.a': No such file or directory
[*] Building 'apt' exited with exit code 1
Failed to build package 'apt' for arch 'aarch64' in format 'debian'

I can see "/home/builder/.termux-build/_cache/android-r23b-api-24-v9/lib64/clang/12.0.9" there,
I can't figure out how to fix it, any hint?

Maxython · 2022-09-30T11:40:11Z

@Maxython , error occured when I build:

$ git clone -b bootstraps --depth 1 https://github.com/termux-pacman/termux-packages.git
$ cd termux-packages
$ ./scripts/run-docker.sh
$ export NDK="/home/builder/lib/android-ndk-r23b"
$ export TERMUX_NDK_VERSION="23b"
$ ./scripts/setup-android-sdk.sh
$ ./scripts/build-bootstraps.sh --architectures aarch64
[*] Building 'apt'...
termux - building apt for arch aarch64...
Building dependency libandroid-support if necessary...
libandroid-support@28-2 built - skipping (rm /data/data/.built-packages/libandroid-support to force rebuild)
...
patching file cmath
patching file math.h
cp: cannot stat '/home/builder/.termux-build/_cache/android-r23b-api-24-v9/lib64/clang/12.0.8/lib/linux/aarch64/libunwind.a': No such file or directory
[*] Building 'apt' exited with exit code 1
Failed to build package 'apt' for arch 'aarch64' in format 'debian'

I can see "/home/builder/.termux-build/_cache/android-r23b-api-24-v9/lib64/clang/12.0.9" there, I can't figure out how to fix it, any hint?

Apparently the environment in this thread is already old. Try running it in the original repository.

diyism · 2022-10-01T06:49:42Z

Apparently the environment in this thread is already old. Try running it in the original repository.

I've tried your build-bootstraps.sh in the original repo, it can succefully build the "./termux-packages/bootstrap-aarch64.zip",
but its size is about 242MB, while the bootstrap-aarch64.zip in releases page(https://github.com/termux/termux-packages/releases/tag/bootstrap-2022.09.18-r1%2Bapt-android-7) is only 25MB,

Is there a special building parameter for a slim recipe like the official release?

Maxython · 2022-10-01T07:09:08Z

Is there a special building parameter for a slim recipe like the official release?

No. Here the correct configured logic is already needed. Thank you for report.

agnostic-apollo · 2022-10-01T07:43:29Z

but its size is about 242MB,

Did you modify build-bootstrap.sh or pass parameters to add more packages. If not, then likely issue is that unneeded dependencies are getting included, or debs of other archs. Use -f flag to fix later.

A build log would be needed for unneeded dependencies or if something else is going on.

#10296 (comment)

I guess I should merge the temp patches for now before the rewrite.

diyism · 2022-10-01T11:04:42Z

I tried again:

$ docker kill $(docker ps -q)
$ git clone --depth https://github.com/termux/termux-packages.git
$ cd termux-packages/scripts
$ rm build-bootstraps.sh
$ wget https://raw.githubusercontent.com/termux-pacman/termux-packages/bootstraps/scripts/build-bootstraps.sh
$ chmod 755 build-bootstraps.sh
$ ./run-docker.sh
$ sudo bash -c "echo 'nameserver 172.17.0.1' >/etc/resolv.conf"
$ ./scripts/setup-android-sdk.sh
$ export NDK="/home/builder/lib/android-ndk-r25b"
$ export TERMUX_NDK_VERSION="25b"
$ ./scripts/build-bootstraps.sh --architectures aarch64
...
[*] Finished successfully (aarch64).

$ ls -al
total 3164
drwxrwxrwx   12 root    root       4096 Oct  1 18:53 .
drwxr-x---    1 builder builder    4096 Oct  1 17:05 ..
-rw-r--r--    1 builder builder 3008794 Oct  1 18:53 bootstrap-aarch64.zip

$ unzip -l bootstrap-aarch64.zip 
Archive:  bootstrap-aarch64.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2022-10-01 18:53   bin/
  1130424  2022-10-01 17:18   bin/coreutils
    22304  2022-10-01 17:18   bin/iconv
        0  2022-10-01 18:53   etc/
...
        0  2022-10-01 17:18   lib/
    47384  2022-10-01 17:17   lib/libgmpxx.so
    20616  2022-10-01 17:17   lib/libandroid-support.so
  1007776  2022-10-01 17:18   lib/libiconv.so
        0  2022-10-01 17:17   lib/pkgconfig/
      326  2022-10-01 17:17   lib/pkgconfig/gmpxx.pc
      291  2022-10-01 17:17   lib/pkgconfig/gmp.pc
     3872  2022-10-01 17:18   lib/libcharset.so
   385632  2022-10-01 17:17   lib/libgmp.so

It's weird, this time the bootstrap-aarch64.zip is only about 3MB,
it lacks many files in it, for example: bootstrap-aarch64.zip/lib/libtermux-exec.so

agnostic-apollo · 2022-10-01T22:30:23Z

Did you wipe the termux-packages/output directory containing the deb files? The build-package.sh does not actually check if deb file exists for the package and its subpackages before skipping build, it only checks a package name file to see if package was built. So if you deleted the debs, only changed packages would get built and only their debs would be added to bootstrap. You should use -f flag to force rebuild like I said earlier, should work around this issue too. I have already added validation for such cases locally, will push it when I push rewrite.

diyism · 2022-10-02T02:11:23Z

Sad, I tried the 3rd time and the 4th time, it failed with the same new error:

$ nano ./packages/liblzma/build.sh
TERMUX_PKG_VERSION=5.2.7
TERMUX_PKG_SHA256=8712e9acb0b6b49a97d443458a3067dc5c08a025e02dc5f773176c51dd7cfc69

$ ./scripts/build-bootstraps.sh --architectures aarch64 -f
...
[*] Building 'command-not-found'...
termux - building command-not-found for arch aarch64...
Downloading https://github.com/termux/command-not-found/archive/2.1.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100  8184    0  8184    0     0   3455      0 --:--:--  0:00:02 --:--:-- 26832
Downloading https://nodejs.org/dist/v16.14.0/node-v16.14.0-linux-x64.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20.9M  100 20.9M    0     0   705k      0  0:00:30  0:00:30 --:--:-- 1006k
-- Android: Targeting API '24' with architecture 'arm64', ABI 'arm64-v8a', and processor 'aarch64'
-- The C compiler identification is Clang 14.0.6
-- The CXX compiler identification is Clang 14.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/builder/.termux-build/_cache/android-r25-api-24-v1/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/builder/.termux-build/_cache/android-r25-api-24-v1/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    BUILD_TESTING
    CMAKE_FIND_ROOT_PATH
    CMAKE_FIND_ROOT_PATH_MODE_LIBRARY
    CMAKE_INSTALL_LIBDIR
    CMAKE_USE_SYSTEM_LIBRARIES
    DOXYGEN_EXECUTABLE


-- Build files have been written to: /home/builder/.termux-build/command-not-found/build
[1/3] cd /home/builder/.termux-build/command-not-found/src && ./generate-db.js
FAILED: CMakeFiles/generate-db /home/builder/.termux-build/command-not-found/build/CMakeFiles/generate-db 
cd /home/builder/.termux-build/command-not-found/src && ./generate-db.js
node:events:498
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20)
Emitted 'error' event on ClientRequest instance at:
    at TLSSocket.socketErrorListener (node:_http_client:442:9)
    at TLSSocket.emit (node:events:520:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}
ninja: build stopped: subcommand failed.
[*] Building 'command-not-found' exited with exit code 1
Failed to build package 'command-not-found' for arch 'aarch64' in format 'debian'

agnostic-apollo · 2022-10-02T04:04:34Z

Is still some network issue.

https://stackoverflow.com/questions/17245881/how-do-i-debug-error-econnreset-in-node-js

Can you connect to https://packages-cf.termux.dev/apt

diyism · 2022-10-02T13:06:58Z

It seems the curl is ok, maybe there's a problem in the nodejs code, I have tried many times, the same result:

$ ./scripts/build-bootstraps.sh --architectures aarch64
...
Error: read ECONNRESET
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20)
Emitted 'error' event on ClientRequest instance at:
    at TLSSocket.socketErrorListener (node:_http_client:442:9)
    at TLSSocket.emit (node:events:520:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}
ninja: build stopped: subcommand failed.
[*] Building 'command-not-found' exited with exit code 1
Failed to build package 'command-not-found' for arch 'aarch64' in format 'debian'


$ curl https://packages-cf.termux.dev/apt/
<html>
<head><title>Index of /apt/</title></head>
<body>
<h1>Index of /apt/</h1><hr><pre><a href="../">../</a>
<a href="dists/">dists/</a>                                             21-Jun-2022 11:19                   -
<a href="pool/">pool/</a>                                              21-Jun-2022 11:19                   -
<a href="termux-games/">termux-games/</a>                                      06-Jul-2021 18:03                   -
<a href="termux-main/">termux-main/</a>                                       07-Mar-2022 11:01

after I get rid of "command-not-found" package from ./scripts/build-bootstraps.sh, the building can continue.

nawbc · 2022-10-27T09:12:43Z

Did you wipe the termux-packages/output directory containing the deb files? The build-package.sh does not actually check if deb file exists for the package and its subpackages before skipping build, it only checks a package name file to see if package was built. So if you deleted the debs, only changed packages would get built and only their debs would be added to bootstrap. You should use -f flag to force rebuild like I said earlier, should work around this issue too. I have already added validation for such cases locally, will push it when I push rewrite.

I had run clean.sh and -f , but still got the huge size archive.

platform: windows11 , docker ghcr.io/termux/package-builder

total 251M
-rw-r--r--    1 builder builder 251M Oct 27 09:00 bootstrap-aarch64.zip
-rwxr-xr-x    1 builder builder 2.9K Oct 26 08:31 build-all.sh
-rwxr-xr-x    1 builder builder  24K Oct 26 09:57 build-package.sh
-rwxr-xr-x    1 builder builder 1.5K Oct 26 08:31 clean.sh
-rw-r--r--    1 builder builder 6.3K Oct 25 17:16 CODEOWNERS
-rw-r--r--    1 builder builder  25K Oct 25 17:16 CONTRIBUTING.md
drwxr-xr-x   47 builder builder 4.0K Oct 25 17:16 disabled-packages
-rw-r--r--    1 builder builder  503 Oct 25 17:16 LICENSE.md
drwxr-xr-x    3 builder builder 4.0K Oct 26 09:15 ndk-patches
drwxr-xr-x    2 builder builder  16K Oct 27 08:58 output
drwxr-xr-x 1733 builder builder  36K Oct 26 09:15 packages
-rw-r--r--    1 builder builder 1.8K Oct 25 17:16 README.md
-rw-r--r--    1 builder builder  220 Oct 25 17:16 repo.json
drwxr-xr-x   58 builder builder 4.0K Oct 25 17:16 root-packages
drwxr-xr-x    2 builder builder 4.0K Oct 25 17:16 sample
drwxr-xr-x    6 builder builder 4.0K Oct 26 11:20 scripts
drwxr-xr-x  394 builder builder  12K Oct 26 09:15 x11-packages

nawbc · 2022-10-27T10:10:38Z

@Maxython
The clang llvm will be packed into the result.

-rw-r--r--  1 builder builder  57M Oct 27 08:07 libllvm-static_15.0.3_aarch64.deb
-rw-r--r--  1 builder builder  27M Oct 27 08:03 clang_15.0.3_aarch64.deb
-rw-r--r--  1 builder builder  22M Oct 27 08:08 libllvm_15.0.3_aarch64.deb
-rw-r--r--  1 builder builder  18M Oct 27 08:05 mlir_15.0.3_aarch64.deb
-rw-r--r--  1 builder builder  14M Oct 27 08:13 perl_5.36.0-1_aarch64.deb
-rw-r--r--  1 builder builder 8.8M Oct 27 07:34 python_3.10.8_aarch64.deb
-rw-r--r--  1 builder builder 8.0M Oct 27 08:04 llvm_15.0.3_aarch64.deb
-rw-r--r--  1 builder builder 4.3M Oct 27 08:04 lldb_15.0.3_aarch64.deb
-rw-r--r--  1 builder builder 3.8M Oct 27 07:33 python-static_3.10.8_aarch64.deb
-rw-r--r--  1 builder builder 2.7M Oct 27 08:04 libcompiler-rt_15.0.3_aarch64.deb
-rw-r--r--  1 builder builder 2.4M Oct 27 07:29 ttf-dejavu_2.37-8_all.deb
-rw-r--r--  1 builder builder 2.2M Oct 27 07:32 tcl_8.6.11-1_aarch64.deb
-rw-r--r--  1 builder builder 2.1M Oct 27 08:24 tor_0.4.7.10_aarch64.deb
-rw-r--r--  1 builder builder 2.0M Oct 27 08:04 lld_15.0.3_aarch64.deb
-rw-r--r--  1 builder builder 1.9M Oct 27 08:04 llvm-tools_15.0.3_aarch64.deb
-rw-r--r--  1 builder builder 1.8M Oct 27 07:30 libx11_1.7.5_aarch64.deb
-rw-r--r--  1 builder builder 1.6M Oct 27 08:48 openssl-static_1:3.0.5_aarch64.deb
-rw-r--r--  1 builder builder 1.6M Oct 27 08:48 openssl_1:3.0.5_aarch64.deb
-rw-r--r--  1 builder builder 1.4M Oct 27 07:26 ndk-sysroot_25b_aarch64.deb
-rw-r--r--  1 builder builder 1.3M Oct 27 07:24 docbook-xsl_1.79.2-1_all.deb
-rw-r--r--  1 builder builder 1.3M Oct 27 08:21 gnupg_2.3.8_aarch64.deb
....

Maxython · 2022-10-27T10:30:06Z

Perhaps I will close this pr as the environment is too old and needs to be improved with the build-bootstraps.sh script. And issues #10462, I will solve in a different way.

agnostic-apollo · 2022-10-27T10:40:01Z

The clang llvm will be packed into the result.

termux/termux-app#2892 (comment)

Perhaps I will close this pr as the environment is too old and needs to be improved with the build-bootstraps.sh script. And issues #10462, I will solve in a different way.

Keep it open, will look into it. Not sure what you mean by old environment. And sub package fix should be fine too, will need to look into how I solved it in my local one.

Maxython · 2022-10-27T10:45:52Z

Keep it open, will look into it. Not sure what you mean by old environment. And sub package fix should be fine too, will need to look into how I solved it in my local one.

Ok, i see

Maxython · 2023-01-21T10:51:46Z

This PR is being reopened (this one will close and a new one will open). I did this to change the "from" branch. This branch is now in this repository.

#14692

Maxython force-pushed the bootstraps branch from e0ce02e to f6fa7e9 Compare May 9, 2022 11:12

stale bot added the wontfix Issue won't be fixed label Jun 23, 2022

agnostic-apollo removed the wontfix Issue won't be fixed label Jun 23, 2022

agnostic-apollo mentioned this pull request Aug 5, 2022

[Bug]: build-package.sh: cannot compile subpackages #10462

Open

stale bot added the wontfix Issue won't be fixed label Aug 7, 2022

2096779623 added not stale and removed wontfix Issue won't be fixed labels Aug 7, 2022

truboxl mentioned this pull request Aug 27, 2022

[Bug]: Building apt fails with configure: exit code 77 #11713

Closed

agnostic-apollo mentioned this pull request Oct 24, 2022

If building for arm fails, will build-bootstrap.sh rebuild the already-built aarch64 pkgs from scratch? #12529

Closed

xtkoba removed the not stale label Dec 24, 2022

build-bootstraps.sh: adding a way to generate bootstrap based on pacman

e117fd3

Maxython force-pushed the bootstraps branch from f6fa7e9 to e117fd3 Compare January 21, 2023 10:04

Maxython mentioned this pull request Jan 21, 2023

build-bootstraps.sh: adding a way to generate bootstrap based on pacman #14692

Draft

Maxython closed this Jan 21, 2023

Maxython deleted the bootstraps branch January 21, 2023 10:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build-bootstraps.sh: adding a way to generate bootstrap based on pacman #10540

build-bootstraps.sh: adding a way to generate bootstrap based on pacman #10540

Maxython commented May 7, 2022

stale bot commented Jun 23, 2022

stale bot commented Aug 7, 2022

diyism commented Sep 29, 2022 •

edited

Loading

Maxython commented Sep 30, 2022

diyism commented Oct 1, 2022

Maxython commented Oct 1, 2022

agnostic-apollo commented Oct 1, 2022

diyism commented Oct 1, 2022 •

edited

Loading

agnostic-apollo commented Oct 1, 2022 •

edited

Loading

diyism commented Oct 2, 2022 •

edited

Loading

agnostic-apollo commented Oct 2, 2022

diyism commented Oct 2, 2022 •

edited

Loading

nawbc commented Oct 27, 2022

nawbc commented Oct 27, 2022 •

edited

Loading

Maxython commented Oct 27, 2022

agnostic-apollo commented Oct 27, 2022

Maxython commented Oct 27, 2022

Maxython commented Jan 21, 2023 •

edited

Loading

build-bootstraps.sh: adding a way to generate bootstrap based on pacman #10540

build-bootstraps.sh: adding a way to generate bootstrap based on pacman #10540

Conversation

Maxython commented May 7, 2022

stale bot commented Jun 23, 2022

stale bot commented Aug 7, 2022

diyism commented Sep 29, 2022 • edited Loading

Maxython commented Sep 30, 2022

diyism commented Oct 1, 2022

Maxython commented Oct 1, 2022

agnostic-apollo commented Oct 1, 2022

diyism commented Oct 1, 2022 • edited Loading

agnostic-apollo commented Oct 1, 2022 • edited Loading

diyism commented Oct 2, 2022 • edited Loading

agnostic-apollo commented Oct 2, 2022

diyism commented Oct 2, 2022 • edited Loading

nawbc commented Oct 27, 2022

nawbc commented Oct 27, 2022 • edited Loading

Maxython commented Oct 27, 2022

agnostic-apollo commented Oct 27, 2022

Maxython commented Oct 27, 2022

Maxython commented Jan 21, 2023 • edited Loading

diyism commented Sep 29, 2022 •

edited

Loading

diyism commented Oct 1, 2022 •

edited

Loading

agnostic-apollo commented Oct 1, 2022 •

edited

Loading

diyism commented Oct 2, 2022 •

edited

Loading

diyism commented Oct 2, 2022 •

edited

Loading

nawbc commented Oct 27, 2022 •

edited

Loading

Maxython commented Jan 21, 2023 •

edited

Loading