CI: packages: Add github CI job to build all packages #10407

hauke · 2022-08-07T16:50:24Z

This will build OpenWrt for MIPS malta BE with all core packages
activated. It is triggered when something changes in the build system or
when a package definition is changed. This task probably needs multiple
hours to execute, but I hope that it will find build problems in pull
requests early.

This is based on the .github/workflows/packages.yml workflow.

Signed-off-by: Hauke Mehrtens hauke@hauke-m.de

aparcar · 2022-08-07T17:55:05Z

Why that arch specifically?

hauke · 2022-08-07T18:22:09Z

I did this only for mips 32 be because this will take some time to build. I think we do not have enough resources to build it for all targets. I am thinking about to add an other with experimental toolchain like gcc 12 and so on.

hauke · 2022-08-07T22:23:29Z

The kernel build takes 22m 50s here, for the all targets build it is about 10 minutes.This workflow will build all core packages and kernel modules and it also found a problem.

It failed because of this problem, that is getting fixed in #10396, I do not know why our build bots do not fail on this.

Packaged contents of /__w/openwrt/openwrt/openwrt/build_dir/target-mips-openwrt-linux-musl_musl/linux-malta_be/packages/ipkg-mips_24kc/kmod-phylink into /__w/openwrt/openwrt/openwrt/bin/targets/malta/be/packages/kmod-phylink_5.10.135-1_mips_24kc.ipk
Package kmod-mdio-devres is missing dependencies for the following libraries:
of_mdio.ko
make[3]: *** [modules/netdevices.mk:155: /__w/openwrt/openwrt/openwrt/bin/targets/malta/be/packages/kmod-mdio-devres_5.10.135-1_mips_24kc.ipk] Error 1
time: package/kernel/linux/compile#25.39#6.55#35.41

ynezz · 2022-08-08T09:35:18Z

.github/workflows/packages.yml

+        shell: su buildbot -c "sh -e {0}"
+        working-directory: openwrt
+        run: |
+          echo CONFIG_ALL=y >> .config


BTW this is how the config is seeded on our buildbots:

seedconfig: |- CONFIG_BUILDBOT=y CONFIG_DEVEL=y # CONFIG_CCACHE is not set CONFIG_KERNEL_KALLSYMS=y

so if the idea is to prevent build breakage later on our buildbots, then I would probably try to do it in similar way.

I would like to build all core kernel and user space packages here to detect if some pull request breaks building them. The build bots use a 2 stage setup, with the SDK, but that is more complicated.

The build bots use a 2 stage setup, with the SDK, but that is more complicated.

Indeed, but changes introduced in this repository might actually break the SDKs (we notice that much later either in Docker container build steps or on buildbots), or produce different (or unusable) results with the SDKs.

ynezz · 2022-08-08T09:40:49Z

I am thinking about to add an other with experimental toolchain like gcc 12 and so on.

Having one BE and one LE target is certainly a good idea. I would vote for x86/64 as that 2nd one due to KVM (much faster in QEMU), familiarity etc.

aparcar · 2022-08-11T10:59:44Z

.github/workflows/packages.yml

+      - name: Build toolchain
+        shell: su buildbot -c "sh -e {0}"
+        working-directory: openwrt
+        run: make toolchain/install -j$(nproc) BUILD_LOG=1


Is this step required when using an external toolchain?

Seems so as it does create some wrapper script

aparcar · 2022-08-11T11:00:43Z

.github/workflows/packages.yml

+      - name: Build Kernel
+        shell: su buildbot -c "sh -e {0}"
+        working-directory: openwrt
+        run: make target/compile -j$(nproc) BUILD_LOG=1


We have our own CI step to test Kernel changes, do you want to rebuild them here to? I'd see if we can disable as much Kernel as possible and only check core packages without kmods.

Agree that we can skip kernel build as we already have jobs that does the same.

The kernel build does not test if we can package the kernel modules. For example if kmod misses a dependency or the .ko file does not exists the kernel test will not fail, but this will fail.

imho we should find a way to compile kmods in the kernel build test. Currently we use make package/linux/compile that afaik package the .ko. Problem is that package like mac80211 require manual compilation as they are not handled by package/linux/compile

hauke · 2022-08-11T21:40:40Z

I am getting the following compile errors:
trace-cmd-v3.1.2 on malta be:

  COMPILE FPIC           trace-timesync-ptp.o
  COMPILE FPIC           trace-timesync-kvm.o
trace-timesync-kvm.c: In function 'kvm_open_vcpu_dir':
trace-timesync-kvm.c:221:19: error: 'PATH_MAX' undeclared (first use in this function)
  221 |         char path[PATH_MAX];
      |                   ^~~~~~~~
trace-timesync-kvm.c:221:19: note: each undeclared identifier is reported only once for each function it appears in
trace-timesync-kvm.c: In function 'kvm_open_debug_files':
trace-timesync-kvm.c:278:19: error: 'PATH_MAX' undeclared (first use in this function)
  278 |         char path[PATH_MAX];
      |                   ^~~~~~~~
make[5]: *** [Makefile:72: /__w/openwrt/openwrt/openwrt/build_dir/target-mips-openwrt-linux-musl_musl/trace-cmd-v3.1.2/lib/trace-cmd/trace-timesync-kvm.o] Error 1
make[4]: *** [Makefile:405: /__w/openwrt/openwrt/openwrt/build_dir/target-mips-openwrt-linux-musl_musl/trace-cmd-v3.1.2/lib/trace-cmd/libtracecmd.a] Error 2
make[4]: Leaving directory '/__w/openwrt/openwrt/openwrt/build_dir/target-mips-openwrt-linux-musl_musl/trace-cmd-v3.1.2'
make[3]: *** [Makefile:50: /__w/openwrt/openwrt/openwrt/build_dir/target-mips-openwrt-linux-musl_musl/trace-cmd-v3.1.2/.built] Error 2
time: package/devel/trace-cmd/compile#3.41#0.35#4.12

iucode-tool-2.3.1 on x86_64:

x86_64-openwrt-linux-musl-gcc -DHAVE_CONFIG_H -I.   -I/__w/openwrt/openwrt/openwrt/openwrt-toolchain-x86-64_gcc-11.3.0_musl.Linux-x86_64/toolchain-x86_64_gcc-11.3.0_musl/usr/include -I/__w/openwrt/openwrt/openwrt/openwrt-toolchain-x86-64_gcc-11.3.0_musl.Linux-x86_64/toolchain-x86_64_gcc-11.3.0_musl/include   -Os -pipe -fno-caller-saves -fno-plt  -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro  -MT iucode_tool.o -MD -MP -MF .deps/iucode_tool.Tpo -c -o iucode_tool.o iucode_tool.c
iucode_tool.c: In function 'load_intel_microcode_bin':
iucode_tool.c:1088:45: error: 'SSIZE_MAX' undeclared (first use in this function); did you mean 'SIZE_MAX'?
 1088 |                           (mcb_space_left < SSIZE_MAX) ? mcb_space_left : SSIZE_MAX);
      |                                             ^~~~~~~~~
      |                                             SIZE_MAX
iucode_tool.c:1088:45: note: each undeclared identifier is reported only once for each function it appears in
iucode_tool.c: In function 'load_intel_microcode':
iucode_tool.c:1268:20: error: 'PATH_MAX' undeclared (first use in this function)
 1268 |         char fnbuf[PATH_MAX];
      |                    ^~~~~~~~
iucode_tool.c: In function 'ssizemax_clamp':
iucode_tool.c:1429:21: error: 'SSIZE_MAX' undeclared (first use in this function); did you mean 'SIZE_MAX'?
 1429 |         return (s < SSIZE_MAX) ? (size_t)s : SSIZE_MAX;
      |                     ^~~~~~~~~
      |                     SIZE_MAX
In file included from iucode_tool.c:29:
iucode_tool.c: In function 'write_cpio_header':
iucode_tool.c:1731:22: error: 'SSIZE_MAX' undeclared (first use in this function); did you mean 'SIZE_MAX'?
 1731 |         assert(pos < SSIZE_MAX);
      |                      ^~~~~~~~~
iucode_tool.c: In function 'xx_check_cpuid_devs':
iucode_tool.c:2767:27: error: 'PATH_MAX' undeclared (first use in this function)
 2767 |         char cpuid_device[PATH_MAX];
      |                           ^~~~~~~~
make[5]: *** [Makefile:440: iucode_tool.o] Error 1
make[5]: Leaving directory '/__w/openwrt/openwrt/openwrt/build_dir/target-x86_64-openwrt-linux-musl_musl/iucode-tool-2.3.1'
make[4]: *** [Makefile:329: all] Error 2
make[4]: Leaving directory '/__w/openwrt/openwrt/openwrt/build_dir/target-x86_64-openwrt-linux-musl_musl/iucode-tool-2.3.1'
make[3]: *** [Makefile:65: /__w/openwrt/openwrt/openwrt/build_dir/target-x86_64-openwrt-linux-musl_musl/iucode-tool-2.3.1/.built] Error 2
time: package/system/iucode-tool/compile#1.88#0.52#2.48

I have to look more closely into these problems.

Ansuel · 2022-08-19T16:59:58Z

.github/workflows/packages.yml

+            --overwrite-config \
+            --config ${{ env.TARGET }}/${{ env.SUBTARGET }}
+
+          make defconfig


In theory the script already run defconfig if i'm not wrong

Ansuel · 2022-08-19T17:01:58Z

.github/workflows/packages.yml

+      - name: Build everything
+        shell: su buildbot -c "sh -e {0}"
+        working-directory: openwrt
+        run: make -j$(nproc) BUILD_LOG=1


I think we have to build everything separately. A simple make won't permit us to skip things like kernel build.

Also having split jobs permit us to better track what is actually failing. (and eventually build only package without creating an actual image)

In that case make package/compile should be used. Running each individually would take forever as it checks the dependencies every time

Yes I mean using all the subcommand, wasn't saying to call compile for each package.

I would like to compile all normal and kernel packages from the OpenWrt core here explicitly.
I restricted it to only two targets, but on these I would like to run a full build to hopefully spot build problems early.

@hauke to make debug quicker i notice a little trick on another workflow.

make -j$(nproc) BUILD_LOG=1 || make -j1 V=s in theory the affected error should be printed. What do you think?

make -j1 V=s

You should probably first backup the logs which would be likely containing the previous error(s), since it might be related to some concurrent build issue and thus single threaded build attempt might succeed and clear out that previous build error(s).

in theory the affected error should be printed

What about following ci: show build failures directly in job log output ?

I copied it form ynezz@2dc201b

hauke · 2022-09-11T22:44:23Z

trace-cmd-v3.1.2 and iucode-tool-2.3.1 are failing to compile because we do not add this: -I$(TOOLCHAIN_DIR)/include/fortify when we use an external musl toolchain.For internal toolchain it is done here:
https://github.com/openwrt/openwrt/blob/master/rules.mk#L190

I will check if this means that fortify source is not working when using external musl toolchains.

Ansuel · 2022-09-11T23:24:12Z

(btw a bit ot but we lack workflow to test toolchain changes)

stintel · 2022-09-23T15:04:47Z

Having one BE and one LE target is certainly a good idea. I would vote for x86/64 as that 2nd one due to KVM (much faster in QEMU), familiarity etc.

Also building for x86/64 on x86/64 is extra sensitive for host lib leakage iirc, so it's definitely a good candidate imo.

stintel · 2022-09-23T15:05:52Z

Is there an easy way to also enable all CONFIG_KERNEL_* symbols? There's a lot of breakage in some of them when we're adding a new kernel.

Ansuel · 2022-10-21T22:32:07Z

@hauke considering we already have test for tools why not use the tools container also for this?

hauke · 2022-10-22T14:37:49Z

Yes I will try to use the pre build tools. unetd is still not building without the LLVM toolchain.

 make[3] -C package/network/services/unetd compile
    ERROR: package/network/services/unetd failed to build.
make -r world: build failed. Please re-run make with -j1 V=s or V=sc for a higher verbosity level to see what's going on
make: *** [/__w/openwrt/openwrt/openwrt/include/toplevel.mk:231: world] Error 1

Ansuel · 2022-10-22T14:41:30Z

I'm probably blind but the tools is not getting compiled? Or if it should be part of the toolchain are we sure we are shipping external toolchain with them?

time: target/linux/prereq#0.09#0.04#0.12
 make[1] tools/install
 make[2] -C tools/flock compile
 make[2] -C tools/xz compile
 make[2] -C tools/sed compile
 make[2] -C tools/tar compile
 make[2] -C tools/patch compile
 make[2] -C tools/m4 compile
 make[2] -C tools/autoconf-archive compile
 make[2] -C tools/ninja compile
 make[2] -C tools/expat compile
 make[2] -C tools/zlib compile
 make[2] -C tools/cpio compile
 make[2] -C tools/lzma compile
 make[2] -C tools/make-ext4fs compile
 make[2] -C tools/mtools compile
 make[2] -C tools/patch-image compile
 make[2] -C tools/squashfskit4 compile
 make[2] -C tools/sstrip compile
 make[2] -C tools/zip compile
 make[2] -C tools/elfutils compile
 make[2] -C tools/autoconf compile
 make[2] -C tools/meson compile
 make[2] -C tools/missing-macros compile
 make[2] -C tools/zstd compile
 make[2] -C tools/pkgconf compile
 make[2] -C tools/automake compile
 make[2] -C tools/libressl compile
 make[2] -C tools/dosfstools compile
 make[2] -C tools/libtool compile
 make[2] -C tools/cmake compile
 make[2] -C tools/flex compile
 make[2] -C tools/e2fsprogs compile
 make[2] -C tools/fakeroot compile
 make[2] -C tools/gengetopt compile
 make[2] -C tools/mklibs compile
 make[2] -C tools/mtd-utils compile
 make[2] -C tools/patchelf compile
 make[2] -C tools/bison compile
 make[2] -C tools/bc compile
 make[2] -C tools/findutils compile
 make[2] -C tools/mkimage compile
 make[2] -C tools/padjffs2 compile
 make[2] -C tools/quilt compile
 make[2] -C tools/b43-tools compile
 make[2] -C tools/firmware-utils compile

hauke · 2022-10-26T22:07:02Z

I will try to use prebuild tools like done in 5d09118 and c27b439

Ansuel · 2022-10-26T22:11:26Z

@hauke remember to set the correct permission

hauke · 2022-10-30T16:12:07Z

@stintel We do not have a option to build all CONFIG_KERNEL_* options, you could add it.

I am using now the tools container like it is done in the kernel.yml file. I syned the code again with the kernel.yml, but did not add the ccache. I would like to detect problems like a gdb build failure after the readĺine upgrade, are they still detected with ccache?

I also updated the permissions to match kernel.yml.

unetd always includes $(INCLUDE_DIR)/bpf.mk. This file always checks if the LLVM version is supported in CLANG_VER_VALID. unetd only needs bpf when UNETD_VXLAN_SUPPORT is set. It fails when UNETD_VXLAN_SUPPORT is not set and llvm is not installed. Fix it by only checking the LLVM version when a LLVM toolchain is available. Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>

This will build OpenWrt for MIPS malta BE and x86 64 Bit with all packages and kernel modules activated. It is triggered when something changes in the build system or when a package definition is changed. This task probably needs 90 minutes to execute, but I hope that it will find build problems in pull requests early. This intentionally does not activate the feeds, because building them too would take too long. We only build x86/64 and malta/be to save resources. I would like to detect build problems when a package is changed. We often had build breaks when a package version was increased sometime even in other packages which used it as a dependency. This is based on the .github/workflows/packages.yml workflow. Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>

hauke added the GitHub/CI pull requests/issues for GitHub, CI and related stuff label Aug 7, 2022

hauke requested a review from aparcar August 7, 2022 16:50

hauke force-pushed the github-all-packages branch 4 times, most recently from e0bf205 to 0d305d5 Compare August 7, 2022 16:54

hauke force-pushed the github-all-packages branch from a9662d9 to f60e7ef Compare August 7, 2022 22:24

ynezz reviewed Aug 8, 2022

View reviewed changes

hauke force-pushed the github-all-packages branch 2 times, most recently from 1b505b7 to 4acd746 Compare August 10, 2022 22:54

aparcar reviewed Aug 11, 2022

View reviewed changes

Ansuel reviewed Aug 19, 2022

View reviewed changes

hauke force-pushed the github-all-packages branch 4 times, most recently from 9111715 to 232ae69 Compare September 11, 2022 22:41

hauke force-pushed the github-all-packages branch 2 times, most recently from fe3666d to 088fd45 Compare September 12, 2022 21:36

hauke force-pushed the github-all-packages branch from 088fd45 to f0165ca Compare October 3, 2022 18:15

hauke force-pushed the github-all-packages branch from f0165ca to 05155c8 Compare October 21, 2022 21:11

hauke force-pushed the github-all-packages branch from 05155c8 to bf73c04 Compare October 26, 2022 21:09

github-actions bot added the build/scripts/tools pull request/issues for build, scripts and tools related changes label Oct 26, 2022

hauke force-pushed the github-all-packages branch from bf73c04 to 0cc0e94 Compare October 26, 2022 21:35

hauke force-pushed the github-all-packages branch 2 times, most recently from 657191f to deaddb9 Compare October 30, 2022 16:09

hauke force-pushed the github-all-packages branch 2 times, most recently from 173e274 to f1abd78 Compare October 30, 2022 16:19

hauke added 2 commits October 31, 2022 00:46

hauke force-pushed the github-all-packages branch from f1abd78 to b99d377 Compare October 30, 2022 23:47

openwrt-bot merged commit b99d377 into openwrt:master Oct 30, 2022

hauke deleted the github-all-packages branch October 30, 2022 23:58

ynezz mentioned this pull request May 24, 2023

kernel: introduce KERNEL_WERROR config option #12713

Merged

ynezz mentioned this pull request Jan 7, 2024

RFC: Try to prevent harder buildbot failures openwrt/actions-shared-workflows#5

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: packages: Add github CI job to build all packages #10407

CI: packages: Add github CI job to build all packages #10407

hauke commented Aug 7, 2022 •

edited

Loading

aparcar commented Aug 7, 2022

hauke commented Aug 7, 2022

hauke commented Aug 7, 2022

ynezz Aug 8, 2022 •

edited

Loading

hauke Aug 8, 2022

ynezz Oct 10, 2022 •

edited

Loading

ynezz commented Aug 8, 2022

aparcar Aug 11, 2022

Ansuel Aug 19, 2022

aparcar Aug 11, 2022

Ansuel Aug 19, 2022

hauke Oct 26, 2022

Ansuel Oct 26, 2022

hauke commented Aug 11, 2022

Ansuel Aug 19, 2022

hauke Oct 26, 2022

Ansuel Aug 19, 2022 •

edited

Loading

aparcar Aug 19, 2022

Ansuel Aug 19, 2022

hauke Sep 11, 2022

Ansuel Sep 13, 2022

ynezz Sep 13, 2022

hauke Oct 26, 2022 •

edited

Loading

hauke commented Sep 11, 2022

Ansuel commented Sep 11, 2022 •

edited

Loading

stintel commented Sep 23, 2022

stintel commented Sep 23, 2022

Ansuel commented Oct 21, 2022

hauke commented Oct 22, 2022

Ansuel commented Oct 22, 2022

hauke commented Oct 26, 2022

Ansuel commented Oct 26, 2022

hauke commented Oct 30, 2022 •

edited

Loading

CI: packages: Add github CI job to build all packages #10407

CI: packages: Add github CI job to build all packages #10407

Conversation

hauke commented Aug 7, 2022 • edited Loading

aparcar commented Aug 7, 2022

hauke commented Aug 7, 2022

hauke commented Aug 7, 2022

ynezz Aug 8, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ynezz Oct 10, 2022 • edited Loading

Choose a reason for hiding this comment

ynezz commented Aug 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hauke commented Aug 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ansuel Aug 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hauke Oct 26, 2022 • edited Loading

Choose a reason for hiding this comment

hauke commented Sep 11, 2022

Ansuel commented Sep 11, 2022 • edited Loading

stintel commented Sep 23, 2022

stintel commented Sep 23, 2022

Ansuel commented Oct 21, 2022

hauke commented Oct 22, 2022

Ansuel commented Oct 22, 2022

hauke commented Oct 26, 2022

Ansuel commented Oct 26, 2022

hauke commented Oct 30, 2022 • edited Loading

hauke commented Aug 7, 2022 •

edited

Loading

ynezz Aug 8, 2022 •

edited

Loading

ynezz Oct 10, 2022 •

edited

Loading

Ansuel Aug 19, 2022 •

edited

Loading

hauke Oct 26, 2022 •

edited

Loading

Ansuel commented Sep 11, 2022 •

edited

Loading

hauke commented Oct 30, 2022 •

edited

Loading