Error in the native build of the armhf(32) version on aarch64 devices.

[150balbes]

When I try to run the build of the 32-bit version, directly on aarch64 devices, I get an error. Perhaps these are the consequences of switching to the "advanced" mode, but perhaps the reason is not related to this. I have previously performed armhf test builds on aarch64 devices without problems. For example, such an assembly worked on the ARM server, which we used for tests of the native assembly. Has anyone encountered a similar problem?

[ o.k. ] Compiling current kernel [ 5.10.88 ]
[ error ] ERROR in function compile_kernel [ main.sh:574 -> main.sh:479 -> compilation.sh:423 -> general.sh:0 ]
[ error ] Architecture [armhf] is not supported
[ o.k. ] Process terminated

[rpardini]

Hmm. This check was introduced by @The-going during that huge extensions+x86 PR, after the 'reverse cross compile' debate. It seems armhf is a victim too, in the sense that "the kernel-headers package produced when compiling armhf on arm64/amd64 would be invalid" so it has been blocked out. It's a very complex situation with the "headers" package, since it not only contain headers, but also binary packaging tools. We need to revisit this very soon, since "compile a working kernel" is a different objective than "compile a working kernel AND working headers package containing binary tools".

[The-going]

Gentlemen! We don't have a native cross-compilation mechanism yet, i.e. aarch64 -> arm, x86_64 -> i386.
Building a header package is a special case that stands at the very beginning of the build path. Next, we will want to build libraries that depend on the core ABI. For example, the choice of Mali or Lima , gpio will be in the dependencies of the corresponding libraries.
Jumping over this check, the image will most likely be assembled and will work, but not all subsequent packages that you want to assemble.

You can bypass this check, but export the correct build environment variables.

diff --git a/lib/compilation.sh b/lib/compilation.sh
index a1942db38..9f9e0204e 100644
--- a/lib/compilation.sh
+++ b/lib/compilation.sh
@@ -415,6 +415,11 @@ compile_kernel()
 	# if it matches we use the system compiler
 	if $(dpkg-architecture -e "${ARCH}"); then
 		display_alert "Native compilation"
+	elif [ "$(dpkg-architecture -qDEB_BUILD_ARCH)" == "arm64" ] && [ "$ARCH" == "armhf" ]; then
+		# DEB_BUILD_ARCH - The architecture of the machine on which the compilation process takes place.
+		# DEB_HOST_ARCH - Architecture of the host for which the target package is being assembled.
+		toolchain=$SYSTEM_CROSS_COMPILER
+		export DEB_HOST_ARCH=armhf
 	elif [[ $(dpkg --print-architecture) == amd64 ]]; then
 		local toolchain
 		toolchain=$(find_toolchain "$KERNEL_COMPILER" "$KERNEL_USE_GCC")

[150balbes]

Building armhf on aarch64 is not a cross-compilation, it is a native build. She doesn't need any checks. This is confirmed by the practice of assembling 32-bit (armhf) images on aarch64. These images and the kernel have been working fine for many months. The assembly of header packages that are actually used (in the Armbian network repositories, an in x86 assembly is used to receive users) has nothing to do with the native assembly of images that users perform on their devices

[The-going]

What are your suggestions?

[150balbes]

IMHO
At this stage, I propose to limit the assembly (support) of the header only in the mode of the central Armbian server (x86). These headers will be presented (posted) on the official servers (repo) and available to users. All other variants of the header assembly should be considered as a special case without a guarantee of support (correct operation). If there are those who want to integrate the header assembly for different build architectures, it's good, but it's not worth wasting resources and time right now, breaking the native image\kernel assembly work (the number of those who will specifically assemble native kernel headers is scanty, everyone uses either a full kernel assembly or an image assembly).

[The-going]

Перейдём на Русский.
Я никогда не занимался родной, но ве же кросс компиляцией т.е. на 64 битной машине я не собирал 32 битные пакеты. Соответственно мне неведомы подводные камни этого процесса. В принципе можно изменить эту проверку например так:

elif [ "$(dpkg-architecture -qDEB_BUILD_ARCH)" == "arm64" ] && [ "$ARCH" == "armhf" ]; then
      display_alert "Native compilation" "on arm64 for armhf"

Но я против того, чтобы в каком либо варианте выбора собирались ломаные пакеты.
И пользователю сообщалось об этом.

Лучше вообще не собирать ломаный пакет.
Но мы можем подумать или подсмотреть как это делают взрослые дистрибутивы.

Здесь хорошее место чтобы сказать свои идеи и мысли.

[150balbes]

Но я против того, чтобы в каком либо варианте выбора собирались ломаные пакеты.

Лучше вообще не собирать ломаный пакет.

можно отключить их сборку вообще, для всех хост, отличных от x86
не настолько это востребовано сейчас

[150balbes]

Еще нюанс, для 64 битных моделей устройств, может использоваться не только aarch64 версия образа\ядра, но и образы\ядро 32 бита, такие образы и ядро часто собирают пользователи сами на своих устройствах (хост aarch64). Поэтому наличие возможности сборки 32 бита на 64 битных важнее сборки пакета заголовков. Я знаю достаточное кол. пользователей, которые используют систему сборки Armbian только для сборки своей версии ядра (изменяют конфиг, добавляют драйвера). Тем и хороша система сборки Armbian, что позволет любому пользователю легко собрать своё ядро и легко его установить (именно пакет ядра, не пакет заголовков), автоматически выполняя всю подготовительную работу (зависимости, выбор правильных компиляторов и т.д.),

[The-going]

Еще нюанс, для 64 битных моделей устройств, может использоваться не только aarch64 версия образа\ядра, но и образы\ядро 32 бита,

Сразу вопрос. Где, в каком месте система сборки узнаёт о том, что мы собираем 32 битный пакет, который предназначен для работы на 64 битной архитектуре? Может я чего то не вижу?

Наверно гдето перед началом сборки пакета нам необходимо проверить эти три переменные:

@orangepipc2:~$ dpkg-architecture | grep _ARCH=
DEB_BUILD_ARCH=arm64
DEB_HOST_ARCH=arm64
DEB_TARGET_ARCH=arm64

и изменить DEB_HOST_ARCH и DEB_TARGET_ARCH на требуемые значения.

[rpardini]

Но мы можем подумать или подсмотреть как это делают взрослые дистрибутивы.

I've been studying a lot. Most distributions simply don't do any cross compile. See for example Launchpad -- each arch's kernels (and headers package) is built on its own arch CI runner. At .deb source level, it's an arch-specific pkg, so LP never even tries to cross compile. (Please someone correct me if I'm wrong)

Further: most embedded-oriented build systems simply don't produce kernel-headers packages at all - it simply does not make sense for them: target devices are tiny, kernel-headers would not even fit anywhere much less compile. The build system is the only place where kernel sources/headers makes sense for them, and most are indeed cross compiling.

Unrelated to distros, I've found this article, about the in-kernel headers that once were available (the feature, _KHEADERS=y, was reverted later, but that is not the important part). The paragraph "Building Kernel Modules" describe our situation perfectly: https://www.linuxjournal.com/content/extending-kernel-built-kernel-headers

I sincerely think that in cross compilation scenarios we can just simply disable the kernel-headers packaging (only!), and simply don't output it, but indeed build everything else normally. (Also, an option for disabling headers completely would accelerate kernel 5.15.y+ rebuilds by a LOT due to avoiding the make clean)

[150balbes]

I sincerely think that in cross compilation scenarios we can just simply disable the kernel-headers packaging (only!), and simply don't output it, but indeed build everything else normally. (Also, an option for disabling headers completely would accelerate kernel 5.15.y+ rebuilds by a LOT due to avoiding the make clean)

I agree with @rpardini , it is possible to disable header assembly for all architectures except x86. Ideally, you can generally move the header assembly point to a separate process, as is done when building only the kernel. For example, add a submenu to the kernel-only assembly selection "assemble headers" and build headers separately (as part of) the kernel-only assembly.

[The-going]

(Please someone correct me if I'm wrong)

Ricardo, you're right. The key fact here is the production of a package with source codes, which includes everything necessary for assembly. Everything else happens after this step.

avoiding the make clean

I will not agree here. The source codes should always be clean before starting the compilation process. You need to properly configure the compiler cache for speed.

For the rest, I agree. And we need to further develop these two scripts (builddeb, mkdebuian) in order to provide a choice or redo existing Ubuntu kernel scripts.

[rpardini]

The source codes should always be clean before starting the compilation process

We could discuss this, but, in fact, the make ... clean for packaging headers is currently being done after the main compile, and only if cross-compiling... so you get my point. Cleaning should not be needed given the way the kernel Makefiles are written. Any changes in .config or source code will trigger a partial rebuild. (of course there are some exceptions, like when changing arch)

Related, even without any cleaning: Currently the way we do it, every build is at least a partial rebuild/relink, since we re-patch on every build (and thus touch the source files, making them "more recent" than the binary counterparts already built from the last run).

You need to properly configure the compiler cache for speed

Yes, it's properly configured. But ccache can only cache compiles, not the linking, which would be "cached" (considered recent enough) by make if you did the same without using the Armbian build system. Try direct git clone, make menuconfig, then make many times. After the second make nothing changed so nothing is really done, it's instant.

I am trying to wrap my head around all these concepts/phases (git checkout, patch, hashing, then build, then package (which currently does clean if cross-compiling)) and will propose some changes when I understand enough.

The refactoring (moving around, renaming, usage of utility functions) of code I've been doing in #3362 has been helpful to understand more of it, but I still lack any real kernel 4.x makefile knowledge. I know some of 5.x if that much, and thus will need help in the endeavour to keep all families working.

Thanks for the great discussion!

[150balbes]

By the way, I noticed another oddity, the difference between the resulting image on x86 and aarch64. When assembling on x86, in the resulting image, the bottsplash works correctly (there is a splash screen display with the inscription Armbian). When building exactly the same image (from exactly the same source code) on aarch64, bootsplash does not work in the image. Perhaps this is only on my aarch64 system such a problem. Has anyone else encountered this behavior ?

[The-going]

the make ... clean for packaging headers

I apologize. I misunderstood. Yes, clearing the code to create a kernel header package and then compiling it on the board is not a good crutch. Alternatively, we can try to build a header package in the native environment.

In the New Year I'll come up with something. Congratulations on the New Year holiday!

[rpardini]

By the way, I noticed another oddity, the difference between the resulting image on x86 and aarch64. When assembling on x86, in the resulting image, the bottsplash works correctly (there is a splash screen display with the inscription Armbian). When building exactly the same image (from exactly the same source code) on aarch64, bootsplash does not work in the image. Perhaps this is only on my aarch64 system such a problem. Has anyone else encountered this behavior ?

I encountered similar problem, but I had not made the connection you did with the build host, thanks. Will investigate.

[150balbes]

I encountered similar problem, but I had not made the connection you did with the build host, thanks. Will investigate.

@rpardini I think I figured out why the bootsplash doesn't work when building an image on ARM. The function boot_logo not called from the file distributions.sh (which should assemble and copy in the /lib/firmware file (bootsplash.armbian) . The function itself is described in the file general.sh . If you manually insert this file from another image and execute the initrd update command, then the botsplash starts working. I haven't figured out yet where the error (change) that caused this behavior when running the build on ARM is. Perhaps you need to check the commits that changed the file distributions.sh , :)

[150balbes]

I found the reason, the binary utility is being called incorrectly. It is designed for x86. :)

https://github.com/armbian/build/blob/master/lib/general.sh#L701

[igorpecovnik]

We also have to prepare to move to plymouth once this year. This kernel splash is getting troublesome to maintain.

[rpardini]

Indeed. Thanks @150balbes for the investigative work. I've added SKIP_BOOTSPLASH during that big extensions/UEFI PR for that reason, I actually never saw the bootsplash work (ever) and the UEFI framebuffer is a different beast. I'm about to refactor qemu invocations in armbian-next branch so this is a great fit for a fix in the meanwhile. I will post here when I do it there...

[rpardini]

Both bootplash, and general unblocking of cross-compile, should have been resolved in armbian-next branch.
When cross-compiling in any scenario different than arm64-on-x86, a warning is logged, regarding the headers package.