building with clang rather than gcc #124

ColinIanKing · 2020-05-01T13:54:42Z

I'm trying to build various dkms packages using clang rather than gcc and I don't know how to override the default gcc setting and get dkms to use clang instead.

The linux kernel now supports native clang builds, setting LLVM=1 enables this. Is there a way to force dkms to use this option too?

I've tried --directive LLVM=1 but that doesn't seem to work for me. Any ideas?

superm1 · 2020-05-15T20:43:17Z

@GoPerry any ideas here?

GoPerry · 2020-05-17T16:28:21Z

@superm1
I will take a look for this clang option.

GoPerry · 2020-05-18T11:55:26Z

I guess that the default cc should be changed to clang ,then dkms built will use the new default compiler .

@ColinIanKing Do you change your default compiler ? It not, could you help to have a try ?

superm1 · 2020-05-18T12:38:41Z

I think specifically they're building using CC environment variable, which is something the kernel allows but dkms currently resets the variable to avoid problems.

GoPerry · 2020-05-19T08:27:06Z

I did some build test on this.
the reason should be that the dkms kernel module should be compiled with the same compiler the kernel was built with.
and i found that Clang does NOT recognize -Wno-unused-but-set-variable.

#build error output
error: unknown warning option '-Wno-unused-but-set-variable'; did you mean '-Wno-unused-const-variable'? [-Werror,-Wunknown-warning-option]

This error Cflag is defined in kernel top Makefile.

KBUILD_CFLAGS += -Wno-unused-but-set-variable

So i would say that we cannot compile the dkms with clang in this situation.

superm1 · 2020-05-20T01:02:20Z

@ColinIanKing would it be possible for the kernel to store somewhere the exact compiler/ environment /flags used? I think sourcing that from DKMS is probably the safest solution.

ColinIanKing · 2020-05-20T08:23:29Z

@superm1, yes, that is a good idea - I can dump these out before running dkms - is there currently a mechanism for DKMS to pull these in by sourcing them?

superm1 · 2020-05-20T20:31:35Z

To me the easiest hack will be in /etc/dkms_framework.conf (or something that you pass in as --dkmsframework).

But if we want this to become a standard place that DKMS reads to get the information of the compiled kernel, probably needs some alignment to make the upstream kernel dump this somewhere too.

torvic9 · 2020-08-26T11:44:35Z

Flexible compiler support would be highly appreciated.
Up to now, I pass something like MAKE[0]='make CC=clang LD=ld.lld KERNELRELEASE=${_kernver}' to dkms.conf when I want to use clang.

dileks · 2020-08-27T07:39:28Z

I have pinged the Debian VirtualBox maintainers again and asked for a handling like in acpi-call-dkms to use the MAKE[0] and CLEAN lines:

[ /usr/src/acpi-call-1.1.0/dkms.conf ]

PACKAGE_NAME="acpi-call"
PACKAGE_VERSION="1.1.0"
MAKE[0]="make -C ${kernel_source_dir} M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build"
CLEAN="make -C ${kernel_source_dir} M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build clean"
BUILT_MODULE_NAME[0]="acpi_call"
DEST_MODULE_LOCATION[0]="/extra"
AUTOINSTALL="yes"

# Don't build module when ACPI is disabled
if ! grep -q "^CONFIG_ACPI=[ym]$" "$kernel_source_dir/.config" 2>/dev/null ; then
  echo "ACPI disabled in this kernel, not building module." >&2
  BUILD_EXCLUSIVE_ARCH="^do_not_build$"
fi

My dkms.conf (to support gcc-10 and ld.bfd with some debug-options like make V=1):

[ /usr/src/virtualbox-6.1.12/dkms.conf ]

PACKAGE_NAME="virtualbox"
PACKAGE_VERSION="6.1.12"
CC_FOR_BUILD="gcc-10"
LD_FOR_BUILD="ld.bfd"
MAKE_OPTS="CC=${CC_FOR_BUILD} LD=${LD_FOR_BUILD}"
MAKE[0]="make V=1 ${MAKE_OPTS} -C ${kernel_source_dir} M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build"
CLEAN="make -C ${kernel_source_dir} M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build clean"
BUILT_MODULE_NAME[0]="vboxdrv"
BUILT_MODULE_LOCATION[0]="vboxdrv"
DEST_MODULE_LOCATION[0]="/updates"
BUILT_MODULE_NAME[1]="vboxnetadp"
BUILT_MODULE_LOCATION[1]="vboxnetadp"
DEST_MODULE_LOCATION[1]="/updates"
BUILT_MODULE_NAME[2]="vboxnetflt"
BUILT_MODULE_LOCATION[2]="vboxnetflt"
DEST_MODULE_LOCATION[2]="/updates"
AUTOINSTALL="yes"

NOTE: Of course, you can pass your distro's default compiler (gcc) and linker (ld).

For me the above works as virtualbox-dkms Debian package does not often change/upgrade.

[1] ClangBuiltLinux/linux#1104 (comment)

torvic9 · 2020-08-27T09:44:27Z

Here's what I use for virtualbox on Arch (excerpt from PKGBUILD):

prepare() {
  _kernver="$(cat /usr/lib/modules/$_extramodules/version)"
  cp /usr/src/vboxhost-${_pkgver}/dkms.conf $srcdir/dkms.conf
  if [[ $_clang -eq 1 ]]; then
	LLVM_UTILS="CC=clang CXX=clang++ LD=ld.lld AR=llvm-ar NM=llvm-nm OBJCOPY=llvm-objcopy OBJSIZE=llvm-size STRIP=llvm-strip"
	echo "MAKE[0]=\"make ${LLVM_UTILS} KERNELRELEASE=${_kernver}\"" >> $srcdir/dkms.conf
  fi
}

build() {
  _kernver="$(cat /usr/lib/modules/$_extramodules/version)"
  # build host modules
  fakeroot dkms build -c $srcdir/dkms.conf --dkmstree "$srcdir" -m vboxhost/${_pkgver} -k "${_kernver}"
}

(I copy over the dkms.conf to the build directory, in order to prevent changing the default one.)

dileks · 2020-08-27T11:34:43Z

@torvic9

Thanks for telling the other half of the story.

torvic9 · 2020-08-27T14:14:20Z

@dileks
I don't understand, did I forget something?

dileks · 2020-08-27T14:16:02Z

You completed your story.

torvic9 · 2020-08-27T14:20:34Z

Well there's nothing really new. Just thought it would be a good idea to post some contents of the PKGBUILD.

nickdesaulniers · 2020-08-28T14:57:55Z

This error Cflag is defined in kernel top Makefile.
KBUILD_CFLAGS += -Wno-unused-but-set-variable
So i would say that we cannot compile the dkms with clang in this situation.

If that's from Kbuild, it should get wrapped in cc-option. If it's DKMS' makefile, it shouldn't require that the compiler used support specific warning flags.

nickdesaulniers · 2021-04-28T21:21:59Z

@GoPerry do you have thoughts on not requiring specific warning flags, just like the Linux kernel's Kbuild doesn't, via cc-option macros?

nickdesaulniers · 2021-04-28T21:28:38Z

are the examples how to build anything with dkms? It would be nice to have some steps to reproduce for others to understand the problem.

GoPerry · 2021-04-29T08:24:24Z

@GoPerry do you have thoughts on not requiring specific warning flags, just like the Linux kernel's Kbuild doesn't, via cc-option macros?

do you mean that the dkms will use system default compiler which is set by "cc option"
cc = clang or cc = gcc ?

ColinIanKing · 2021-04-29T16:17:12Z

Just my 2 cents worth:

One can check of the kernel has been built using clang using:
sed -n 's/^[^$]*[(][^$]*[)][ ]*[^$]*[(]\([^$]*\)[)].*/\1/p' /proc/version | grep -q clang
If the above returns 0 then one needs to set the dkms make_command string to contain:

CC=clang CXX=clang++ LD=ld.lld AR=llvm-ar NM=llvm-nm OBJCOPY=llvm-objcopy OBJSIZE=llvm-size STRIP=llvm-strip

as per nickdesaulniers comment, a simple dkms example can be found in https://kernel.ubuntu.com/~cking/hello-0.1.tar - put the contents into /usr/src and do:

sudo dkms add -m hello -v 0.1
sudo dkms build -m hello -v 0.1
sudo dkms install -m hello -v 0.1
sudo modprobe hello

ColinIanKing · 2021-04-29T16:43:06Z

This hack seems to work for me:

diff --git a/dkms b/dkms
old mode 100644
new mode 100755
index b5ffa30..c327400
--- a/dkms
+++ b/dkms
@@ -692,6 +692,9 @@ read_conf()
 
     # Get the correct make command
     [[ ${MAKE_MATCH[0]} ]] || make_command="${MAKE[0]}"
+    if sed -n 's/^[^\(]*[(][^\)]*[)][ ]*[^\(]*[(]\([^\)]*\)[)].*/\1/p' /proc/version | grep -q clang ; then
+        make_command="make V=1 CC=clang CXX=clang++ LD=ld.lld AR=llvm-ar NM=llvm-nm OBJCOPY=llvm-objcopy OBJSIZE=llvm-size STRIP=llvm-strip"
+    fi
     for ((index=0; index < ${#MAKE[@]}; index++)); do
     [[ ${MAKE[$index]} && ${MAKE_MATCH[$index]} && \
         $1 =~ ${MAKE_MATCH[$index]} ]] && \

But I'm not a dkms expert, so I expect this is wrong on a few levels.

nickdesaulniers · 2021-04-29T18:08:46Z

That looks like it's parsing the contents of the current running kernel (/proc/version). Let's say I'm currently running a kernel built with GCC but building a kernel (and dkms modules) with Clang. Would it be better to get the toolchain from the version string from the vmlinux binary?

do you mean that the dkms will use system default compiler which is set by "cc option"

No; and I don't expect users to be changing their system default CC to clang; that's way too error prone. Instead build systems should allow for CC to be passed via env var to override the system default when appropriate.

ColinIanKing · 2021-04-29T19:09:11Z

Getting the toolchain version from the vmlinux binary makes total sense, completely overlooked that.

My original thoughts were that a clang built kernel should have dkms modules built for it using clang as I was under the impression that the modules needed to be built with clang to match the internal kernel ABIs. But I suppose if they are compatible ELF binaries that assumption maybe wrong.

torvic9 · 2021-04-30T08:19:04Z

Getting the toolchain version from the vmlinux binary makes total sense

Out of curiosity, how do you extract that information from vmlinux?

ColinIanKing · 2021-04-30T08:26:21Z

If the kernel has the config settings built in one can check for this in /proc/config and grep for the various config settings such as CONFIG_CC_VERSION_TEXT or CONFIG_CC_IS_GCC. However, quite a few distros disable this feature.

Or use https://stuff.mit.edu/afs/sipb/contrib/linux/scripts/extract-ikconfig for the binary vmlinux image.

torvic9 · 2021-04-30T08:51:24Z

Thanks! That's very useful.
Something like zgrep -i gcc_version /proc/config.gz | awk -F'=' '{print $2}' , which should return 0 with clang.
Or checking CC_IS_GCC or CC_IS_CLANG.

ColinIanKing · 2021-04-30T08:55:25Z

Yes, but remember that some kernels don't have this config built in, so the vmlinux image also needs to be scanned as a fallback :-/

dileks · 2021-04-30T08:57:48Z

You can use (llvm-)readelf to get the information (here: selfmade LLVM toolchain v12.0.0):

$ llvm-readelf-12 -p .comment arch/x86/boot/compressed/vmlinux
String dump of section '.comment':
[     0] dileks clang version 12.0.0 (https://github.com/llvm/llvm-project.git d28af7c654d8db0b68c175db5ce212d74fb5e9bc)
[    70] Linker: LLD 12.0.0 (https://github.com/llvm/llvm-project.git d28af7c654d8db0b68c175db5ce212d74fb5e9bc)

$ llvm-readelf-12 -p .comment vmlinux
String dump of section '.comment':
[     0] dileks clang version 12.0.0 (https://github.com/llvm/llvm-project.git d28af7c654d8db0b68c175db5ce212d74fb5e9bc)
[    70] Linker: LLD 12.0.0 (https://github.com/llvm/llvm-project.git d28af7c654d8db0b68c175db5ce212d74fb5e9bc)

torvic9 · 2021-04-30T09:08:51Z

LTO does not work here with virtualbox-dkms, see my yesterday's comment here: ClangBuiltLinux/linux#1104
IAS doesn't seem to pose problems.

nickdesaulniers · 2021-05-03T23:49:51Z

Getting the toolchain version from the vmlinux binary makes total sense

Out of curiosity, how do you extract that information from vmlinux?

You can use (llvm-)readelf to get the information (here: selfmade LLVM toolchain v12.0.0):

$ llvm-readelf-12 -p .comment arch/x86/boot/compressed/vmlinux
String dump of section '.comment':
[     0] dileks clang version 12.0.0 (https://github.com/llvm/llvm-project.git d28af7c654d8db0b68c175db5ce212d74fb5e9bc)
[    70] Linker: LLD 12.0.0 (https://github.com/llvm/llvm-project.git d28af7c654d8db0b68c175db5ce212d74fb5e9bc)

$ llvm-readelf-12 -p .comment vmlinux
String dump of section '.comment':
[     0] dileks clang version 12.0.0 (https://github.com/llvm/llvm-project.git d28af7c654d8db0b68c175db5ce212d74fb5e9bc)
[    70] Linker: LLD 12.0.0 (https://github.com/llvm/llvm-project.git d28af7c654d8db0b68c175db5ce212d74fb5e9bc)

This is the recommended answer. If you're in a hurry:

$ strings vmlinux | grep 'Linux version'
Linux version 5.12.0+ (ndesaulniers@ndesaulniers1.mtv.corp.google.com) (Nick Desaulniers clang version 13.0.0 (git@github.com:llvm/llvm-project.git 07128a828bc9ec24c6bf788f34c324d9db4762f6), LLD 13.0.0 (git@github.com:llvm/llvm-project.git 07128a828bc9ec24c6bf788f34c324d9db4762f6)) #22 SMP Fri Apr 30 19:08:59 PDT 2021

OneOfOne · 2021-07-02T17:15:45Z

Is there any updates to this? shouldn't dkms try to use the same compiler as the kernel instead of us hacking random files?

This gets old fast when you have multiple dkms modules (nvidia, virtualbox, ntfs3)

torvic9 · 2021-07-02T17:44:13Z

Using llvm-readelf does not seem to work, I get a "could not find section .comment" error message. Maybe that's because the binaries were stripped?

@OneOfOne, I agree that DKMS should find the proper settings by itself, but don't ask me how it can be done best...
I intend to submit a patch that allows users to choose a directory for the ThinLTO cache once I get "approval" from people who know much more about this stuff than I do :)
A recent version is here: https://gist.github.com/torvic9/17c6a86ca4345d62e7907c96231c2775

StatusCode404 · 2021-07-13T03:33:05Z

Hi Guys, does anyone know if this will get fixed? We need DKMS to figure out which build-tools to use based off the way the kernel was built otherwise DKMS will fail for LLVM/Clang built kernels especially now that the flto=thin genie is out of the box!

Without hacking continually we can't roll with newer kernel versions easily if we rely on DKMS for many applications and drivers. For my case which I'm guessing is a common use-case, Virtualbox! It's one of the most common desktop VM Managers and we can't use it with clang LTO'ed kernels!.

Please someone fix this!

torvic9 · 2021-07-13T06:00:44Z

Regarding the LTO thing, I submitted the potential workaround mentioned above upstream:
https://lore.kernel.org/linux-kbuild/b45b2430-3670-b310-b6ad-2d6db50c2d18@mailbox.org/

StatusCode404 · 2021-07-13T06:33:47Z

Thanks torvic9!

However that only addresses part of the problem if I understand the PR correctly, it only addresses the location of the cache needed for LTO'ing.

Does it address how to figure out whether to use gcc and its associated build tools or llvm/clang? I think that's the biggest problem.

torvic9 · 2021-07-13T09:58:05Z

Does it address how to figure out whether to use gcc and its associated build tools or llvm/clang? I think that's the biggest problem.

No, this only covers LTO.

For those who are running Arch, here is a very dirty and quick fix for DKMS, adding a simple clang check:
https://gist.github.com/torvic9/3d7605f54dc6ad07fe340dd181d7a22c

I don't put it into a PR because, as said, it's dirty, it misses a couple of checks and is only for Arch, where I succesfully tested it with VBox.
More adept people are welcome to set this up in a proper way, IMO there is no major difficulty in adding a check to DKMS, although distro-specific differences have to be taken into account.

torvic9 · 2021-07-13T12:45:29Z

Funny things happening.
Please take with a pinch of salt, as I'm not sure about this:
With the above DKMS patch, it seems that the ThinLTO patch is not needed??
Any Arch users here who can test this please?

EDIT: I did something wrong, please disregard.

torvic9 · 2021-07-13T15:12:50Z

After another round of testing on Arch, I can say that the combination of the ThinLTO patch (I will send a new version shortly) and the patched DKMS script results in correct automatic building of the VBox modules.

nickdesaulniers · 2021-08-06T21:31:49Z

adding a simple clang check:
https://gist.github.com/torvic9/3d7605f54dc6ad07fe340dd181d7a22c

Probably slightly safer to check via:

$ readelf -p .comment vmlinux

String dump of section '.comment':
  [     0]  Nick Desaulniers clang version 14.0.0 (git@github.com:llvm/llvm-project.git 9e729e177aa4631d45b5370e7d13929b52599865)
  [    76]  Linker: LLD 14.0.0 (git@github.com:llvm/llvm-project.git 9e729e177aa4631d45b5370e7d13929b52599865)

or

$ strings vmlinux | grep "Linux version" | grep clang
Linux version 5.14.0-rc3-00353-ga8554b46d7bf (ndesaulniers@ndesaulniers1.mtv.corp.google.com) (Nick Desaulniers clang version 14.0.0 (git@github.com:llvm/llvm-project.git 9e729e177aa4631d45b5370e7d13929b52599865), LLD 14.0.0 (git@github.com:llvm/llvm-project.git 9e729e177aa4631d45b5370e7d13929b52599865)) #45 SMP Mon Aug 2 16:33:14 PDT 2021

foundObjects · 2021-08-26T23:43:08Z

After another round of testing on Arch, I can say that the combination of the ThinLTO patch (I will send a new version shortly) and the patched DKMS script results in correct automatic building of the VBox modules.

Did you ever finish your patch for this? I'd like to start experimenting with clang LTO kernel builds here and dkms is a blocker.

torvic9 · 2021-08-27T07:23:10Z

Did you ever finish your patch for this? I'd like to start experimenting with clang LTO kernel builds here and dkms is a blocker.

Patch was rejected upstream for valid reasons.
Still it's good enough for my personal purposes. Together with the dirty "clang check" patch above, LTO kernel plus virtualbox extramodules build just fine here.
Nick's check is probably safer, but using readelf to grep for the .comment section doesn't seem to work with stripped kernels.

foundObjects · 2021-08-28T10:48:34Z

Patch was rejected upstream for valid reasons.
Still it's good enough for my personal purposes. Together with the dirty "clang check" patch above, LTO kernel plus virtualbox extramodules build just fine here.
Nick's check is probably safer, but using readelf to grep for the .comment section doesn't seem to work with stripped kernels.

Yeah, I think the dirty clang check is the way to go simply because there are so many kernel packages shipped with stripped binaries.

I'm a little confused (sorry!); you're using the patched dkms script linked in the gist above and what else?

torvic9 · 2021-08-29T07:31:42Z

and what else?

A patch which allows you to choose a custom thinlto-cache directory (see above).

nickdesaulniers · 2021-10-30T11:29:00Z

Ok, I got this shit to work. The answer is buried up in @ColinIanKing 's comment above.

Problem number 1: dkms doesn't use the same toolchain as the kernel when building out of tree modules:

--- dkms.orig   2021-10-30 03:22:06.695198591 -0700
+++ dkms        2021-10-30 03:50:53.636569028 -0700
@@ -703,6 +703,17 @@
         [[ ! $clean ]] && clean="make -C $kernel_source_dir M=$dkms_tree/$module/$module_version/build clean"
     fi
 
+    if [[ -e $kernel_source_dir/vmlinux ]]; then
+      if  readelf -p .comment $kernel_source_dir/vmlinux | grep -q clang; then
+        make_command="${make_command} CC=clang"
+      fi
+    fi
+    if [[ -e $kernel_source_dir/vmlinux ]]; then
+      if  readelf -p .comment $kernel_source_dir/vmlinux | grep -q LLD; then
+        make_command="${make_command} LD=ld.lld"
+      fi
+    fi
+
     # Set modules_conf_array
     for ((index=0; index < ${#MODULES_CONF[@]}; index++)); do
         [[ ${MODULES_CONF[$index]} ]] && modules_conf_array[$index]="${MODULES_CONF[$index]}"

$ cp /usr/sbin/dkms dkms
$ vim dkms
$ sudo cp dkms /usr/sbin/dkms

This doesn't precisely use the exact same tools necessarily for the out of tree modules, but it should invoke make with the two (explicitly, three implicitly) tools that matter; the compiler (clang) and linker (lld) (with clang's integrated assembler being the third). We can't test which assembler was used for the kernel, though we could check the .config file for CONFIG_AS_IS_LLVM=y vs CONFIG_AS_IS_GNU=y. Of course, vmlinux or .config might not actually be the kernel we're interested in building the out of tree module against; same could be said for /boot/config-$(uname -r) and /boot/vmlinuz-$(uname -r).

Problem number 2: I don't like to install clang to /usr/bin/ and friends, especially on the machine I'm developing LLVM on. DKMS currently limits the PATH to specifically `/bin:/sbin:/usr/bin:/usr/sbin:/usr/lib/dkms". So I at least need to create temporary symlinks:

$ sudo ln -s `which clang` /usr/bin/clang
$ sudo ln -s `which ld.lld` /usr/bin/ld.lld
$ sudo rm -rf /var/lib/dkms/openrazer-driver/2.8.0 
$  sudo dkms build -m openrazer-driver -v 2.8.0

Creating symlink /var/lib/dkms/openrazer-driver/2.8.0/source ->
                 /usr/src/openrazer-driver-2.8.0

DKMS: add completed.

Kernel preparation unnecessary for this kernel.  Skipping...Building module:
cleaning build area...
KERNELDIR=/lib/modules/5.15.0-rc7+/build make driver CC=clang LD=ld.lld.....
cleaning build area...

DKMS: build completed.
$ sudo dkms install -m openrazer-driver -v 2.8.0

razerkbd.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.15.0-rc7+/kernel/drivers/hid/
razermouse.ko:
Running module version sanity check. - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.15.0-rc7+/kernel/drivers/hid/

razermousemat.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.15.0-rc7+/kernel/drivers/hid/

razerkraken.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.15.0-rc7+/kernel/drivers/hid/

razeraccessory.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.15.0-rc7+/kernel/drivers/hid/

razercore.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.15.0-rc7+/kernel/drivers/hid/

depmod...

DKMS: install completed.
$ sudo insmod /var/lib/dkms/openrazer-driver/2.8.0/5.15.0-rc7+/x86_64/module/razercore.ko
$ dmesg | tail -n2
[ 4612.792088] razercore: loading out-of-tree module taints kernel.
[ 4612.792128] razercore: module verification failed: signature and/or required key missing - tainting kernel
$ lsmod | grep razer 
razercore              40960  0
hid                   135168  5 i2c_hid,razercore,usbhid,hid_generic,hid_rmi
$ sudo rm /usr/bin/clang /usr/bin/ld.lld

When building the Linux kernel, one may use `make LLVM=1` to set multiple flags, akin to `make CC=clang LD=ld.lld NM=llvm-nm ...` (see the link below for kernel docs explaining this further in detail). When building kernel modules, to ensure we're using the same toolchain as the underlying core kernel image, Kbuild will error if it's reinvoked with a toolchain that differs. This causes DKMS to fail, since it's not re-specifying the same compiler (or linker, etc). Check the .comment section of the vmlinux file in the Linux kernel source dir, and set CC= and LD= flags for make based on that. Alternatively or additionally, we could also check for the presence of CONFIG_CC_IS_CLANG=y and CONFIG_LD_IS_LLD=y in the .config in the kernel sources. We could also check the .comment section of the booted kernel. Fixes: dell#124 Link: https://docs.kernel.org/kbuild/llvm.html Suggested-by: Colin Ian King <colin.king@canonical.com> Signed-off-by: Nick Desaulniers <nick.desaulniers@gmail.com>

When building the Linux kernel, one may use `make LLVM=1` to set multiple flags, akin to `make CC=clang LD=ld.lld NM=llvm-nm ...` (see the link below for kernel docs explaining this further in detail). When building kernel modules, to ensure we're using the same toolchain as the underlying core kernel image, Kbuild will error if it's reinvoked with a toolchain that differs. This causes DKMS to fail, since it's not re-specifying the same compiler (or linker, etc). Check the .comment section of the vmlinux file in the Linux kernel source dir, and set CC= and LD= flags for make based on that. Alternatively or additionally, we could also check for the presence of CONFIG_CC_IS_CLANG=y and CONFIG_LD_IS_LLD=y in the .config in the kernel sources. We could also check the .comment section of the booted kernel. Fixes: dell#124 Fixes: ClangBuiltLinux/linux#1104 Link: https://docs.kernel.org/kbuild/llvm.html Suggested-by: Colin Ian King <colin.king@canonical.com> Signed-off-by: Nick Desaulniers <nick.desaulniers@gmail.com>

scaronni · 2021-11-04T10:54:48Z

This doesn't precisely use the exact same tools necessarily for the out of tree modules, but it should invoke make with the two (explicitly, three implicitly) tools that matter; the compiler (clang) and linker (lld) (with clang's integrated assembler being the third). We can't test which assembler was used for the kernel, though we could check the .config file for CONFIG_AS_IS_LLVM=y vs CONFIG_AS_IS_GNU=y. Of course, vmlinux or .config might not actually be the kernel we're interested in building the out of tree module against; same could be said for /boot/config-$(uname -r) and /boot/vmlinuz-$(uname -r).

As pointed in the pull request, many distributions use a bzImage compressed kernel, and the detection logic does not work there. I think parsing the .config file is the best options.

Problem number 2: I don't like to install clang to /usr/bin/ and friends, especially on the machine I'm developing LLVM on. DKMS currently limits the PATH to specifically `/bin:/sbin:/usr/bin:/usr/sbin:/usr/lib/dkms".

I think we can safely remove this and let it just use system defined paths.

torvic9 · 2021-11-04T11:43:46Z

distributions use a bzImage compressed kernel

Yes, but is the vmlinux in $kernel_source_dir really compressed?
At least not here on Arch, which also uses bzImage, but only on the actual kernel residing in /boot.
The uncompressed vmlinux is copied over to /lib/modules when the kernel headers are installed.

readelf -p .comment /lib/modules/5.14.15-arch1-1/build/vmlinux                                                                                                   

String dump of section '.comment':
  [     0]  GCC: (GNU) 11.1.0

What I personally do is to simply grep for the clang string in vmlinux, something like this:

+    set_kernel_source_dir "$kernelver"
+    if [[ ! -z $(strings ${kernel_source_dir}/vmlinux | grep clang) ]] ; then
+        clangopts=("CC=clang" "LD=ld.lld" "AR=llvm-ar" "NM=llvm-nm" \
+        	"RANLIB=llvm-ranlib" "STRIP=llvm-strip" "OBJCOPY=llvm-objcopy" \
+        	"OBJDUMP=llvm-objdump" "READOBJ=llvm-readobj")
+        echo "Using llvm tools from $(which clang)"
+    else
+        clangopts=()
+    fi

Then add clangopts array to the make command.
(I don't think it's necessary to pass all the llvm tools)

scaronni · 2021-11-06T12:57:58Z

distributions use a bzImage compressed kernel

Yes, but is the vmlinux in $kernel_source_dir really compressed? At least not here on Arch, which also uses bzImage, but only on the actual kernel residing in /boot. The uncompressed vmlinux is copied over to /lib/modules when the kernel headers are installed.

Quite different on Fedora/Red Hat. Here is the full structure:

$ pwd
/usr/lib/modules/5.14.16-301.fc35.x86_64
$ file vmlinuz
vmlinuz: Linux kernel x86 boot executable bzImage, version 5.14.16-301.fc35.x86_64 (mockbuild@bkernel01.iad2.fedoraproject.org) #1 SMP Wed Nov 3 13:55:42 UTC 2021, RO-rootFS, swap_dev 0xA, Normal VGA
$ find build/ -name "vmlinu*"
build/vmlinux.h
build/include/asm-generic/vmlinux.lds.h
build/scripts/gdb/vmlinux-gdb.py
build/vmlinux.id

There is no uncompressed kernel image left around. Besides this, I don't think we should really parse binaries around.
What about setting something in /etc/dkms/framework.conf by the distro maintainers that is then used to compile modules depending on how you compile the kernel? We're trying to avoid putting too many distribution specifics in the latest code cleanups.

When building the Linux kernel, one may use `make LLVM=1` to set multiple flags, akin to `make CC=clang LD=ld.lld NM=llvm-nm ...` (see the link below for kernel docs explaining this further in detail). When building kernel modules, to ensure we're using the same toolchain as the underlying core kernel image, Kbuild will error if it's reinvoked with a toolchain that differs. This causes DKMS to fail, since it's not re-specifying the same compiler (or linker, etc). Check the .comment section of the vmlinux file in the Linux kernel source dir, and set CC= and LD= flags for make based on that. If vmlinux does not exist, grep .config for CONFIG_CC_IS_CLANG=y and CONFIG_LD_IS_LLD=y. Fixes: dell#124 Fixes: ClangBuiltLinux/linux#1104 Link: https://docs.kernel.org/kbuild/llvm.html Suggested-by: Colin Ian King <colin.king@canonical.com> Signed-off-by: Nick Desaulniers <nick.desaulniers@gmail.com>

nickdesaulniers · 2021-11-19T20:59:51Z

There is no uncompressed kernel image left around.

How does one get debug info for their kernel then? Usually, during the production of bzImage, vmlinux is stripped of debug info so that it doesn't occupy space on disk or take longer to decompress. For Android, we usually have kernel build artifacts of the compressed kernel image, and the vmlinux with debug symbols.

GoPerry added the enhancement label May 17, 2020

torvic9 mentioned this issue Aug 27, 2020

Building VirtualBox kernel modules with Clang and DKMS ClangBuiltLinux/linux#1104

Closed

flindeberg mentioned this issue Aug 7, 2021

Rudimentary clang / llvm support? Frogging-Family/nvidia-all#55

Closed

nickdesaulniers mentioned this issue Oct 30, 2021

Support CC=clang and LD=ld.lld #169

Merged

scaronni closed this as completed in #169 Nov 20, 2021

This was referenced May 26, 2023

Compiling your own kernel module 🏗️ OpenMandrivaAssociation/distribution#2919

Open

Add dkms.conf t2linux/apple-bce-drv#2

Open

building with clang rather than gcc #124

building with clang rather than gcc #124

Comments

ColinIanKing commented May 1, 2020

superm1 commented May 15, 2020

GoPerry commented May 17, 2020

GoPerry commented May 18, 2020

superm1 commented May 18, 2020

GoPerry commented May 19, 2020 • edited Loading

superm1 commented May 20, 2020

ColinIanKing commented May 20, 2020

superm1 commented May 20, 2020

torvic9 commented Aug 26, 2020

dileks commented Aug 27, 2020

torvic9 commented Aug 27, 2020 • edited Loading

dileks commented Aug 27, 2020

torvic9 commented Aug 27, 2020

dileks commented Aug 27, 2020

torvic9 commented Aug 27, 2020

nickdesaulniers commented Aug 28, 2020

nickdesaulniers commented Apr 28, 2021

nickdesaulniers commented Apr 28, 2021

GoPerry commented Apr 29, 2021

ColinIanKing commented Apr 29, 2021 • edited Loading

ColinIanKing commented Apr 29, 2021

nickdesaulniers commented Apr 29, 2021

ColinIanKing commented Apr 29, 2021 • edited Loading

torvic9 commented Apr 30, 2021

ColinIanKing commented Apr 30, 2021 • edited Loading

torvic9 commented Apr 30, 2021

ColinIanKing commented Apr 30, 2021

dileks commented Apr 30, 2021

torvic9 commented Apr 30, 2021

nickdesaulniers commented May 3, 2021

OneOfOne commented Jul 2, 2021

torvic9 commented Jul 2, 2021

StatusCode404 commented Jul 13, 2021

torvic9 commented Jul 13, 2021

StatusCode404 commented Jul 13, 2021

torvic9 commented Jul 13, 2021 • edited Loading

torvic9 commented Jul 13, 2021 • edited Loading

torvic9 commented Jul 13, 2021

nickdesaulniers commented Aug 6, 2021

foundObjects commented Aug 26, 2021

torvic9 commented Aug 27, 2021

foundObjects commented Aug 28, 2021

torvic9 commented Aug 29, 2021 • edited Loading

nickdesaulniers commented Oct 30, 2021

scaronni commented Nov 4, 2021

torvic9 commented Nov 4, 2021

scaronni commented Nov 6, 2021

nickdesaulniers commented Nov 19, 2021

GoPerry commented May 19, 2020 •

edited

Loading

torvic9 commented Aug 27, 2020 •

edited

Loading

ColinIanKing commented Apr 29, 2021 •

edited

Loading

ColinIanKing commented Apr 29, 2021 •

edited

Loading

ColinIanKing commented Apr 30, 2021 •

edited

Loading

torvic9 commented Jul 13, 2021 •

edited

Loading

torvic9 commented Jul 13, 2021 •

edited

Loading

torvic9 commented Aug 29, 2021 •

edited

Loading