Missing graphics drivers in kernel configuration

[WasIstDasDingDa]

When configuring the patched kernel, under Graphics support only ATI Radeon and AMD GPU are selectable. I need the Nouveau driver for my old NVIDIA card.

[NTULINUX]

The nouveau driver enables/requires WMI, both drivers can cause terrible latency, kernel warnings/panics and other problems. It is advised to use software acceleration for those with nVIDIA and Intel cards, however due to the popularity of the Intel graphics, that driver may be re-enabled.

In either case, I am unable to test IPIPE with either nVIDIA or Intel graphics due to all my hardware being AMD.

If I re-enable the nouveau driver, will you post your latency scores?

Thanks!

[WasIstDasDingDa]

Thank you for your reply. Of course I would post latency scores if you enable nouveau for me and the result would be interesting for you.
On the other hand, knowing now about a possible regression, I would have no problem buying a cheap Radeon HD 6450 card. Then a direct comparison would also be possible. Besides the processor is a 3200 MHz quad-core Phenom and the base system is OpenSUSE Leap 15.1.

When compiling RTAI there was another problem. To build the RTAI kernel modules I had to run make modules in the kernel tree to get the required file Module.symvers, this is not in your instructions.

[NTULINUX]

make compiles the modules, make modules_install installs the modules. make modules is only required if you used make bzImage instead of make. If you're sure you ran make and not make bzImage then OpenSUSE's build system must be weird (modified version of GNU make or similar.)

I will get around to re-enabling the drivers, in the mean time, have you ran the latency test on your hardware to make sure you don't have any BIOS problems with RTAI? If latency is good now, when the driver is re-enabled and it's high, you'll know why. I'll also bump the kernel version as well.

Thanks!

[WasIstDasDingDa]

Thanks for your effort to help and overall for putting together complex pieces of software.

Yes, the building of RTAI was aborted with a message about a missing symvers file, after make modules this was there and everything was fine.
Before that, I had built and installed the kernel according to your instructions. The system also booted into the new kernel, all necessary modules were present (without make modules until then, only nouveau was missing). There was the effect that the kernel had loaded radeon and I ended up in a kind of emergency console in VGA mode. But I could startx as root in VESA mode 1024x768 and then build RTAI.

The fact that the RTAI modules for the latency test could not be loaded is my fault, because in the meantime I played around with kernel config in search of nouveau. There was a change in the custom string between make and make modules and the RTAI modules got a wrong kernel version. I'm going to repeat the entire procedure today without mismatch and for the interim with disabled radeon. In the meantime I have land in sight and already ordered the mentioned Radeon card.

Actually I'm just a somewhat advanced Linux user and just wanted to mill, thanks again for the help.
Next time I'll report success.

[WasIstDasDingDa]

Thanks for the k̶correction, there are just some small differences between German and English ;-)

[WasIstDasDingDa]

Therefore I also use KDE and not CDE

[WasIstDasDingDa]

I have succeeded, the following I have done with the kernel after patching and configuration:

make bzImage -j4 && make modules -j4
make install && make modules_install
dracut --kver 4.14-148-RTAI-5.2.3-2
update-bootloader

The building of RTAI was then straightforward, the latency test gave good results right away (at least I think so).

The kernel had not loaded an extra graphics driver, the X server had loaded vesa and fbdev, I don't know which one was used.

Thanks!

[NTULINUX]

Looks good, glad all is OK! 6 microseconds isn't bad at all, I'll add the nouveau driver and just to cover more distros, I will change make to make bzImage && make modules to ensure both are always built.

[WasIstDasDingDa]

It was still LinuxCNC to build. In this context I also experimented with Debian 10.3 on a virtual and a real machine.

The invocations to build the kernel like in OpenSUSE (Leap 15.1) definitely don't work with Debian (10.3.)
But your instructions are not quite correct either, it just worked for me:

make    /* also builds the modules and generates Module.symvers file */
make modules_install && make install   /* it only works in this order */

After that everything is done, initrd has been generated and the bootloader has been updated.

Building LinuxCNC Master based on a self-built RTAI kernel also stumbles a bit under Debian 10.3, more about that later.

[NTULINUX]

All DRM drivers are re-enabled, let me know how it goes!

[WasIstDasDingDa]

The dedicated computer now runs Debian Stretch and I build the kernels conveniently the Debian way. Below some test results:

AMD vs. nVIDIA
RTAI, test-suite RTAI, µs
             AMD        nVIDIA
idle          7            6
burdened     16           21


RTAI vs. Preemt-RT
AMD, burdened, latency-test LinuxCNC RIP, Servo Thread/Base Thread, µs
RTAI         13/16
Preemt-RT    17/37

AMD:        Radeon HD 6450
nVIDIA:     Geforce GT610
idle:       no App, without mouse movements and actions
burdened:   glxgears + glxheads + yt HD-Video
RTAI:       4.14.148-rtai-amd64 (Alec)
Preemt-RT:  4.9.0-12-rt-amd64 (Debian)

Then I have another problem. In order to build Debian packages of LinuxCNC to install to /usr, the kernel and RTAI must also be available in this form.
For the kernel the means are available, for RTAI so-so.
There is an deprecated article on this topic (http://wiki.linuxcnc.org/cgi-bin/wiki.pl?Building_RTAI_5.2_Packages),
where the instructions do not match the recent content of /debian. Can you help?

[NTULINUX]

Which instructions do not match? I'll need specifics.

[WasIstDasDingDa]

Relevant part of the article referred to

1 Configure, build and install RTAI (MUST BE BOOTED INTO YOUR NEW RTAI KERNEL)
2
3 $ cd ~/RTAI
4 $ ./autogen.sh
5 $ ./configure
6
7 Now set up debian. The NTULINUX repo now contains the magic sauce from:
8 https://github.com/SebKuzminsky/rtai/tree/old-3.9-debs/debian
9
10 Consider changing the maintainer name in configure.in Create a git tag to have a release number:
11 git tag -a 5.2.2-linuxcnc -m "experimental RTAI debs for LinuxCNC"
12
13 update the changelog:
14
15 $ debian/update-dch-from-git
16 $ debian/configure 4.14.148 rtai amd64
17
18 build packages:
19
20 $ dpkg-buildpackage
21
22 This will give many messages about missing math functions. Ignore these.
23 There will also be an error about package signing and gpg, I need to find out how to fix that,
24 but the debs are still built.
25
26 YOU SHOULD NOW HAVE PACKAGES FOR RTAI. (in the level below /RTAI )
27
28 Install them
29
30 $ sudo dpkg -i rtai-modules-4.14.148_5.2.2_amd64.deb

Line 10 "... name in configure.in"
I can't find a configure.in file

Line 11 ok, no output

Line 15 "$ debian/update-dch-from-git"
The output of the invocation is:

+ '[' '!' -d debian -o '!' -d base -o '!' -d testsuite ']'
+ echo 'this script must be run from the root of the source tree (the directory with debian and src in it)'
this script must be run from the root of the source tree (the directory with debian and src in it)
+ exit 1

I copied this script to the root, output the same, did mkdir base (is it DESTDIR?), oh:

+ '[' '!' -d debian -o '!' -d base -o '!' -d testsuite ']'
++ git describe --tags
++ sed -re 's/-[^-]+-[^-]+$//'
+ GIT_TAG=v5.2.3
++ git describe --tags
++ sed -e s/-test/~test/
+ GIT_VERSION=v5.2.3-27-g31a656e
+ '[' 0 -ne 0 ']'
++ git show HEAD:debian/changelog
++ head -1
++ sed 's/.*(\(.*\)).*/\1/'
+ DEB_VERSION=5.2.3-linuxcnc
++ echo v5.2.3-27-g31a656e
++ sed -r 's/-pre/~pre/; s/-/./g'
+ NEW_DEB_VERSION=v5.2.3.27.g31a656e
+ '[' v5.2.3.27.g31a656e = 5.2.3-linuxcnc ']'
+ set -e
++ lsb_release -cs
+ echo 'rtai (v5.2.3.27.g31a656e) stretch; urgency=low'
+ echo
+ git log '--pretty=format:  * %w(72,0,6)%s' v5.2.3..
+ echo
+ echo
++ date -R
+ echo ' -- buildbot <buildbot@example.com>  Mon, 16 Mar 2020 22:52:45 +0100'
+ echo
+ git show HEAD:debian/changelog
+ dch -r --nomultimaint ''
dch: warning:     debian/changelog(l1): version 'v5.2.3.27.g31a656e' is invalid: version number does not start with digit
LINE: rtai (v5.2.3.27.g31a656e) stretch; urgency=low

Line 16 "$ debian/configure 4.14.148 rtai amd64":

++ git describe --tags
++ sed -e y/-/./
+ RTAI_VERSION=v5.2.3.27.g31a656e
+ '[' -z 4.14.148 ']'
+ KERNEL_VERSION=4.14.148
+ shift
+ FEATURESET=rtai
+ shift
+ FLAVORS=amd64
+ KERNEL_HEADERS=
+ KERNEL_IMAGES=
+ for FLAVOR in $FLAVORS
+ '[' -z '' ']'
+ KERNEL_HEADERS=linux-headers-4.14.148-rtai-amd64
+ KERNEL_IMAGES=linux-image-4.14.148-rtai-amd64
+ subst debian/control
+ cp debian/control.in debian/control
cp: cannot stat 'debian/control.in': No such file or directory
+ sed --in-place -e s/@KERNEL_VERSION@/4.14.148/g -e s/@FEATURESET@/rtai/g -e s/@KERNEL_HEADERS@/linux-headers-4.14.148-rtai-amd64/g -e s/@KERNEL_IMAGES@/linux-image-4.14.148-rtai-amd64/g debian/control
sed: can't read debian/control: No such file or directory
+ subst debian/rules
+ cp debian/rules.in debian/rules
cp: cannot stat 'debian/rules.in': No such file or directory
+ sed --in-place -e s/@KERNEL_VERSION@/4.14.148/g -e s/@FEATURESET@/rtai/g -e s/@KERNEL_HEADERS@/linux-headers-4.14.148-rtai-amd64/g -e s/@KERNEL_IMAGES@/linux-image-4.14.148-rtai-amd64/g debian/rules
sed: can't read debian/rules: No such file or directory

Yes, there really is no file rules.

Line 20 "$ dpkg-buildpackage":

dpkg-buildpackage: warning:     debian/changelog(l1): version 'unknown' is invalid: version number does not start with digit
LINE: rtai (unknown) stretch; urgency=low
dpkg-buildpackage: info: source package rtai
dpkg-buildpackage: info: source version unknown
dpkg-buildpackage: error: version number does not start with digit

Is it really only the small letter "v"'s fault? I gave up at this point.

[NTULINUX]

@andypugh Can you offer any input on this?

[andypugh]

I guess I would need to recompile the kernel to test?

[NTULINUX]

I suppose so. One of the lines worth looking into at first glance is this:

'[' '!' -d debian -o '!' -d base -o '!' -d testsuite ']'

There is no more base directory, it was too tedious to toggle through numerous sub-directories all the time (especially base/arch/x86/hal/patches.)

[andypugh]

OK, master or new_libm?

[andypugh]

Here are my ISO making notes: LinuxCNC ISO Build notes.

Custom Pi image info: https://www.raspberrypi.org/forums/viewtopic.php?t=201517

Preempt-RT kernels https://packages.debian.org/search?keywords=linux-image-rt
RTAI Kernels rm-one: https://github.com/andypugh/RTAI/blob/master/README.INSTALL
deb-pkg LOCALVERSION="" to make kernel deb
./debian/configure 4.14.148 rtai amd64
dpkg-buildpackage
LinuxCNC
./debian/configure -r
dpkg-buildpackage

wheezy ISO > VMware LinuxCNC2.7+ISO-maker
sudo lb config -m http://archive.debian.org/debian --parent-mirror-chroot-security http://archive.debian.org/debian-security --parent-mirror-binary http://archive.debian.org/debian
sudo lb build

Puts the .iso file in images{desktop}

www.linuxcnc.org / dists ( login ssh @* )
Debian packaging / archive manual: https://wiki.debian.org/DebianRepository/Setup
infrastructure/update-deb-archive seems to be the update script.

checked build-dependencies (after build-essential and autotools)
pkg-config
libmodbus-dev
libusb-1.0-0-dev
glib
libgtk2.0-dev
yapps2
intltool
tcl8.6-dev
tk8.6-dev
libtk-img
tclx
python-gtk2
python-glade2
python-gtkglext1
libreadline-gplv2-dev
python-tk
libboost-python-dev
libglu1-mesa-dev
libxmu-dev
python-yapps

[NTULINUX]

OK, master or new_libm?

Definitely master as new_libm has been merged into it, and there's more fixes now. Up next is kernel 4.19 support.

[andypugh]

I built and made packages of the kernel and RTAI with the latest RTAI master, compiled LinuxCNC and ran the latency test.
But it seems that the issue here is making RTAI packages?

[NTULINUX]

I built and made packages of the kernel and RTAI with the latest RTAI master, compiled LinuxCNC and ran the latency test.
But it seems that the issue here is making RTAI packages?

If you changed any files in the debian folder, can you make a pull request? If anyone does a git clone of my repository, I want them to be able to make a debian package with no changes needed. If you didn't make any changes, then I don't know what he's doing wrong. I haven't tested any of the code in the debian folder for awhile. Thanks Andy!

[WasIstDasDingDa]

In the meantime I am learning some Debian Packaging. Maybe no magic sauce is needed at all.

[andypugh]

I just made RTAI debs like this...

1. Clean out anything in /usr/src that begins with linux-headers, or you may see "--with-linux-dir=/usr/src/linux-headers-4.14.148-rtai-amd64 /usr/src/linux-headers-4.14.148-rtai-amd64b.deb" (ie a folder and a deb, concatenated)

2. ./autogen.sh

3. cd RTAI

4. ./debian/configure 4.14.148 rtai amd64

5. sudo dpkg-buildpackage
   (I tried without sudo and got "command not found")

And that should build you some debs.

[WasIstDasDingDa]

@andypugh Your recipe is simple and works.
LinuxCNC is up and running now on Buster with max. 18 µs latency at high system load. During the installation some additional dependencies had to be met, almost all automatic.
I like Debian.
Thanks!

[WasIstDasDingDa]

Thank you, I was able to achieve my goal. I hope that your efforts in relation to my feedback are useful for other people too.

[WasIstDasDingDa]

@andypugh I forgot something: On Buster, dpkg-buildpackage can basically be run with user rights. It only required root privilege when it was started to process the dsc file (GPG signature). Most likely, dpkg-buildpackage on Stretch is hidden in a /sbin folder.

[NTULINUX]

Glad everything is good, once I have kernel 4.19 support ready to go, I'll make a new RTAI release.

Thank you both for your help!

[andypugh]

One more point. It's not good to use "sudo" when making the packages, as that leads to many of the intermediate build files belonging to root, which causes problems later.
Fakeroot to build the debs is almost certainly better, if needed.

[WasIstDasDingDa]

@andypugh Good to know, I'll look into that. But so far there have been no problems with the installed software.
By the way, I got annoyed about the result of sudo make a long time ago.

[WasIstDasDingDa]

Oops, at a first glance, the implementation of the fakeroot environment is a job for the developer of the built system. I think RTAI is not affected by possible issues when creating packages with root privileges.

[NTULINUX]

Whether RTAI is affected or not by creating packages as root is irrelevant. It's bad practice, please use fakeroot.

[WasIstDasDingDa]

I've read this very interesting discussion https://unix.stackexchange.com/questions/9714/what-is-the-need-for-fakeroot-command-in-linux and understood.
For practice I will build the RTAI packages again.

[andypugh]

I have found the RTAI kernel reliable during operation, but it sometimes (about 1 time in 200) locks up the PC on exit.
We are trying to get to the bottom of this, but it would be interesting to know if you have the same problem. If you have the LinuxCNC source (including the tests directory) then this script will generally crash the PC in about 5 minutes on my system.

#!/bin/bash
for PASS in $(seq 1 1000); do
    echo starting pass ${PASS}
    runtests ${HOME}/linuxcnc-dev/tests/abs.0
done

[WasIstDasDingDa]

Three times I ran the script, until the crash it took regularly between 3 and 4 minutes.

[andypugh]

Right, so it crashes on your system too.
I have had it do 2 x 1000 cycles in succession, and also have had it fail after 64 iterations.

What is your system, as a data point? Which kernel?

[WasIstDasDingDa]

Buster with RTAI Kernel based on 4.14.174
MB Asrock 960GC-GS FX
Proc Phenom II X4 955 @ const. 4x3200 MHz
Mem 2x2GB DDR3
Graphics Radeon HD 6450 1 GB Mem

[WasIstDasDingDa]

Correction: The RAM is DDR2 (but certainly not relevant)

[NTULINUX]

I've looked into this and I'm getting crashes as well, usually around 40 interations, hacked at Kconfig some more, removed the usermode check, all latency calibration, and it ran until about 300. Delaying a problem is not a fix so I'm not pushing any of those changes to git. Not sure what else there is for me to try without a serial cable to log the stack traces.

[andypugh]

Moved this to new issue #13

[obrire]

error: ../../grub-core/loader/i386/efi/linux.c:267:kernel dosen't support EFI handover.
error: ../../gurb-core/loader/i386/efi/linux.c:94:you need to load the kernel first.

Some people(Ubuntu) claim that this is a bug. I don't know if we can make a corresponding patch for this, which can be applied to Ubuntu 20.10 and CentOS 8, PREEMPT_RT works well on CentOS 8 and Ubuntu 20.10

I use eufi to start Grub2, with 4.19.152 kernel and IPIPE patch