Machinekit Build for Multiple RT Operating Systems
Machinekit Build for Multiple RT Operating Systems
The Rationale for Multiple Realtime Operating Systems Support
A key aspect of LinuxCNC is its realtime capability. Up to about 2011, LinuxCNC essentially relied on a single realtime Linux extension, namely RTAI (http://www.rtai.org). Originally there was a second option, RT-Linux, but that product faded away over time and support for it in LinuxCNC has suffered substantial "bit rot." Realistically, RT-Linux has not been a viable option for LinuxCNC for years.
As the old adage goes, 'don't put all your eggs in one basket'. Reliance on RTAI has become a more pressing issue over time as some question marks have arisen over RTAI’s long term viability and the robustness of its community. Several alternatives - some of them freely available open source - have appeared, namely the RT-PREEMPT Linux realtime patch (https://rt.wiki.kernel.org) and the Xenomai Realtime kernel (http://www.xenomai.org). Regardless, there have been no coordinated attempts to widen the LinuxCNC support base for other realtime operating systems (from here on called \'RTOS').
There have been some isolated efforts to remedy this single-source situation a bit, namely the MiniEMC2 Project by Sergey Kaydalov (Xenomai-based) and work by Michael Büsch to support a RT-PREEMPT patched kernel, as well as several others. Various of them have been based on outdated versions of LinuxCNC or have relied on custom kernels, making them rather less useful.
The original focus of the RTOS work therefore has been to bring together these isolated efforts, bring them up to date, integrate them into the LinuxCNC build process, and provide means to make the process of supporting another future RTOS option easier.
A second aspect has been its reliance on the x86 architecture, largely driven by the fact that RTAI is not available for non-x86 architectures. While mostly written in portable languages, some x86 assumptions and inline assembly code have crept in over time into the LinuxCNC code base, meaning builds for other architectures were challenging.
A third aspect of LinuxCNC has been its packaging. While the current method of distribution and installation continues to work for serious users, its relatively monolithic nature hinders the wider acceptance of LinuxCNC. Getting LinuxCNC into major software repositories has been hampered by the fact that a build based on a non-mainstream kernel extension is difficult to get accepted for inclusion in major distributions like Debian or Fedora. However, since RT-PREEMPT is becoming more mainstream, and a stock RT-PREEMPT kernel is in fact now available through the Debian wheezy package stream, this is becoming an option which wasn’t available before.
Evolution of the work
The original focus of the recent RTOS work in LinuxCNC was to bring together the isolated efforts mentioned above, bring them up to date, integrate them into the LinuxCNC build process, and provide means to make the process of supporting another future RTOS option easier.
The result of this work has been branches which enable the use of RTAI, Xenomai or RT-PREEMPT kernels as a build-time option; that is, one had to build LinuxCNC for a particular kernel, and all the kernel-specific support would be linked into this binary. That binary would run only on the kernel it was built for, but not any others.
While it has resulted in usable and useful code, this approach has several
downsides. First, only an RT-PREEMPT build might make it into Debian;
any package based on RTAI or Xenomai is still unlikely to be
accepted. Second, the per-RTOS build approach increases the
build-distribute-support burden: now there is one package per RTOS. Also, having
only one option, but not others, being available through simple standard
apt-get install linuxcnc is less than helpful.
Viewing more closely, this was essentially a design and build system defect of LinuxCNC, which has used a monolithic build approach. The supporting operating systems have for almost two decades provided a mechanism to separate the upgrade of supporting infrastructure - like runtime libraries - from the upgrade of applications proper, however, LinuxCNC has not made this transition until recently.
Therefore, the driving factor for the 'unified build' work on LinuxCNC has been: separate out the specifics of a particular RTOS from the rest of the software build by packaging the RTOS-specific parts into shared libraries and runtime-loadable modules. There is no particular magic behind this step, it is essentially tracking the status quo how modern software packages are built.
The aim of the 'unified build' work can be described as follows:
make LinuxCNC’s major software components build without regard of any specific realtime operating system by providing an RTOS-independent abstraction layer
package those specifics as separately built loadable (and distributable, hence upgradable modules and libraries), likely as packages
provide hooks in LinuxCNC to determine the underlying RTOS, and load those parts of the RTOS-specific modules as needed.
The upsides of such an approach are manifold:
the main LinuxCNC build, together with its Debian-compatible RTOS support, becomes amenable for major distro inclusion
the build and distribution logistics for the LinuxCNC project are substantially simplified
bugs in the supporting infrastructure can be fixed by upgrading a support package, or maybe just a single file - without wholesale pulling of repositories and constant rebuilds
due to the RTOS abstraction layer, it becomes rather easy to include support for a new RTOS option should it arise.
the drive to remove x86 dependencies has resulted in a code base which not only builds but also routinely runs on other architectures, giving a welcome exposure to remaining architecture dependencies (which have turned out to be rather few)
There are downsides: - the abstraction layer and some of the RTOS support has not seen wide use yet, meaning bugs are likely. - the build process has become substantially more complex, suggesting work will be needed for new platforms.
- support for Xenomai and RT-PREEMPT realtime threads besides RTAI
There should be minimal user configuration changes for using the new RT options.
- kernel autodetection
The 'unified build' branch will detect the RT features of the running kernel and choose an appropriate thread flavor.
- runtime loading of support modules
All thread-specific code has been wrapped into shared objects and libraries which are loaded on demand. This enables fixes, upgrades or tests by just exchanging a file.
- unified thread status reporting
The procfs support (/proc/rtapi etc) is useful for kernel threads only. This has been superseded by status reporting through halcmd which applies to all threads, and includes support for thread-system specific values.
- multiple instance support
RTAPI and HAL have been extended to support several instances running in parallel on a single CPU. Kernel threads instances export a new sysfs entry /proc/rtapi/instance - without that it is rather cumbersome to determine the instance id - something which is relevant for startup and collision detection.
- common shared memory subsystem
The deprecated System V IPC code has been removed and replaced by Posix shared memory. Alternatively, userland flavors may (kernel thread flavors must) use the new common shared memory driver (shmdrv). This removes the need to employ RTOS-specific allocators, hence their restrictions do not apply. The common shared memory driver has the important property that shared memory segments created in a user process can be attached later by kernel modules, which significantly simplifies startup, shutdown and - later - interworking of instances (for instance cross-linking of pins and ringbuffers). See also the section 'Multiple instances and the shmdrv shared memory driver'.
- common startup/shutdown
some effort has been made to create a common flow of startup and shutdown sequence. This work is not complete (see below under 'Further work').
- unified logging
All logging out of RTAPI, RT HAL and user HAL components goes through a single, operating system independent channel which works identically for userland as well as kernel thread flavors.
- separate global log levels for user components and RT
these loglevels can be set at startup, or through halcmd (
log rt <level>and
log user <level>). The user logging level applies to all processes, as it is now a global variable.
- debut of ringbuffer code
This work by Pavel Shramov and myself [MH] will form a key element of the subsequent new middleware infrastructure which will replace NML. At the moment it is used for the unified logging code and works flawless (kudos to Pavel!). See src/rtapi/ring.h.
- no more inline assembly
The last remnants of x86 inline assembly code have been removed and replaced by equivalent gcc/llvm intrinsic operations, meaning the core code should compile on pretty much any modern architecture. (src/rtapi/rtapi_bitops.h)
- HAL segment size configurable
This used to be a compiled-in constant. It is now a startup environment variable (HAL_SIZE).
- exception handling - separating mechanism and policy
The rather ad-hoc reporting of RTAPI exceptions (like realtime delays, traps due to invalid floating point operations etc) have been replaced by a redefinable exception handler which works identically across all flavors. The core RTAPI code supports collecting such exceptions and funneling through this exception handler; however, it is now possible to define - through a normal HAL component - how these exceptions are dealt with (see src/hal/components/rtmon.comp). There is a default exception handler in place which just logs exceptions.
- support for thread-specific RT status collection
Status collection for RT threads is important to track down sources of delays, but it incurs overhead. By making this an optional RTAPI method which can be called by a thread function this can be applied as needed, using a standard mechanism.
./configure && makerun builds many RT options
The Unified Build feature reworks the build system and runtime scripts so that binaries for many RT thread flavors and many kernels may be built in a single run and installed on a single host without conflict. In use, thread flavors applicable to the current running environment are user-selectable with a sensible default. These features simplify distribution and remove barriers for including LinuxCNC packages in the big distributions.
Principles of Operations
The overall structure and cooperation of major components is a bit different from the past modus operandi.
Major data structures
Before the 'unified build' work was undertaken, the RT build (RTAI) used a shared memory segment at the RTAPI layer and a segment at the HAL layer. The 'simulator mode' only used the HAL segment, no RTAPI shared memory segment.
In contrast to the earlier approach, the shared memory segments in use in the 'unified build' branch are:
The Global Data Segment
This is a per-instance shared memory segment which is assumed to exist before any RT operations start (either flavor). It carries parameters which apply globally to the instance (thread flavor, log levels, thread statistics etc). It also carries the ring buffer transporting messages generated by rtapi_print_msg() et al from wherever they were generated into the rtapi_msgd address space, from which the messages are logged to syslog and - optionally - to stderr.
By intent, this segment must work with any thread flavor as-is (i.e. without layout changes). Any structures which are thread flavor specific hence must be represented as union types - see for instance the thread statistics structs (see rtapi_threadstatus_t in src/rtapi/rtapi_global.h and src/rtapi/rtapi_exception.h (admittedly disputable file naming)).
The driving factor for introducing this segment was recognizing the fact that there needs to be a mechanism to coordinate per-instance operations; the options are too diverse and heuristic in detection to relegate decisions of per-instance nature to autodetection mechanisms at lower levels.
The HAL Data Segment
Besides small changes in per-object (thread, component, pin etc) structures there are no major changes except provisions for a configurable segment size, plus data structures and macros/functions to access foreign instance HAL data segments. This is not used extensively in the current branch.
The RTAPI Data Segment
The RTAPI data segment is essentially unchanged in layout respective to previous versions.
One major difference is that the userland thread flavors do not employ a shared memory segment for RTAPI data as it is all local variables in the rtapi_app process. In retrospect this lack of uniformity was a mistake, although not a showstopper.
Relation of the major data structures
The obvious candidates for the global segment is the logging ringbuffer, plus key parameters driving overall instance parameters. As it is known to exist when any RT operations commences, over time some parameters and statistics structures have found a better place here than in one of the other segments.
I [MH] have considered merging the global and RTAPI data segments; however, from a stability perspective it is a good decision to relegate access to RTAPI data to 'need to know' entities.
It would be a worthwhile effort to reintroduce the RTAPI shared memory segment for userland threads flavor; provisions have been made for this at the per-flavor configuration information (follow the logic of FLAVOR_RTAPI_DATA_IN_SHM usage to see how).
Multiple instances and the role of the
shmdrv shared memory driver
Running multiple RTAPI instances side by side will make sense eventually, for instance for multi-spindle setups. However, these instances will need to interact in some way at the HAL level, and that feature (tentatively called 'crosslinking') applies to pins, and messages through ringbuffers, and it is already working in a development branch.
For this to work, the prerequisite is that instances access the HAL data segment of a foreign instance. That in turn suggests that access to shared memory segments must happen in a uniform (i.e. thread- flavor unspecific) way, and regardless whether the instance is a kernel-threads or user-threads flavor. The status ante however was that flavors employed all sorts of different shared memory mechanisms - the RTAI-specific method, the Xenomai-specific method, and the deprecated System V IPC calls for the simulator build. However, for RTAPI purposes there is absolutely no reason to use the flavor-specific shm API’s since there is no shared memory allocation or deallocation in an RT thread; all this happens during module init and exit routines, and therefore in a non-RT context.
The solution looks as follows:
as long as only userland threads instances are used, Posix shared memory does the job.
if userland and kernel threads instances are used, Posix shm - as a user process API - cannot be used, in which all instances use the shared memory driver.
Hence, userland thread flavors use the shmdrv method if the corresponding kernel module is loaded, or Posix shm otherwise; kernel threads instances (RTAI, Xenomai) must use the shmdrv facilities and do so through an in-kernel API (see src/rtapi/shmdrv/shmdrv.h for the kernel and userland API’s; the latter is provided through the routines in src/rtapi/shmdrv/shmdrvapi.c).
Hence, shmdrv does not fit the normal module loading and unloading scheme very well as its lifetime transcends a particular instance using it. Consider the following scenario:
a userland threads and a kernel threads instance are to be run, and have HAL crosslinking capabilities.
first, shmdrv must be loaded
now userland or kernel threads instance can be started and will cooperate fine through the shmdrv API.
when either instance is shut down, the other instance continues to use the shmdrv API - either in-kernel or from userland through mmap().
Hence, neither instance shutdown may unload shmdrv (and it will not succeed anyway). Normally, the method to employ is reference counts, allowing an unload to succeed when the last reference has gone away. However, due to current restrictions of how shm segments are handled there is a chance that orphaned shared memory segments will 'hang around' making problems on instance restart. This is currently being addressed (tracker entry #26).
A key reason why shmdrv was done is the sequencing of operations; some of the flavor-specific shm API’s do not support access of a shm segment by a kernel module which was previously created by a user process. This is a severe restriction not only for instance interoperability, but also for startup and shutdown.
The rtapi_msgd Process
The primary purpose of the rtapi_msgd process is to create, populate and service the per-instance global_data_t shared memory segment. In detail, the jobs are:
determine the thread flavor applicable to this instance, and set variables accordingly
accept per-instance options, like RT and userland message levels, HAL data segment size, HAL stack size etc, use of the shmdrv shared memory driver etc
populate the global segment with these values
poll the message ring buffer for new messages generated by rtapi_print_msg() in other components and log them to syslog (optionally to stderr too).
in case of userland thread flavors, observe the rtapi_app process (see below) and shut down if it goes away.
The rtapi_msgd changes its argv to
msgd:<instance number> once started
successfully to aid duplicate startup attempt detection, and instance
The rtapi_app Process (Userland threads)
This is based largely on the sim_rtapi_app process used in the 'simulator environment' in previous releases. It is present only in userland thread flavors, and is the process context where RT threads run. What is does is:
attach to the global segment prepared by rtapi_msgd, inheriting essential parameters and data structure access
harden memory for RT use (pre-faulting and locking memory etc)
privilege handling - RT process access I/O hardware
load the rtapi.so and hal_lib.so components applicable for the thread flavor
accept commands over a Unix domain socket
The commands accepted are all generated by halcmd (for instance
loadrt compname causing rtapi_app to find and dlopen() the
corresponding shared object, and calling the rtapi_app_init()
functions on load, as well as rtapi_app_exit() on
It is possible to manually call rtapi_app for debugging purposes; see scripts/realtime and the halcmd code in hal/utils how to do that.
The rtapi_app program changes its argv to
rtapi:<instance number> once started
With RTAI and Xenomai-kernel flavors, there is no corresponding rtapi_app process since HAL modules are just kernel modules. There is no conceptual change here - modules are inserted by the setuid module_helper.
Tested Operating Systems
- rtai 2.6.32-122-rtai
as used in the 10.04LTS live CD
- rtai 3.5.7
Schooner/Arceye/Mick Private Bin kernel - reported to work, Axis screenshot seen. See the LinuxCNC Forum (http://forum.linuxcnc.org) and the emc-developer email list for ongoing discussions.
- xenomai 3.5.7-220.127.116.11 i686 and x86_64
John’s [JM] Xenomai kernel, see http://wiki.linuxcnc.org/cgi-bin/wiki.pl?XenomaiKernelPackages
as per wheezy distro (x86_64)
xenomai 18.104.22.168 for beaglebone running wheezy
No attempt has been made, and none will be made, for the hardy RTAI kernel.
Ubuntu Lucid, Ubuntu Precise, Debian Wheezy
The OS architecture (i386/i686 versus x86_64) must be identical in the build and run environments - building LinuxCNC on say an i386 kernel and trying to run the result when booting an x86_64 kernel will not work.
Universal Build Changes
The Universal Build supports building for all RT environments in a
./configure && make run, and supports simultaneous
installation and execution of all RT environments on a single host.
The run-time setup and teardown processes needed new mechanisms for retrieving flavor configuration and for locating separate run-time binaries with separate paths when setting up or tearing down the various RT environments.
In turn, the build system needed new mechanisms for configuring and building for each target RT flavor, keeping all build objects separate to maintain dependency integrity.
The changes to the run-time and build systems to effect these requirements are described here.
Run-Time System Changes
With the Universal Build, support for many RT environments may be simultaneously installed on a host system. Each of the five supported RT flavors has its own separate configuration and its own set of RTAPI and support binaries that may not be mixed with other RT flavors. With support for multiple RT flavors installed on a single host, for example, it is possible for an operator to boot a Xenomai kernel and run LinuxCNC in any of the POSIX, Xenomai userland or Xenomai kernel RT thread flavors (chosen at run-time). She may then shut down and boot an RT_PREEMPT kernel, and then run LinuxCNC in either of the POSIX or RT_PREEMPT RT thread flavors.
To make this possible requires compiling and installing binaries separately when those depend upon flavor, enabling configuration to be separately specified for each flavor, and refactoring run-time initialization code to enable selecting a flavor and loading matching configuration and binaries.
Flavor configuration is separated by replacing the old
script with an INI-style
per-flavor configuration sections.
Per-flavor binary files are kept separate by adding flavor and kernel
version to ensure non-conflicting file paths. RTAPI modules are built
rtlib/<flavor> for userland styles and
rtlib/<flavor>/<kernel-version> for kernel styles. Userland style
modules have a matching
bin/rtapi_app_<flavor> executable. The
ULAPI module is built into
Run-time environment initialization starts in the
It obtains run-time parameters from
flavor executable output and the
rtapi.ini configuration file. It starts
performing flavor-specific initialization, described next.
For kernel threads systems, the script runs
libexec/linuxcnc_module_helper to load each kernel module listed in
linuxcnc_module_helper looks for the named kernel
rtlib/<flavor>/<kernel-version>, and optionally in the
RTDIR parameter from
rtapi.ini (needed for RTAI), and loads the
For userland threads, the
realtime script start the
executable defined in
default. During the build, the linker sets an
rpath pointing to the
rtlib/<flavor> for run-in-place builds, so that
dlopen() the module with no need to read module path
location from the configuration file.
On the ULAPI side,
liblinuxcnchal.so again is given an
lib directory so that
ulapi-<flavor>.so may be loaded without
reading external configuration.
At this point, the realtime environment setup is complete. Taking
down the environment is simple: for userland threads,
shut down; for all threads,
rtapi_msgd is shut down, any kernel
modules are unloaded, and if needed, shmdrv is unloaded.
Build System Changes
In order to build multiple RT thread systems in a single run, both
build parameters and intermediate build objects for each flavor must
be kept separate, requiring extensive changes to
src/Makefile, and several other files.
Most of the Autoconf configuration was refactored. A new section detects each of the supported RT thread flavors. Another new section automatically detects kernel sources, classifying them into lists based on RT capabilities.
RT thread flavor parameters must be passed from the configure script
Makefile.inc separately. For example, the value of
is different for Xenomai user and RTAI kernel threads, and so
RTAI_KERNEL_THREADS_RTFLAGS are passed
separately. During the thread-specific
make modules run, a
THREADS variable is set so that something like
$($(THREADS)_THREADS_RTFLAGS) does the right thing.
The list of all detected thread flavors to be built is in
Makefile.inc in the
BUILD_THREAD_FLAVORS variable. For kernel
thread flavors, the kernel source directories are listed by flavor in
make starts a top-level build that looks much the same for
the parts of LinuxCNC not affected by the RT flavor. The top-level
modules: target, however, does not itself build any
flavor-specific objects. Instead, it executes second-level
modules runs, one run for each configured userland RT thread flavor
and one more run for each unique combination of kernel thread flavor
and kernel source directory.
make modules runs build the RTAPI binaries and
matching ULAPI module, keeping both build results and intermediate
build objects separate for each flavor. The three categories of
userland RTAPI, kbuild RTAPI and ULAPI objects each had special
considerations to enable separate builds.
Userland RTAPI sources simply build into the RT flavor-specific
objects, such as `objects/xenomai'.
Linux kbuild provides no simple way to specify a location for
intermediate build objects. For kernel thread flavors,
target works around this limitation by creating a tree of hard links
to the original sources under
Then kbuild is run with that as the top-level modules directory. This
works fine most of the time, except during development when a new file
is added to the original source tree, it is not automatically hard
linked into the object tree.
The ULAPI sources in the
rtapi/ directory must also be built
separately for each flavor. Limitations in the
TOOBJS requires source file paths not to overlap in order that
object file paths also do not overlap. This was overcome by creating
one link in
rtapi/ to the current directory for each RT flavor so
rtapi/rtapi_task.c can instead be compiled from
rtapi/posix/rtapi_task.c with the result going into
Install the following packages:
$ sudo apt-get install libudev-dev libmodbus-dev libboost-python-dev libusb-1.0-0-dev
If you want to build the emcweb Web UI (--enable-emcweb), you also need these:
$ sudo apt-get install libboost-serialization-dev libboost-thread-dev
Configuring and Building: The Basic procedure
In case you have an existing
linuxcnc directory and want to add this
branch, run this:
$ cd linuxcnc $ git remote add machinekit https://github.com/machinekit/machinekit.git $ git fetch machinekit $ git checkout -b machinekit-master machinekit/master
To clone a new copy:
$ git clone --origin machinekit https://github.com/machinekit/machinekit.git [<directory>]
The simplest way to compile this package is:
srcdirectory under the directory containing the package’s source code.
./autogen.shto regenerate files necessary for the following steps.
./configureto configure the package for your system. If you’re using
cshon an old version of System V, you might need to type
sh ./configureinstead to prevent
cshfrom trying to execute
configuretakes a while. While running, it prints some messages telling which features it is checking for. See the section 'The Configure script' below for specific options.
maketo compile the package.
sudo make setuidto set permissions. If you get any warning during this step, see the next two sections and resolve those issues. There is no point in running linuxcnc without having these warnings resolved.
source scripts/rip-environmentto set up the environment.
linuxcncto test the software.
Preparing Linux Logging
All LinuxCNC-related log messages go through rtapi_msgd, which logs them to the syslog 'LOCAL1' facility. This includes messages generated by kernel RT components; it does not include any messages which are generated by various supporting components which use 'printk' (I think I caught most of these though; please report if you discover such a case).
sudo make setuid step will check if logging is properly configured; if
not, you will get a message like this:
Warning: Logfile '/var/log/linuxcnc.log' does not exist. Hint: Be sure the (possibly empty) logfile exists; and restart rsyslogd; rsyslogd will not create missing files.
Warning: No rate limit in rsyslogd is set. The rsyslogd drops logs when incoming at higher than the rate configured by 'SystemLogRateLimitBurst'. The default rate is lower than LinuxCNC requires when running in debug mode. Hint: Put src/rtapi/rsyslogd-linuxcnc.conf into /etc/rsyslog.d/linuxcnc.conf for a reasonable default, and restart rsyslogd. /etc/rsyslog.d/linuxcnc.conf does not exist - consider running 'sudo make log'
In this case, just run:
$ sudo cp rtapi/rsyslogd-linuxcnc.conf /etc/rsyslog.d/linuxcnc.conf
Make sure the logfile exists and is readable (the rsyslog documentation says a restart will create non-existent logfiles - does not work for me):
$ sudo touch /var/log/linuxcnc.log $ sudo chmod 644 /var/log/linuxcnc.log
and tell rsyslogd to reload the configuration:
$ sudo service rsyslog restart
Then you can watch the logfile like so:
$ tail -f /var/log/linuxcnc.log
Verify this works by executing
$ logger -p local1.notice testing logging
This text should show up in /var/log/linuxcnc.log.
Resolving other warnings during 'sudo make setuid'
You might see the following warnings during 'sudo make setuid':
Warning: /etc/security/limits.d/linuxcnc.conf does not exist, and a reasonable 'memlock' value not found in configuration. Please check the system configuration and correct. Hint: src/rtapi/shmdrv/limits.d-linuxcnc.conf may be a reasonable example to install in /etc/security/limits.d/linuxcnc.conf.
Resolve this by:
$ sudo cp rtapi/shmdrv/limits.d-linuxcnc.conf /etc/security/limits.d/linuxcnc.conf
Warning: No udev configuration for shmdrv was found. The user running LinuxCNC must have write access to /dev/shmdrv when running kernel threads. This may be configured in /etc/udev/rules.d. Hint: see src/rtapi/shmdrv/shmdrv.rules for a reasonable default.
Resolve this by:
$ sudo cp rtapi/shmdrv/shmdrv.rules /etc/udev/rules.d/50-LINUXCNC-shmdrv.rules
You should reboot after these steps.
The Configure script
configure autoconf script attempts to guess correct values for
various system-dependent variables used during compilation, and places
those values in several files, such as
It also creates a shell script
config.status that can be run in the
future to recreate the current configuration, a file
that saves the results of its tests to speed up reconfiguring, and a
config.log containing compiler output (useful mainly for
Real-time Thread Support: the "Flavors"
To run a particular flavor, two conditions must be satisfied:
LinuxCNC must have been built to support this flavor
the running kernel must be compatible with the desired flavor.
The following thread flavor names are understood (
FLAVOR environment variable):
the traditional RTAI threading system, compiled as .ko kernel modules. Compatible with RTAI kernels only.
Normal Posix threads, runs on any Linux kernel. No realtime properties. This is what used to be 'sim` or 'simulator mode`. Runs on any Linux kernel.
RT-hardened Posix threads running on a kernel with the RT-PREEMPT patch applied (see https://rt.wiki.kernel.org/index.php/Main_Page) Compatible with RT-PREEMPT kernels, but will also run on Xenomai kernels (the results of doing so have not been evaluated)
Xenomai user process RT threads. Requires a Xenomai-patched Linux kernel (see www.xenomai.org). Runs on Xenomai kernels only.
Xenomai kernel RT threads, also using kernel modules. Runs on Xenomai kernels only. While build support is in place, this is deprecated and not recommended for use.
Each of the RT thread flavors requires special kernel support. Xenomai and RTAI kernel packages are available from the project, and RT_PREEMPT kernel packages are available from upstream vendors and third-party package repositories. Please install one of these RT kernels (refer to the documentation of the project on how to do that).
If multiple RT flavors are available, LinuxCNC will attempt to detect and build for all of them. A subset may be selected on the configure command line:
./configure --with-xenomai --with-posix
Build only Xenomai and POSIX userland threads. No other flavors will be built.
./configure --with-posix --with-rtai-kernel
Build only POSIX userland and RTAI kernel threads. If more than one set of RTAI kernel headers is found, modules will be built for all of them.
Build all detected RT thread flavors. In addition to standard locations for kernel sources, also look for Xenomai headers in a non-standard location.
Specify a location for system installation. By default, LinuxCNC will build to "run in place" out of the build directory.
Enable generating documentation from source. Building documentation is disabled by default because of the long compilation time.
Run `./configure --help' for more details on these and other available options.
configure recognizes the following options to control how it
Use and save the results of the tests in FILE instead of
./config.cache. Set FILE to
/dev/nullto disable caching, for debugging
Print a summary of the options to
configure, and exit.
Do not print messages saying which checks are being made. To suppress all normal output, redirect it to
/dev/null(any error messages will still be shown).
Print the version of Autoconf used to generate the
configurescript, and exit.
Options to the realtime script
To start the realtime environment, do as usual:
$ realtime start
To stop, execute
$ realtime stop
The realtime script reads default values from etc/linuxcnc/rtapi.ini; most values here will never need to be changed.
The following defaults from rtapi.ini can be overridden via environment variables:
set the rt and user logging level (0..5, the maximum). A lot of detail will be logged to /var/log/linuxcnc.log. If you suspect problems, run
DEBUG=5 realtime start.
FLAVOR=<flavor name> <linuxcnc command>
Start a particular (non-default) thread flavor. FLAVOR must be one of:
The default size of the HAL data shared memory segment is 262000. A larger size can be set via this variable.
MSGD_OPTS=<options to rtapi_msgd>
extra startup options can be passed to rtapi_msgd. A useful one is
--stderrwhich causes rtapi_msgd to write all log output to stderr as well:
RTAPI_APP_OPTS=<options to rtapi_app>
extra startup options can be passed to rtapi_app. The only meaningful option here is
--driverswhich enables I/O for the
posixflavor. This requires the
sudo make setuidstep.
Meaningful only for userland thread flavors. Forces the use of the common shared memory driver even for userland threads instances (normally it would default to Posix shared memory). This is relevant only in the future scenario where interworking between kernel and user threads instances is desired, so ignore for now.
Instances are numbered 0-31. By default the instance number is 0; another instance can be referred to by the INSTANCE environment variable. See the section 'Running instances side by side'.
Startup Option Usage Examples
Run a 'sim' (Posix threads) instance
$ export FLAVOR=posix $ realtime start $ haldcmd -f -k
Capturing the complete log of a single session
DEBUG=5 MSGD_OPTS="--stderr" realtime start >logfile 2>&1
Running realtime with a larger HAL segment
HAL_SIZE=512000 realtime start
Running the 'Posix' Flavor and enable I/O through drivers
RTAPI_APP_OPTS="--drivers" FLAVOR=posix realtime start
Running instances side by side:
Status of Multiple Instance support
The status of instance support for running several side-by-side instances of linuxcnc on a single host is:
support in RTAPI/HAL as well as startup/shutdown is feature complete
support for multiple instances in NML is currently at a 'gross hack' level - the issue is the TCP port number usage. It might not make sense to fix this as NML is being replaced anyway.
the linuxcnc script needs work - the first instance to shut down kills the other instances too.
Running separate HAL/RTAPI instances
- INSTANCE=2 realtime start
starts the instance #2 of RTAPI/HAL
- INSTANCE=3 linuxcnc
starts the instance #3 of RTAPI/HAL and Linuxcnc (see restrictions noted above)
Isolating and Reporting an error
After building in the
srcdirectory as outlined above, execute as usual
Make sure logging is set up as outlined in the 'Preparing Linux Logging' section above.
watch the file /var/log/linuxcnc.log, for instance with
tail -f /var/log/linuxcnc.login a separate terminal window.
Verify that logging works - do a
realtime startfollowed by a
realtime stop. There should be a few lines of log entries added.
First, verify basic health of the build: Please run the
runtestsscript and save the list of failed tests if any. This can take a long time, it’s more than 120 tests by now.
runtestsstep, log file entries should appear in /var/log/linuxcnc.log.
run the failed configuration with increased logging detail in a terminal window like so:
DEBUG=5 linuxcnc <yourconfig.ini>and save the output to a file; running the configuration from the linuxcnc config selector will make you miss likely important output.
pastebin the list of failed tests, the console output, /var/log/linuxcnc.log and the configuration files if not using a stock configuration.
if the error is verified to be genuine please add an issue to the tracker: https://github.com/zultron/linuxcnc/issues?&state=open.
Man pages for exception handler, update_stats
Building the Xenomai kernel for the BeagleBone board
run configure like so:
$ ./configure --with-platform=beaglebone --with-xenomai --with-posix
This will build both the realtime and 'simulator' (Posix) flavor.
Building the Xenomai kernel for the Raspberry Pi
run configure like so:
$ ./configure --with-platform=raspberry --with-xenomai --with-posix
This will build both the realtime and 'simulator' (Posix) flavor.
Current runtests failures
hm2-idrom fails on the Beaglebone (naturally - no PCI support; this can be ignored).
The issue tracker for the Machinekit is here:
Feel free to add issues so they are not lost.
contains some undocumented new methods. They do not impact HAL functionality.
- rtapi_msgd naming
this name rtapi_msgd is a bit misleading - it sets up the per-instance global data segment which is essential for LinuxCNC operation.
- history cleaning, and squashing out 'wip' and 'FIXME' commits
These are features which can be added as the branch matures:
- RTAPI shutdown exception
The exception handler feature currently has no way to signal an impending RTAPI shutdown, which would be very valuable to for instance cause an estop first thing. Again, this would be easier to do if we had a proper RT demon.
- RTAPI status reporting ala /proc/rtapi
The is currently no equivalent for userland threads flavors; it should be straighforward to add along the lines of thread status reporting.
The rtapi_set_logtag("string") was intended to mark a log message with the origin (user process, RT, kernel etc). It is a bit halfbaked idea; a better solution would be to extend the first argument (message level) to rtapi_print_msg() to support an origin enumeration type (note message level only needs 3 bits of the 32bit integer parameter, so there are lots of bit left to tag the origin and the change is backwards-compatible). This would make writing log messages much more uniform and less verbose, while supporting automatic filtering by origin for multiple publish channels in a future version.
Longer Term plans
Unified command API to the RT environment
I [MH] think once we have the new middleware infrastructure in place it makes sense to fold the kernel threads startup/shutdown/module loading functions into a common RTAPI demon, which would handle all RT commands alike regardless of kernel-versus-userland threads. This would make it much easier on the using side to script commands for startup, shutdown and loading.
That really makes sense only once we have the new middleware stack (zmq/protobuf) in place - the RT environment should be addressable over this vehicle like any other entity, not with arcane shell scripts run from here and there. It makes no sense anymore to do that in NML.
Currently rtapi_msgd is a standalone process and it will evolve to support a publish functionality; arbitrary clients may subscribe to one of the channels to receive updates. This might well be folded into the common RTAPI demon, taking out some complexity of startup and shutdown.
Unified thread creation API
The current method of creating an RT thread for kernel thread flavors stands improvement. A common RTAPI demon could do this for userland and kernel thread flavors just alike, using a simple procfs interface for thread creation/deletion like shown here: http://tinyurl.com/mowmmyl
Use the Xenomai posix threads skin
The Xenomai code currently uses the 'native skin'. Using the 'Posix skin' instead would allow merging all of Xenomai, Posix and RT-PREEMPT into a single code base, easing maintenence a bit. Not very important.
Thread status display in halcmd
After RT threads are started. the
show thread <threadname> command
will display details like so:
$ halcmd -f -k halcmd: loadrt threads halcmd: show thread thread1 Realtime Threads (flavor: xenomai) : Period FP Name ( Time, Max-Time ) 1000000 YES thread1 ( 0, 0 ) Lowlevel thread statistics for 'thread1': updates=455 api_err=0 other_err=0 wait_errors=454 overruns=2598 modeswitches=0 contextswitches=734 pagefaults=0 exectime=158813uS status=0x300180
The values are as returned by the underlying system calls and might need code and manual reading to understand exactly. Some of the values (in particular execution times) seem not to make much sense.
Displaying Thread Status on RT-PREEMPT
The RT threads are named like in HAL (but with the instance number suffixed). Example for ps output of instance 0 on RT-PREEMPT:
mah@wheezy:~$ ps -Leo pid,tid,class,rtprio,stat,comm,wchan |grep `pidof rtapi:0` 4880 4880 TS - SLsl rtapi:0 ? 4880 4883 FF 98 RLsl fast:0 ? 4880 4884 FF 97 SLsl slow:0 ?