Skip to content

KernelNotes

Jump to bottom
hq6 edited this page Jun 30, 2017 · 14 revisions

Learning about the Linux Kernel

It seems that there is a pm_qos subsystem that can be used to specify latency requirements although it seems to be for drivers within the kernel. Unclear if RAMCloud currently uses it either directly or indirectly.

Building the kernel on rc68

sudo apt-get install libncurses5-dev
sudo apt-get install kernel-package
make menuconfig
make-kpkg clean
fakeroot make-kpkg -j 30 --initrd kernel-image kernel-headers
# The above command built into the real parent directory.
dpkg -i ../linux-image-3.16.39+_3.16.39+-10.00.Custom_amd64.deb

Adding a system call

1. Add a syscall number in `arch/x86/syscalls/syscall_64.tbl`
2. Add declaration of the syscall to the bottom of `include/linux/syscalls.h`
3. Create the syscall inside `kernel/helloworld.c`
4. Add the `helloworld.o` to `obj-y` target in `kernel/Makefile`.
5. Recompile
     fakeroot make-kpkg -j 30 --initrd --append-to-version=-hellosyscall  kernel-image kernel-headers
6. Install
    sudo dpkg -i ../*hellosys*
7. Set as the default kernel with grub.
   Edit `/etc/default/grub`, and use the full name, as below
   GRUB_DEFAULT="Advanced options for Debian GNU/Linux>Debian GNU/Linux, with Linux 3.16.39-hellosyscall+"
   The list of valid strings is in `/boot/grub/grub.cfg`
   Run `sudo update-grub`

Establishing shared-memory communication with UserSpace

1. Read a value from the user's address space using `copy_from_user`.
2. Write a value to the user's address space using `copy_to_user`.
3. Not enough space to install, we can remove non-running kernels by
   looking at output of `dpkg --list | grep linux-image`.

Establishing core isolation

1. Read the list of in-kernel threads that might introduce jitter.
    - There is a reference to eHCA infiniband which looks like an IBM brand
      of infiniband, but we should verify that these per-CPU kernel threads
      are not running.
    - Need to get a deeper understanding of softirqs and see if any must
      run on a particular core.
    - First step may be to ignore in-kernel threads and just focus on
      ensuring no other user threads run.
    - "Of course, you can also run your workload at a real-time priority,
      thus preempting vmstat_update(), but if your workload is CPU-bound,
      this is a bad idea."
2. Decide to ignore the kernel jitter for now and focus on scheduling itself.
3. Read through Documentation/scheduler/.

Random Notes

  1. When using git or even building stuff, rcnfs is a bottleneck. Thus, I have opted to put the kernel sources in /var/local/hq6/, and use RCNFS as a git remote instead. This will work for now, since a git push is much more efficient than most other commands that assume the filesystem is fully local.

  2. The installation command above is successful except that there are errors from the dkms module. Grep for the string /var/lib/dkms/iser/1.7.0/build/make.log in the output. This is not an immediate blocker, but we may need to look into it if we cannot get RAMCloud to run on it eventually.

  3. linux/README --> Documentation/00-INDEX --> Documentation/kernel-per-CPU-kthreads.txt && Documentation/kernel-parameters.txt && Documentation/scheduling/ && Documentation/oops-tracing.txt

    The same README also mentions that gdb can be used on a running kernel.

  4. printk messages may be buffered for a very long time, and running with the same.

Current directory for cpusets

/sys/fs/cgroup/cpuset

Remove a cpuset

rmdir Directory

Assign processes to a cpuset

cd /sys/fs/cgroup/cpuset
mkdir AllOthers
cd AllOthers
echo 0 > cpuset.cpus
echo 0 > cpuset.mems
echo $$ > cgroup.procs

Attempt to assign all processes to a single-cpuset

# The command below will move all processes that can be moved, and return errors for the remainder.
for i in $(cat cgroup.procs ); do echo $i > AllOthers/cgroup.procs ; done
# Below, we count and view the processes that we failed to move and observe that they should all be in-kernel processes.
ps -Ao "%U,%p,%a" | grep -w -f <(cat cgroup.procs | awk '{print $1","}')  | wc -l
ps -Ao "%U,%p,%a" | grep -w -f <(cat cgroup.procs | awk '{print $1","}')  | less

Create a few exclusive cpusets

mkdir Exclusive1
echo 1 > Exclusive1/cpuset.cpus
echo 0 > Exclusive1/cpuset.mems

mkdir Exclusive2
echo 2 > Exclusive2/cpuset.cpus
echo 0 > Exclusive2/cpuset.mems

mkdir Exclusive3
echo 3 > Exclusive3/cpuset.cpus
echo 0 > Exclusive3/cpuset.mems

Forcing cores offline

echo 0 > /sys/devices/system/cpu/cpuN/online

Watch interrupts happen in real-time

watch -n0.1 --no-title cat /proc/interrupts

References

https://www.debian.org/releases/jessie/i386/ch08s06.html.en
http://stackoverflow.com/questions/4943857/linux-kernel-live-debugging-how-its-done-and-what-tools-are-used
https://lwn.net/Articles/384146/
http://www.embeddedlinux.org.cn/EssentialLinuxDeviceDrivers/final/ch21lev1sec9.html
http://askubuntu.com/questions/637647/dkms-install-failed
https://en.wikipedia.org/wiki/User-mode_Linux
https://arvindsraj.wordpress.com/2012/10/05/adding-hello-world-system-call-to-linux/
http://franksthinktank.com/howto/addsyscall/
http://askubuntu.com/questions/148095/how-do-i-set-the-grub-timeout-and-the-grub-default-boot-entry
http://askubuntu.com/questions/216398/set-older-kernel-as-default-grub-entry
http://stackoverflow.com/questions/13538618/printk-is-line-buffered
http://stackoverflow.com/questions/10509850/how-to-access-user-space-memory-from-the-linux-kernel
http://www.ibm.com/developerworks/library/l-kernel-memory-access/
man mlock
http://askubuntu.com/questions/18804/what-do-the-various-dpkg-flags-like-ii-rc-mean
http://unix.stackexchange.com/questions/181569/what-is-the-functionality-of-sched-softirq-in-linux
stackoverflow.com/questions/20887918/why-softirq-is-used-for-highly-threaded-and-high-frequency-uses
https://www.kernel.org/doc/Documentation/cpu-hotplug.txt
http://www.makelinux.net/ldd3/chp-10-sect-4
http://unix.stackexchange.com/questions/5788/how-is-an-interrupt-handled-in-linux
http://www.tldp.org/LDP/tlk/kernel/kernel.html
https://www.freebsd.org/cgi/man.cgi?query=ps&manpath=SuSE+Linux/i386+11.3
http://man7.org/linux/man-pages/man7/cpuset.7.html

https://www.quora.com/How-do-I-implement-different-task-scheduling-algorithms-in-the-Linux-Kernel
http://stackoverflow.com/questions/3086864/how-to-create-a-new-linux-kernel-scheduler
http://unix.stackexchange.com/questions/46077/where-to-download-linux-kernel-source-code-of-a-specific-version
http://serverfault.com/questions/227920/is-it-possible-to-tell-which-cpus-are-hyperthreads-of-the-same-core

http://unix.stackexchange.com/questions/8699/is-there-a-utility-that-interprets-proc-interrupts-data-in-time
Clone this wiki locally