Skip to content

LinuxFoundation

Jump to bottom
Dej edited this page May 10, 2020 · 38 revisions
  1. _The Boot Process

Power On -> BIOS SW(stored on a ROM chip on the motherboard) initializes the HW -> BootLoader(stored on the first sector of the Hard Disk aka MBR or in the EFI/UEFI partition) which loads the kernel image and initial RAM disk into memory.

  • The boot loader loads the Kernel and an initial RAM-based filesystem (initramfs)
  • the kernel run /sbin/init and becomes the initial process
  1. Linux Filesystem Tree Layout

  • one large logical filesystem which can contain one or many distinct filesystems mounted at various points

  • Filesystem Hierarchy Standard : shareable (can shared between hosts)

shareable unshareable
static /usr /opt /etc /boot
----------- ------------------ --------------------
variable /var/mail /var/run /var/lock
  • Main directories:
    • /bin : contains binary executable files (cat, kill, ps) . Command binaries which are deemed non essential enough are in /usr/bin.
    • /lib: contains libraries needed to execute the binaries in /bin or /sbin . Also kernel module are in /lib/modules/
    • /boot: essential files for booting the system (vmlinuz compressed linux kernel and initrd initial RAM filesystem which is mounted before the real root filesystem becomes available, config to configure kernel compilation)
    • /dev: device nodes aka device files (byte-stream or block I/O devices). Network devices (eth1 , eth2) do not have device nodes
    • /usr: can be thought of as a secondary hierarchy, need not reside in the same partition as the root directory
    • /etc: contains machine-local configuration files/scripts (/etc/systemd contains config scripts for starting, stopping system services using systemd ; also /etc/init.d which contains scripts for System V initialization)
    • /var: logs in /var/log and cron jobs, mail file in /var/spool, lock files in /var/lock or cat /var/log/yum.log
    • /proc: is the mount point for a pseudo-filesystem, where all information resides only in memory, not on disk. The entires in /proc are called virtual files with zero bytes in size.
# List files opened by a process
lsof -p PID 
# equivalent more or less with 
ls -l /proc/PID/fd
  1. Processes
  • Process = executing program and associated resources (open files, signal handlers) and has various states: running, sleeping
  • Every process has a PID , PPID(parent PID), pgid(process group ID)
  • init usually is the first process run on a system, and is the ancestor of all subsequent processes running on the system (except for thos with [] arounf their name , they are initiated by the kernel)
  • orphaned processes (parent process dies before child) are adopted by init (thus the ppgid is set to 1) or (in distributions which use systemd) the ppgid is set to 2 (adopted by kernel thread known as kthreadd)
  • zombie process (defunct) is a process which terminates before his parent and released almost all the resources and remained only to convey (communicate) his exit status
  • processes are controlled by scheduling
  • process context = snapshot of process by trapping the state of the CPU registers
  • process permissions: programs marked with s (execute bit on) aka setuid programs run with the user-id of the user who owns the program versus program non-setuid which run with the permissions of the user who starts the program. (setuid program owned by root can be a security issue)
  • when process is launched: it runs with the effective user-id and group-id of the user who started it, and with the corresponding privileges. This behaviour can be modified by using special permissions
#setuid programs (e.g.: owned by root)
/etc/passwd
/etc/shadow

# setuid bit is represented by an s in place of x
ls -l /bin/passwd
-rwsr-xr-x. 1 root root 27768 Feb 11  2017 /bin/passwd

  • The setuid bit has no effect on directories. setuid or setgid bits are set, but the executable bit is not

  • When a process is started it has its own isolated space and it uses system calls to indirectly access the HW (the HW is managed by the kernel)

# ulimit command that displays/resets a number of resource limits associated with processes running under a shell
ulimit -a


# increase no of file descriptors (soft resources)
ulimit -n 1600

# no of leak-ed file descriptors supported by OS
ulimit -n

  • process states : running, sleeping(waiting), stopped(suspended Ctrl+Z sends SIGSTOP), zombie

  • fork - original parent process keeps running while the child process starts

  • exec - original parent process terminates and child parent inherits the process ID of the parent

  • when the user types a command in the shell a new process is created (using fork from the user's login shell then the command is loaded onto child process space via exec system call)

# list all process on the system PID PPID PRI(priority) NI(nice value)
ps -elf
  • nice -n 5 command [args] = niceness value can range from -20 (the highest priority) to +19 (the lowest priority) aka set priorities
# start bash with nice value of 10
nice -n 10 bash

# change the value of bash session to 15
renice -n 15 -p (ppid)
  1. Signals (Inter_Process Communication)
  • Are used to emit notifications for processes to take action in response to unpredictable events
  • kill -l list signals , basic syntax kill <SIGNAL> <PID>
  • pkill -u libby foobar kills process foobar of libby user
  1. Process monitoring
  • process monitoring tools: top-process activity dynamically updated , ps-detailed information about processes , uptime-how long the system is running and average load, mpstat-multiprocessor usage, iostat-CPU utilisation and I/O statistics, sar-display and collect info about system activity, strace-information about sys calls a process makes
  • ps is the workhorse for displaying characteristics and it reads information from /proc
  • ps possibilities
    1. UNIX-style: options which must be preceded by -
    2. BSD-style: options, which must not be preceded by -
    3. GNU-style long option, each of which must be preceded by -- 
###  BSD-style: display all processes owned by you and Lift the BSD-style "must have a tty" restriction
## USER  PID %CPU %MEM    VSZ   RSS TTY  STAT START   TIME COMMAND
ps aux 
ps auxf # will show the ancestry tree

# specified user-defined format for the ps output: [parent PID, PID, percentage memory usage
ps -o ppid,pid,etime,command,pmem,pcpu
ps e -o ppid,pid,etime,command,pmem,pcpu

### UNIX-style: display all processes (background ones to) using BSD long format with full format listing
## UID  PID  PPID  F %CPU PRI NI   SZ    RSS WCHAN     S     ADDR TTY    TIME CMD     STIME
ps -elf


# add columns besides the default values (PPID is shown in the default ps -elf)
ps -ef -O etime,pmem,command


# visual description of the process ancestry
pstree -aAp <PID>
  • top = interactive use 1 (each CPPU shown separately) , i (only interactive processes are shown), l (show load)
  1. Package Management Systems

  • rpm = Redhat Package Manager (Centos, Fedora) + SUSE (OpenSuse)

  • dpkg = Debian Package Manager (Ubuntu, Debian, Mint)

  • https://www.digitalocean.com/community/tutorials/package-management-basics-apt-yum-dnf-pkg.

  • package types: Binary, Source (rrpmbuild --rebuild -rb p7zip-16.02-16.el8.src.rpm)

  • low-level utilities (rpm, dpkg) and high-level utilities (yum,dnf,apt,apt-get)

  • EPEL (Extra Packages for Enterprise Linux) - external package repository

    4.1 RPM (RedHat Package Manager)

    • rpm in itself doesn't retrieve packages over the network and installs only from the local machine
    • types of packages: binary (<name>-<version>-<release>.<distro>.<architecture>.rpm) and source (<name>-<version>-<release>.<distro>.src.rpm) also packages with noarch.rpm extension don't depend on CPU architecture
    • /var/lib/rpm = default directory which holds the RPM database (in form of Berkeley DB Hash Files).
# error: rpmdb: damaged header #929 retrieved -- skipping.  
rm -f /var/lib/rpm/__db* # remove lock files
rpm --rebuilddb

# alternative db directory
rpm --dbpath

# repair/rebuild DB
rpm --rebuilddb

# list installed packages 
# yum list installed
rpm -qa

# check file to which package it's belonging to
rpm -qf </path/to/file>

# information about the package (e.g if it is Relocatable)
rpm -qip <package.rpm> 

# list all files from the package aka $ls -lF $(rpm -ql </path/package>)
rpm -qil <package.rpm>

# verify if the files from the package are consistent with RPM database
rpm -Va </path/to/package> # no output means the package is ok

# install pacakage
rpm -ivh <package.rpm>

# upgrade package (also remove the old package)
rpm -Uvh <package.rpm>

# freshening packages in current dir (when download patches and what to upgrade packages already installed)
rpm -Fvh *.rpm

# uninstall a package using --test flag before
rpm -e --test <package> (not path to package)

4.2 YUM (high-level package manager):

  • resolves dependencies automatically it also caches information to speed up performance
  • repos configuration: /etc/yum.repos.d
  • yum configuration: /etc/yum.conf
  • OS patches and update: /var/log/yum.log
  • toggle a particular repo in /etc/yum.repos.d change value from enabled to 0 or 1
  • dnf = next generation replacement for yum
# clear repo cache
yum clean all

# search for package
yum search <package>
yum list

# install package e.g yum install ngnix
yum install <package> 

# install rpm package
yum localinstall <package.rpm>

# check if updates are needed on local system
yum update
yum list updates
yum check-updates

# update package
yum update <package>

# list installed / available packages
yum list installed [installed | updates | available ]

# info about package $rpm -qip package
yum info <package>


# check file to which package it's belonging to $rpm -fq file
yum provides </path/to/file>
yum provides "/logrotate.conf"

# all packages with bash reference/installed and available/package info/dependencies
yum search bash
yum list bash
yum info bash
yum deplist bash
  • adding a new yum repo: vim /etc/yum.repos.d/webmin.repo 

[Webmin]
name=Webmin Distribution Neutral
baseurl=http://download.webmin.com/download/yum
mirrorlist=http://download.webmin.com/download/yum/
mirrorlist
enabled=1
gpgcheck=0
  1. I/O monitoring and I/O scheduling
  • system I/O bound when the CPU is in IDLE waiting for I/O to complete or the network is waiting to clear buffers
  • Network, RAM, CPU can wait for the I/O to complete
  • iostat = monitoring I/O devices activity iostat -k [KB] iostat -m [MB] iostat -xk [extended] (utilisation % close to 100 => system I/O bound)
  • iotop = display table of current I/O usage dynamically
  • ionice = set I/O scheduling and priority for a given process ionice [-c class] [-n priority] [-p pid] [command [args]]
  • I/O raporting bonnie and fs_mark
  • system performance depends on optimising I/O scheduling strategy
  • block layer [I/O scheduler - interface] low-level physical devices. I/O scheduler prioritises requests from Virtual Memory and Virtual File System to block-devices

Clone this wiki locally