-
Notifications
You must be signed in to change notification settings - Fork 0
/
linux.txt
3870 lines (3310 loc) · 187 KB
/
linux.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
## Apropos:
Content is versioned in git. commits, issues and pull-requests welcome!
@[https://www.github.com/earizon/DevOps]
LINUX ADMINISTRATION RECIPES
## External Links: [[{]]
- @[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/]
- @[http://www.linuxfromscratch.org/]
- @[https://linux.die.net/man/]
- @[https://linux.die.net/Linux-CLI/]
- @[https://en.wikipedia.org/wiki/Linux_Standard_Base]
- @[https://docs.fedoraproject.org/en-US/fedora/f34/system-administrators-guide/]
- @[https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/]
[[}]]
# Linux Basics [[{linux.101]]
- Linux itself is just an OS kernel in charge of sharing the LIMITED hardware
resources amongst potentially many running tasks and users working simultaneously
on the systems. More preciselly, the main tasks in charge of kernel control are:
- Orchestate(schedule) how much time each running-task is allowed to
run on each CPU before such task is put on stand-by to let
another task proffit from such CPU.
- Assign which process has access to the (limited)
RAM memory on the system and move data on RAM used by stand-by processes
to secondory storage (disks) in case of RAM space shortage.
- Provide support for users and permissions, so that different users will
be able to isolate and protect its data from other users.
Kernel control is transparent to running tasks or processes, so user-space task
will run with the perception that they run alone in its own CPU and with
all available memory for themself. When the kernel puts them on-hold
such tasks will be freezed and once restarted it will NOT notice any
change to the state before being put on-hold. Only the kernel is aware of
the complex trickeries needed to make tasks run in parallel and isolated
from each other.
Other important tasks offered by the kernel are:
- Abstract the running hardware into standarized interfaces, so
user application will not need to work differently with different hardware.
For example an application will ask the kernel to write data to the disk and
the kernel will take care of the internal difference between the miriads
of diferent disk hardware technologies.
- Provide an easy-to-use file-system to work with data on disk, so that apps
can orginized data in terms of files and directories, instead of just
bunch-of-bytes on the hard-disk.
- Provide network communication and support for standard network protocols
like TCP/UP, bluetooth, WiFI, ... so that each app does not need to reimplement
them.
- Provide mechanisms to allow two running tasks to communicate with each other
at will.
*Kernel mode vs User mode*
- When the CPU is executing kernel code it's running with elevated privileges.
The software has full control of the hardware and can do "anything" on
the system and access all RAM, disk, network resources at will.
- Standard applications will run in user-mode and will have limited access
to only the RAM memory assigned by the kernel. They will not be able to
inspect the memory of other processes running on the system. In fact they
are not aware of such memory due to the many trickeries done by the kernel
to isolate each task.
*Files, files and more files*
- Any running process needs some incomming data to work with and
produced new data that must be stored somewhere.
This data can be provided by some storage system (hard-disk,
usb, tape, ...), arrive (continuosly) from the network,
or be generated by another concurrent process.
Linux (UNIX actually) treats all input data sources and
output data sinks as *"file devices"*.
Internally there can be many differences (block devices with
random access vs char devices with just sequential access), but
running processes mostly always use the file methaphor to access
all of them.
Any running process will have 3 devices available "for free":
- STDIN : The standard input file.
- STDOUT: The standard output file
- STDERR: The standard error file
The standard shell provides many utilities to juggle with those
three standard files. In particular it allows to easely forward
the STDOUT output from a running-process to the STDIN input of
another running process using the "|" pipe syntax:
$ command1 *|*command2 # ← Send STDOUT output from command1 to
STDIN input of command2
STDOUT and STDERR by default are assigned to the running-process
associated terminal (the console where the user has been logged).
The shell allows also to redirect STDOUT/STDERR to any other
file in our file system. Ex:
$ command1 1>output.log 2>error.log
^^^^^^^^^^^^ ^^^^^^^^^^^
redirects STDOUT(1) redirects STDERR
to output.log to error.log
$ command1 1>output.log 2>&1
^^^^^^^^^^^^ ^^^^
redirects STDOUT(1) redirects STDERR
to output.log to STDOUT (&1, aka output.log)
[[}]]
## Process model [[{linux.101.process_model,job_control.101,]]
- Linux follows a parent-child process model.
- Once loaded and initialized during the boot process,
the kernel will launch an initial user-space process in
charge of reading system configuration and (re)start
all other user-space processes that builds a running system.
- Normally this initial process is systemd in modern
systems (or initd in older or embedded ones).
- Each process can optionally launch new children processes
up to the resource limits established on the running system.
- By default a child-process inherits the same user (and so, permissions)
than its parent process. Some processes like the remote
login "sshd" service (ssh is an acronymn for secure-shell) will
change the child-process user/permission to a less privileged
account.
- A simplified process-tree of a running-system will look like: [[{02_doc_has.diagram]]
PROCESS USER PROCESS PARENT-ID
UNIQUE-ID
systemd·······································root 1 0
└─crond·································root 23 1
|-cupsd·································root 70 1
|-rtkit-daemon··························rtkit 100 1
|-sshd··································root 10 1
| └─sshd·····························mike 300 10
| └─bash························mike 301 300
| └─firefox················mike 302 301
|-systemd·······························alice 705 1
| └─at-spi-bus-laun···············alice 706 705
| |···············└─dbus-daemon···alice 707 706
| |-gnome-terminal················alice 883 705
| ─bash-+·········alice 884 883
| └─top·····alice 885 884
|-systemd-journal·······················root 10 1
...
Notice for example that the same process "bash" runs as a user or another
( *mike* or *alice*) depending on the "path"
followed until the process is executed. [[}]]
- The initial sshd running as root user, will span a new sshd child process
with restricted *"mike"* privileges/permissions once the user
has introduced its correct user and password in the remote ssh session, and
from there on, all children will just be able to run with *"mike"*
privileges/permissions.
- Similarly the root systemd process will span a new child process will
restricted *"alice"* privileges/permissions once logged in
the local console, and from there on, all children will just be able to
run with *"alice"* privileges/permissions.
## Executable file vs in-memory process [[{linux.101]]
- Applications are stored on disk drives as files or "bunch-of-instructions and initial data".
- When the kernel executes and application it will read the executable file, load
the "bunch-of-instructions" into RAM memory, setup the initial data, assign
restricted privileges and finally allow the program-in-memory to be executed by
any free-available CPU on the system.
[[}]]
## Basic file permissions [[{linux.101,security.aaa]]
Standar file permissions allows to assign different access permissions to
the owner of the file, the group owner of the file and anyone else.
$ ls -l myFileOfInterest.db
-rw?-r-?---? john accountDepartment .... myFileOfInterest.db
└┼┘│└┼┘│└┼┘│ └┬─┘ └──────┬────────┘
│ │ │ │ │ │ │ │
│ │ │ │ │ │ │ └─ group owner
│ │ │ │ │ │ └──────────── user owner
│ │ │ │ │ │
│ │ │ │ │ └──────────────── sticky bit (hidden if not set)
│ │ │ │ │
│ │ │ │ └────────────────── permissions allowed to others: read access
│ │ │ │
│ │ │ └──────────────────── SUID bit (hidden if not set)
│ │ │
│ │ └────────────────────── permissions allowed to group : read access
│ │
│ └──────────────────────── SUID bit (hidden if not set)
│
└────────────────────────── permissions allowed to user : read&write access
Previous line can be read as:
"""Allow read and write permissions to file-owner "john",
read permissions to group-owner "accountDepartment"
and no permissions to anyone-else """
┌──────┬─────────────────────────────┬─────────────────────────────┐
Permissions │Symbol│ FILE │ DIRECTORY │
┌───────────┼──────┼─────────────────────────────┼─────────────────────────────┤
│ read│ r │ Allows to read the content │ Allows to list the files in │
│ │ │ of the file │ and file-attributes in the │
│ │ │ │ directory │
├───────────┼──────┼─────────────────────────────┼─────────────────────────────┤
│ write│ w │ Allows to write, modify, │ Allows to add and delete │
│ │ │ append or delete the file │ files into the directory and│
│ │ │ content. │ modify metadata (access │
│ │ │ │ or modification time, ...) │
├───────────┼──────┼─────────────────────────────┼─────────────────────────────┤
│ execute│ x │ Allows to execute the │ Allows to enter into the │
│ │ │ program or script │ directory │
├───────────┼──────┼─────────────────────────────┴─────────────────────────────┤
│ sticky │ T │ Only the person that created the file/dir. can change it, │
│ │ │ even if other people have write permissions to file/dir. │
│ │ │ turn on: $ chmod +t someFileOrDir │
│ │ │ Normal /tmp (temporary user files) is an sticky directory │
├───────────┼──────┼───────────────────────────────────────────────────────────┤
│ suid│ S │ Allow SUID/SGID (switch user/group ID). When executed it │
│ │ │ it will be executed with creator/group of file, instead of│
│ │ │ current user. │
│ │ │ turn on: $ chmod +s someFileOrDir │
└───────────┴──────┴───────────────────────────────────────────────────────────┘
[[}]]
## Network Time Protocol 101 [[{configuration.clock,troubleshooting]]
# REF:@[https://www.server-world.info/en/note?os=Fedora_25&p=ntp&f=2]
$ sudo dnf -y install ntp # <- STEP 1) Install ntp client on RedHat flavours
# sudo apt install ntp on Debian/Ubuntu flavours
$ edit /etc/ntp.conf # <- STEP 2) Create/Modify config. file
┌──────────────────────────────────────
│ ...
│ restrict 10.0.0.0 mask 255.255.255.0 nomodify notrap
│ server 0.fedora.pool.ntp.org iburst
│ # server 1.fedora.pool.ntp.org iburst
│ # server 2.fedora.pool.ntp.org iburst
│ # server 3.fedora.pool.ntp.org iburst
│ # server ntp.nict.jp iburst
│ # server ntp1.jst.mfeed.ad.jp iburst
│ # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
│ # Enable/disable NTP servers "at will"
$ sudo systemctl start ntpd # <- STEP 3) Start daemon as (SystemD) service
$ ntpq -p # <- STEP 4) check
$ sudo systemctl enable ntpd # <- STEP 5) Enable at boot time.
$ sudo timedatectl set-timezone Europe/London # <- STEP 5) Set timezone
[[}]]
#[linux_users_groups_summary]
## User Management [[{linux.101,security.aaa]]
$ useradd -D ← display (D)efault values to apply when creating new uses.
(Default group / HOME, SHELL, ...)
Use next flags to update defaults:
--base-dir BASE_DIR : (default to /home) Ignored if --home-dir set
--expiredate EXPIRE_DATE
--inactive INACTIVE : day # after pass.expiration before disabling
--gid GROUP: existing group name or ID for initial group (when --no-user-group used)
--shell SHELL
$ addgroup PROJECT01 ← Best pattern when allowing multi-user access to the linux machine.
Create a new group per project ,then assign permissions to group (vs
to user). This greately simplifies permissions administration in
multi-user setups.
$ useradd alice \ ← Create user alice.
--shell=/usr/bin/git-shell \ ← Assign shell (restricted git-shell in this example)
--shell=/usr/bin/git-shell \ ← upon succesful login alice will be presented with a git shell *1
--groups PROJECT01,PROJECT02\ ← Assign PROJECT01 as default group for alice.
--password ... \ ← create an initial and strong password (or provide alice with ssh keys.
--no-user-group \ ← Do NOT create a new group (otherwise indicated by --gid)
--no-create-home \ ← Do not create the /home/alice directory. (Some sort of
/var/lib/git/PROJECT01 directories already exist with read/write permissions
for PROJECT01, ... groups)
*1 By default /bin/bash o similar is used. git-shell restricts
access to just git-related commands.
Other useful 'useradd' options are:
--skel SKEL_DIR : skel. dir. to be copied in the user's home directory
--key KEY=VALUE : Overrides /etc/login.defs defaults
(UID_MIN, UID_MAX, UMASK, PASS_MAX_DAYS and others).
Example: -K PASS_MAX_DAYS=-1 can be used when creating
system account to turn off password ageing, even though
system account has no password at all.
--no-log-init : Do not add user to lastlog and faillog databases
--non-unique : Allow duplicate (non-unique) existing UID in --uid. Most of the times used to
allow 'root' users with non-root name accounts.
--system : Create system account (no aging, uid chosen in SYS_UID_MIN-SYS_UID_MAX range)
--root CHROOTDIR: Apply changes in chrooted directory.
--uid UID : numerical value for user's ID.
--user-group : Create group with the same name as user, and use as initial group
--selinux-user SEUSER : SELinux user for the user's login
## su/sudo: Switch user: [[{security.aaa,linux.101,01_PM.WiP]]
- su and sudo are mostly used to allow temporal root/superuser access
to standard users for administrative tasks like installing new applications,
re-configuring the network, ...
- sudo is ussually considered safer than su. Ubuntu was the first
distribution to allow sudo-only. Others distributions are also
changing to sudo-only as time passes.
- sudo offers also a plugable architecture not offered by su
to provide different authentication and audit mechanisms.
REF:
- https://www.sudo.ws/ (Sudo Home page)
- https://www.sudo.ws/plugins.html (sudo Third-party plugins)
https://linux.die.net/man/1/su
https://linux.die.net/man/8/sudo
- Ex. ussage:
$ sudo vim /etc/passwd # edit /etc/passwd as root
$ su # Change to root user
[[}]]
## Linux ACLs [[{security.aaa,01_PM.TODO]]
@[https://wiki.archlinux.org/index.php/Access_Control_Lists]
[[}]]
#[sysctl_summary]
## Kernel Tunning [[{configuration.sysctl,configuration.network,storage.fs,troubleshooting.performance,]]
## sysctl
- Cli app to view/modify kernel parameters affecting the runtime behavior
of network, file-systems, kernel logging, ...
- Not to be confused with (SystemD) 'systemctl' used to add/remove/modify/...
services (applications running in background), centralized app logging,...
- TIP: *read comments in /etc/sysctl.conf for most widely tunned params.*
$ sysctl -a ← View all configured kernel params
Output can be filtered out easily with
'grep'.
$ sysctl vm.min_free_kbytes ← View a given param
-n: opt. Show only value (vs param name)
$ sudo sysctl -w net....=0 ← Set new value for kernel param.
$ sudo edit /etc/sysctl.conf ← make changes permanent after reboot
edit /etc/sysctl.d/99-custom.conf for custom
changes.
$ sudo sysctl -p ← apply any change in /etc/sysctl.conf
Use -p /etc/sysctl.d/99-custom.conf for
alternative (non default) path
/proc/sys/ OVERVIEW:
├ abi/vsyscall32
├ debug
│ ├ exception-trace
│ └ kprobes-optimization
├ dev
│ ├ parport
│ │ ├ default/...
│ │ └ parport0/...
│ │ ├ devices
│ │ ...
│ ├ raid/speed_limit_max(min)
│ ├ scsi/logging_level
│ ├ tty/ldisc_autoload
│ ...
├ kernel REF: https://www.kernel.org/doc/html/v5.7/admin-guide/sysctl/kernel.html
│ ├ acct
│ ├ sched_*
│ ├ sem
│ ├ watchdog*
│ ...
├ net
│ ├ core/... (40 tunnable params)
│ ├ ipv4/* (devices, conf, route, tcp, udp,...)
│ ├ ipv6/...
│ ├ netfilter ... (65 tunnable params)
│ ├ nf_conntrack_max
│ ├ unix
│ │ └ max_dgram_qlen
│ ...
├ fs
│ ├ inotify/...
│ ├ nfs/...
│ ├ ext4/...
│ ├ xfs/...
│ ...
├ sunrpc ... (11 tunnable params)
├ user/... (9 tunnable params)
└ vm/... (48 tunnable params)
[[configuration.sysctl}]]
## Kernel Tunning with 'tuned' [[{linux.101,monitoring.101,configuration.kernel,01_PM.TODO,qa.UX]]
/proc/sys/vm/ Kernel FS "control pannel" [[{01_PM.TODO}]]
https://tuned-project.org/
- About tuned:
- OS Tunning is done in practice through:
- long-profiling
- continuous-monitoring
- Tunning becomes harder if system load changes frequently.
Ex.: A system with a peak of load certain hours a day
can be tuned for performance those known hours
and tuned for power-efficency the rest of day
- man 8 tuned: Dynamic Adaptive system tuning daemon
@[https://linux.die.net/man/8/tuned]
- cron-friendly system service that lets to select a tuning profile
(pre-build or custom).
- Tuning include:
- sysctl settings (/proc/sys/)
- settings for disk-elevators
- power management options
- transparent hugepages
- custom-scripts
- Install):
$ sudo dnf -y install tuned # <· RedHat/Fedora/CentOS package install
$ sudo systemctl enable tuned # <· enable tuned service at boot
...
$ sudo systemctl start tuned # <· Start tuned service now
$ sudo systemctl status tuned # <· Check tuned service status
$ sudo systemctl status tuned
→ * tuned.service - Dynamic System Tuning Daemon
→ Loaded: loaded (/usr/lib/systemd/system/tuned.service; disabled; vendor preset: disabled)
→ Active: *active (running)* since Sun 2019-01-20 16:29:05 EST; 15s ago
→ Docs: man:tuned(8)
→ man:tuned.conf(5)
→ man:tuned-adm(8)
→ Main PID: 10552 (tuned)
→ Tasks: 4 (limit: 4915)
→ Memory: 15.7M
→ CGroup: /system.slice/tuned.service
→ └─10552 /usr/bin/python3 -Es /usr/sbin/tuned -l -P
*Ussage:*
$ *tuned-adm list* # ← List existing tunning profiles
→ Available profiles:
→ - balanced - General non-specialized
→ - desktop - Optimize for the desktop
→ - latency-performance - deterministic performance (increased power consumption)
→ - network-latency - deterministic performance low-latency (increased power consumption)
→ - network-throughput - Optimize for streaming network throughput
generally only necessary on older CPUs or
40G+ networks
→ - powersave - Optimize for low power consumption
→ - throughput-performance - provides excellent performance across a
variety of common server workloads
→ - virtual-guest - Optimize for running inside a virtual guest
→ - virtual-host - Optimize for running KVM guests
→ Current active profile: balanced
$ *sudo tuned-adm active* # ← query status
Current active profile: balanced
$ *sudo tuned-adm profile powersave* # ← select profile
[[}]]
[[}]]
## Job/Process control: Scheduling Tasks [[{101,job_control.task_scheduling,01_PM.TODO]]
Officially in Linux/UNIX/Posix, a Job or task is a running process.
- cron : program task to be run periodically
- at
- anacron :
Unlike cron(8), it does not assume that the machine is running continuously.
Hence, it can be used on machines that aren't running 24 hours a day.
Anacron checks whether this job has been executed in the last n days.
If not, Anacron runs the job's shell command, after waiting for the
number of minutes specified as the delay parameter.
NOTE/WARN: Only the date , not hour is used.
e.g.: On Debian-based systems, anacron will be activated hourly every day
from 07:30 local time to 23:30 local time through cron job
## Job control++
- nice : run a process with modified scheduling priority.
- STOP/CONT : kill -STOP $pid "freezes" a process.
- kill -CONT $pid un-freezes it.
- chroot : (Ch)ange Root, runs a process withing a modified view of [[{security.101]]
the OS File System, where the root seen by the process is
just a subdirectory of the real root path seen by the OS.
Basically containerization extends the idea of chroot to
let processes view only a subset of the FS and network
through (kernel supported) namespaces. [[}]]
- cgroups : linux kernel (C)ontrol groups allow processes to be organized
into hierarchical groups whose usage of various types of
resourcescan then be limited and monitored.
/sys/fs/cgroup pseudo-FS is used to control them.
└ cgroup : kernel component modifying the behavior of processes.
subsystem (CPU time, available memory, ...).
also known as resource controllers (or simply, controllers).
...
[[}]]
## GNU Parallel [[{job_control,performance.distributed]]
@[https://linux.die.net/man/1/parallel]
@[https://www.gnu.org/software/parallel/parallel_cheat.pdf]
@[https://www.reddit.com/r/programming/comments/ayew9r/never_got_around_to_learning_gnu_parallel_here_is/]
- Replacement for xargs and for-loops.
- It can also split a file or a stream into blocks and pass those to commands running in parallel.
Ex:
$ parallel --jobs 200% gzip ::: *.html # ← Compress all *.html files in parallel
200% → 2 per CPU thread
$ parallel lame {} -o {.}.mp3 ::: *.wav # ← Convert all *.wav to *.mp3 using lame
$ cat bigfile | \ # ← Chop bigfile into 1MB blocks and grep
parallel --pipe grep foobar for the string foobar
*INPUT SOURCES*
$ parallel echo ::: cmd line input source
$ cat input_from_stdin | parallel echo
$ parallel echo ::: multiple input sources ::: with values
$ parallel -a input_from_file echo
$ parallel echo :::: input_from_file
$ parallel echo :::: input_from_file ::: and command line
*Replacement string*
{} ← mydir/mysubdir/myfile.myext
{.} ← mydir/mysubdir/myfile
{/}, {//}, {/.} ← myfile.myext, mydir/mysubdir, myfile
{#} ← The sequence number of the job
{%} ← The job slot number
{2} ← Value from the second input source
{2.} {2/} {2//} {2/.} ← Combination of {2} and {.} {/} {//} {/.}
{= perl expression =} ← Change $_ with perl expression
$ parallel --keep-order "sleep {}; echo {}" ::: 5 4 3 2 1 # ← *Keep input order in output*
*Control the execution*
$ parallel --jobs 2 "sleep {}; echo {}" ::: 5 4 3 2 1 # ← Run 2 jobs in parallel
$ parallel --dryrun echo ::: Red Green Blue ::: S M L # ← Dry-run. See will be executed
without real execution
*Remote execution*
$ parallel -S server1 -S server2 "hostname; echo {}" ::: foo bar
*Pipe mode*
cat bigfile | parallel --pipe wc -l
$ parallel -a bigfile --pipepart --block -1 grep foobar # ← Chop bigfile into one block per CPU
thread and grep for foobar
[[}]]
# Monitoring
## basic process monit/control [[{job_control.101,linux.101,monitoring.jobs,monitoring.i/o]]
$ *ps* ← shows list of the processes running. i
Without options: processes belonging to current user&with a controlling terminal
Ex. options include:
-a: all processes from all users
-u: add user names, %cpu usage, and %mem usage,...
-x: add also processes without controlling terminals
-l: add information including UID and nice value
--forest: show process hierarchy.
$ *pstree* ← show parent/children process tree (-p flag show pid)
$ *top -n * ← Display top by cpu processes once and finish
$ *top* ← real-time display processes ordered by memory/CPU/...(as in CPU usage)
Z,B,E,e Global: 'Z' colors; 'B' bold; 'E'/'e' summary/task memory scale
l,t,m Toggle Summary: 'l' load avg; 't' task/cpu stats; 'm' memory info
0,1,2,3,I Toggle: '0' zeros; '1/2/3' cpus or numa node views; 'I' Irix mode
f,F,X Fields: 'f'/'F' add/remove/order/sort; 'X' increase fixed-width
L,&,<,> . Locate: 'L'/'&' find/again; Move sort column: '<'/'>' left/right
R,H,V,J . Toggle: 'R' Sort; 'H' Threads; 'V' Forest view; 'J' Num justify
c,i,S,j . Toggle: 'c' Cmd name/line; 'i' Idle; 'S' Time; 'j' Str justify
x,y . Toggle highlights: 'x' sort field; 'y' running tasks
z,b . Toggle: 'z' color/mono; 'b' bold/reverse (only if 'x' or 'y')
u,U,o,O . Filter by: 'u'/'U' effective/any user; 'o'/'O' other criteria
n,#,^O . Set: 'n'/'#' max tasks displayed; Show: Ctrl+'O' other filter(s)
C,... . Toggle scroll coordinates msg for: up,down,left,right,home,end
k,r Manipulate tasks: 'k' kill; 'r' renice
d or s Set update interval
W,Y Write configuration file 'W'; Inspect other output 'Y'
q Quit
$ *iotop* # ← Simple top-like I/O monitor
@[https://linux.die.net/man/1/iotop]
$ *kill - * ← Display existing signals (Default to SIGTERM that most of the times
will just terminate the process "cleanely")
→ 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
→ 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1
→ 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
→ 16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
→ 21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ
→ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR
→ 31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3
→ 38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8
→ 43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
→ 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
→ 53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7
→ 58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2
→ 63) SIGRTMAX-1 64) SIGRTMAX
$ *kill [ -s (signal name)] 'process_id'* ← Send signal to process. kill -9 kills unconditionally
$ killall "process_name" ← send signal to all processes matching full name
$ pkill "process_name" ← send signal to all processes matching part of the name
$ skill ← send a particular signal to command/username/tty.
-L --- list the various signals that can be sent
-u --- specify a username;
-p --- process id (followed by the process id)
-c --- command name (this is the same as killall)
-t --- (tty number)
-v --- verbose mode
-i --- interactive mode.
PAUSE AND CONTINUE A PROCESS:
$ *kill -STOP "pid"* # Pauses
$ *kill -CONT "pid"* # Continues
$ *nice -20 make* ← Sets make priority to -20
-20 is maximum priority (negative only allowed to root)
20 is the minimum priority.
$ *renice 10 "pid"* ← Changes priority of running process.
[[}]]
## Disk Space [[{troubleshooting.disk_full]]
$ df -kh <·· (D)isk (F)ree report.
-k : take block-size=1K
-h : human-readable, print sizes in powers of 1024 (e.g., 1023M)
$ du -sch dir1 dir2 <·· (D)isk (U)ssage report for files&directories.
-s: summarize. Display only a total for each argument
-c: produce grand total
-h : human-readable
[[}]]
## `pv` monit data pipe progress [[{monitoring.pipes,monitoring.i/o,profiling.storage.FS,storage.profiling]]
## pv provides time elasped, %compl., progressbar, throughput, ETA, ...on STDERR.
## cat /dev/zero | pv | cat > /dev/null # Check CPU/memory perf.
## pv file | nc -w 1 somewhere.com 3000 # Check network perf.
## pv -EE /dev/sda > disk-image.img # Check disk image (skip errors)
man pv summary
OUTPUT MODIFIERS
--wait: until first byte has been transferred.
--delay-start SEC (before showing info)
--size SIZE: Set total size (if piping from STDIN and size is known)
--line-mode: (count lines vs counting bytes)
--null for null terminated.
--interval SEC: (1 sec by default)
--width N / --height N: Assume width / height for terminal
--name NAME: Useful with --cursor for "compilcated pipelines"
--cursor: Use cursor pos. escape sequences (vs CR)
--force: (even if STDOUT is not terminal)
DATA TRANSFER MODIFIERS
--rate-limit RATE
--buffer-size BYTES
--no-splice: Never use splice(2). Ussually more efficient way of
transferring data from/to pipe than regular read/write.
but means that the transfer buffer may not be used.
This prevents -A and -T from working.
-E/--skip-errors (set twice to only report a read error once per file)
--stop-at-size
--watchfd PID[:FD]. Wath FIle Descriptor of process PID, and show its
progress.
--remote PID : where PID is an running instance of pv already running.
Add other commands to such instance.
--pidfile FILE save PID of (first running) pv instance.
[[}]]
## Netdata (Glances++) [[{]]
https://github.com/netdata/netdata
(Used by AWS, ..)
[[}]]
## Glances [[{monitoring.101,01_PM.low_code,QA.UX,troubleshooting]]
@[https://www.tecmint.com/glances-an-advanced-real-time-system-monitoring-tool-for-linux/]
@[https://github.com/tldr-pages/tldr/blob/master/pages/common/glances.md]
- Alternative to top
- Linux/FreeBSD python command-line, curses-based, using psutils
under the hood.
- It makes easier to find an application/process consuming lots
of system resources by highlighting programs consuming too much
resources and providing maximum of information about the server.
- Allows to define thresholds (careful, warning and critical) in
config files.
- Display info about:
- CPU (user, kernel, idle processes).
- RAM, Swap, Free memory,... etc.
- Average CPU load for past 1min, 5mins and 15 mins.
- Network Download/Upload rates of network connections.
- Total number of processes, active ones, sleeping processes etc.
- Disk I/O related (read or write) speed details
- Currently mounted devices disk usages.
- Top processes CPU/Memory usages, Names and file path of exec
- official packages for Debian/RedHat/...
*USSAGE*
$ glances
GREEN : OK (everything is fine)
BLUE : CAREFUL (need attention)
VIOLET: WARNING (alert)
RED : CRITICAL (critical)
*Shortcuts:*
a – Sort processes automatically
c – Sort processes by CPU%
m – Sort processes by MEM%
p – Sort processes by name
i – Sort processes by I/O rate
d – Show/hide disk I/O stats ols
f – Show/hide file system statshddtemp
n – Show/hide network stats
s – Show/hide sensors stats
y – Show/hide hddtemp stats
l – Show/hide logs
b – Bytes or bits for network I/Oools
w – Delete warning logs
x – Delete warning and critical logs
x – Delete warning and critical logs
1 – Global CPU or per-CPU stats
h – Show/hide this help screen
t – View network I/O as combination
u – View cumulative network I/O
q – Quit (Esc and Ctrl-C also work)
*default thresholds: /etc/glances/glances.conf*
(careful=50, warning=70 and critical=90)
- Client/Server mode:
ON THE SERVER ON THE CLIENT
$*# glances -s -B $ADDRESS -p $PORT * $*# glances -c -P 172.16.27.56 *
(0.0.0.0) (61209)
Define the password for the Glances server
Password:
Password (confirm):
Glances server is running on 0.0.0.0:61209
[[}]]
## Dstat (vmstat + iostat+ ifstat + mpstat) [[{monitoring.101,monitoring.memory,monitoring.I/O,monitoring.network,monitoring.cpu,01_PM.TODO,troubleshooting]]
@[https://linux.die.net/man/1/dstat]
- Dstat: moderm replacement joining the info from [[{02_doc_has.comparative]]
vmstat, iostat, ifstat an mpstat. [[}]]
- Dstat overcomes some of the limitations and adds some extra features.
- Dstat allows you to view all of your system resources instantly,
you can eg.:
- compare disk usage in combination with interrupts from disk controller.
- compare network bandwidth numbers directly with the disk throughput (in the same interval).
[[}]]
## pidstat: process stats [[{monitoring.jobs,job_control,monitoring.i/o]]
[[profiling.jobs.IO.block,profiling.storage,storge.profiling]]
By Sebastien Godard (http://pagesperso-orange.fr/sebastien.godard/)
-d Report I/O statistics
kB_rd/s
kB_wr/s
kB_ccwr/s: Cancelled-writes eg. task truncates some dirty pagecache.
(IO which another task has been accounted for).
iodelay : Block-I/O delay of task-monitored, measured in clock ticks. [[{profiling.jobs.]]
including delays waiting for sync-block-I/O and swaping-block-I/O completion. [[}]]
--dec= 0|1|2* number of decimal-places to use.
-e 'program' arg1 arg2 ... Exec. program + args to be monitored.
-G cmd_name_regex Filter by processes mathching regex for its command-name .
-H Display timestamp-in-seconds since epoch. (1970-01-01)
-h Display activities horizontally in a single line, without statistics (for batch tasks)
--human
-I In SMP archs, indicate to divide task CPU ussage should be divided by total num.or proccessors.
-l Display command-name + all its arguments.
-p $PID Filter by process $PID Select tasks (processes)
Use SELF for the pidstat process itself.
Use ALL for all tasks.
-R Report realtime-priority and scheduling-policy info, including:
-r Report pagefaults and memory utilization. [[profiling.jobs.pagefaults]]
minflt/s : Total number of minor-faults per sec, (not loading from disk).
minflt-nr: Total number of minor-faults for task-and-children for time-interval.
majflt/s : Total number of major-faults per secd,
majflt-nr: Total number of major-faults for task-and-children for time-interval.
VSZ : virtual memory usage of entire task in kilobytes.
RSS : Resident Set Size: (non-swapped physical memory used in kilobytes).
%MEM : Resident Set Size %
-nr : Total number of major-faults by task-and-children for time-interval.
-s Report stack utilization. [[profiling.jobs.memory.stack]]
StkSize : memory in kilobytes reserved for the task as stack, (not necessarily used)
StkRef : memory in kilobytes used as stack, referenced by the task.
-T {TASK*|CHILD|ALL}
-t display statistics per-thread.
-U [ username ] Display real-user-name or filter by user-name.
-u Report CPU utilization. [[{profiling.jobs.cpu]]
%usr : CPU used by task at user-level NOT include time spent running a Virt.CPU.
usr-ms : Total millisecs spent by task+children at user level.
%system : CPU used by task while executing at the system level (kernel).
system-ms: Total millisecs spent by task+children at systeml level (kernel).
%guest : Percentage of CPU spent by the task in virt. machine. (KVM/Qemu/...?)
guest-ms : Total millisecs spent by task+children in virt. machine. (KVM/Qemu/...?)
%wait : Percentage of CPU spent by the task while waiting to run.
%CPU : Total percentage of CPU time used by the task. [[}]]
Total number of milliseconds spent by the task and all its children in virtual
machine (running a virtual processor).
-v Report some kernel tables. [[monitoring.jobs.kernel]]
-w Report task switching activity: [[profiling.jobs.kernel]] [[{profiling.jobs.kernel]]
cswch/s : Total number of voluntary context-switches per sec.
voluntary == task blocks waiting for a resource.
nvcswch/s: Total number of non-voluntary context switches per sec.
non voluntary == task executes for the duration of its time slice and then
is forced to relinquish the processor. [[}]]
ENV.VARS:
S_COLORS := never|always|auto (display statistics in color)
EXAMPLES
$ pidstat 2 5 <··· Dump 5-reports of CPU-statistics for every active task at 2-secs intervals.
$ pidstat -r -p 1643 2 5
Display five reports of page faults and memory statistics for PID 1643 at two second
intervals.
$ pidstat -C "fox|bird" -r -p ALL
Display global page faults and memory statistics for all the processes whose command
name includes the string "fox" or "bird".
$ pidstat -T CHILD -r 2 5
Display five reports of page faults statistics at two second intervals for the child
processes of all tasks in the system. Only child processes with non-zero statistics
values are displayed.
[[}]]
## iostat(CPU, I/O, FS) [[{MONITORING.I/O,MONITORING.STORAGE,01_PM.TODO]]
@[https://linux.die.net/man/1/iostat]
$ iostat -xt 2 # -x Show extended statistics
# -t Print time
avg-cpu: %user %nice %system %iowait %steal %idle
4.27 0.00 4.27 2.26 0.00 89.20
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
sda 0.00 5.50 0.00 36.00 0.00 5.00 0.00 47.62 0.00 17.55 0.09 0.00 6.55 0.82 0.45
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 3.00 0.00 12.00 0.00 0.00 0.00 0.00 0.00 123.00 0.37 0.00 4.00 0.50 0.15
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 7.50 0.00 30.00 0.00 0.00 0.00 0.00 0.00 0.47 0.00 0.00 4.00 0.40 0.30
^^^^ ^^^^ ^^^^ ^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^
read r_await r_await aqu-sz ignore elapsed
req. avg msec avg msec avg to be time %
for read for read queue remove during
requests requests length which
of req I/O req
issued were
issued
BANDWIDTH
USSAGE
[[}]]
## blktrace (block I/O traffic) [[{monitoring.storage.blocks]] (man 8 blktrace)
generate traces of the i/o traffic on (low-level) block devices vs (high-level) File Systems.
$ sudo blktrace -d /dev/sda -o - | blkparse -i - <··· trace i/o for /dev/sda
Alt. using btrace script:
$ btrace /dev/sda
$ blktrace /dev/sda /dev/sdb <··· trace /dev/sda+/dev/sdb to current dir.
$ blkparse sda sdb <··· View saved trace
SYNOPSIS
blktrace -d dev [ -r debugfs_path ] [ -o output ] [ -w time ] [ -a action ] [ -A action_mask ] [ -v ]
DESCRIPTION
OPTIONS
-a/--set-mask=hex-mask : Set filter mask to hex-mask (see below for masks)
barrier : barrier attribute
complete: completed by driver
discard : discard / trim traces
fs : requests
issue : issued to driver
pc : packet command events
queue : queue operations
read : read traces
requeue : requeue operations
sync : synchronous attribute
write : write traces
notify : trace messages
drv_data: additional driver specific trace
--act-mask=mask : Add mask to current filter (see below for masks)
--buffer-size=size : Specifies (1024) buffer size for event extraction. Def 512KiB.
--dev=dev : Adds dev as a device to trace
--input-devs=file : Adds the devices found in file as devices to trace
--num-sub-buffers=X : number of buffers to use. Def: 4 sub buffers.
--listen : Run in network listen/server mode.
--host=hostname : Run in network client mode
--port=number : Network port to use (default 8462)
--no-sendfile : Make the network client NOT use sendfile() to transfer data
--output=basename : Specifies base name for input files. Default is device.blktrace.cpu.
Specifying -o - runs in live mode with blkparse (writing data to standard out).
--output-dir=dir : Prepend file to output file name(s)
--relay=rel-path : Specifies debugfs mount point
--version
-w seconds
--stopwatch=seconds
REQUEST TYPES:
- file system : normal read/write op from a specific disk location at a given size.
typically originating from a user process
- SCSI commands: blktrace sends the command data block as a payload so that blkparse can decode it.
[[}]]
## powertop [[{monitoring.hardware]]
Allows to:
- diagnose device/CPU power consumption issues
- Tune/control device/CPU power management.
$ sudo powertop # ← Interactive mode if no other option is provided
$ sudo powertop --auto-tune # ← Callibrate non-interactively
^^^^^^^^^^^^^^^^^^^^
To enable at system boot add next systemd Unit:
*STEP 1: Create/Edit powertop.service like:*
$ sudoedit /etc/systemd/system/powertop.service
(Add next lines)
+ [Unit]
+ Description=Powertop auto-tune
+
+ [Service]
+ ExecStart=/usr/bin/powertop --auto-tune
+ RemainAfterExit=true
+
+ [Install]
+ WantedBy=multi-user.target
*STEP 2: Enable the new service like:*
$ sudo systemctl daemon-reload
$ sudo systemctl enable powertop
$ sudo systemctl start powertop
*STEP 3: Check it has run properly*
$ sudo journalctl -u powertop
→ ...
→ systemd[1]: Started Powertop auto-tune.
→ powertop[4778]: modprobe cpufreq_stats failedLoaded 0 prior measurements
→ powertop[4778]: RAPL device for cpu 0
→ powertop[4778]: RAPL Using PowerCap Sysfs : Domain Mask d
→ powertop[4778]: RAPL device for cpu 0
→ powertop[4778]: RAPL Using PowerCap Sysfs : Domain Mask d
→ powertop[4778]: Devfreq not enabled
→ powertop[4778]: glob returned GLOB_ABORTED
→ powertop[4778]: *Leaving PowerTOP*
OTHER PERTINENT OPTIONS:
--calibrate : Runs in calibration mode: When running on battery,
powertop can track power consumption as well as system
activity.
When there are enough measurements, powertop can start
to report power estimates.
-csv=file : Generate CSV report.
-html=file : Generate an HTML report.
-extech=$USBDEV: Use Extech Power Analyzer for analysis
USBDEV will be a USB adaptor similar to /dev/ttyUSB0
-iteration=$Num : Number of times to run each test.
-time[=seconds] : Generate report for specified number of seconds.
-workload=file : Execute workload file as a part of calibration
....
[[}]]
## perf (kernel counters) [[{monitoring.kernel.counters,profiling.jobs,dev_stack.java,profiling.locks,01_PM.TODO]]
TODO @[http://www.brendangregg.com/perf.html]
TODO @[http://www.brendangregg.com/flamegraphs.html]
- Perf: new kernel-based subsystem that provide a framework for all things performance analysis.
- It covers hardware level (CPU/PMU, Performance Monitoring Unit)
features and software features (software counters, tracepoints) as well.
$ perf stat <- -gather perf-counter stats. @[https://linux.die.net/man/1/perf-stat]
[--event=EVENT] ← PMU event in the form:
- symbolic event name (perf list to list)
- raw PMU event (eventsel+umask) with format
rNNN where NNN is an hexadecimal event descriptor
[--no-inherit] ← child tasks do not inherit counters
[--all-cpus]
[--pid=$pid] ← comma separated list of existing processes
[--tid=$tid] ← comma separated list of existing thread id
[--scale] ← scale/normalize counter values
[--repeat=n] ← repeat command, print average + stddev (max: 100)
[--big-num] ← print large numbers with thousands-local-separator
[--cpu=] ← comma separated list of cpus (All if not provided)
[--no-aggr] ← Do not aggregate counts across all monitored CPUs
in system-wide mode (-a). Only valid in system-wide mode.