-
Notifications
You must be signed in to change notification settings - Fork 27
/
tutorial.txt
1350 lines (1099 loc) · 62 KB
/
tutorial.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Cpuset (cset) Tutorial
======================
Alex Tsariounov <tsariounov@gmail.com> +
Copyright (c) 2009-2011 Novell Inc., cset v1.6pre1 +
Verbatim copying and distribution of this entire article are permitted
worldwide, without royalty, in any medium, provided this notice is preserved.
This tutorial describes basic and advanced usage of the *cset* command to
manipulate cpusets on a Linux system. See also the manpages that come with
the *cset* command: cset(1), cset-shield(1), cset-set(1), and cset-proc(1) for
more details. Additionally, the *cset* command has online help.
Introduction
============
In the Linux kernel, the cpuset facility provides a mechanism for creating
logical entities called "cpusets" that encompass definitions of CPUs and NUMA
Memory Nodes (if NUMA is available). Cpusets constrain the CPU and Memory
placement of a task to only the resources defined within that cpuset. These
cpusets can then be arranged into a nested hierarchy visible in the "cpuset"
virtual filesystem. Sets of tasks can be assigned to these cpusets to
constrain the resources that they use. The tasks can be moved from one cpuset
to another to utilize other resources defined in those other cpusets.
The *cset* command is a Python application that provides a command line front
end for the Linux cpusets functionality. Working with cpusets directly can be
confusing and slightly complex. The *cset* tool hides that complexity behind
an easy-to-use command line interface.
There are two distinct use cases for *cset*: the basic shielding use case and
the "advanced" case of using raw +set+ and +proc+ subcommands. The basic
shielding function is accessed with the +shield+ subcommand and described in
the next section. Using the raw +set+ and +proc+ subcommands allows one to
set up arbitrarily complex cpusets and is described in the later sections.
Note that in general, one either uses the +shield+ subcommand 'or' a
combination of the +set+ and +proc+ subcommands. One rarely, if ever, uses
all of these subcommands together. Doing so will likely become too
confusing. Additionally, the +shield+ subcommand sets up its required cpusets
with exclusively marked CPUs. This can interfere with your cpuset
strategy. If you find that you need more functionality for your strategy than
+shield+ provides, go ahead and transition to using +set+ and +proc+
exclusively. It is straightforward to implement what +shield+ does with a few
extra +set+ and +proc+ subcommands.
Obtaining Online Help
---------------------
For a full list of *cset* subcommands::
+# cset help+
For in-depth help on individual subcommands::
+# cset help <subcommand>+
For options of individual subcommands::
+# cset <subcommand> (-h | --help)+
The Basic Shielding Model
=========================
Although any set up of cpusets can really be described as "shielding," there
is one prevalent shielding model in use that is so common that *cset* has a
subcommand that is dedicated to its use. This subcommand is called +shield+.
The concept behind this model is the use of three cpusets. The 'root' cpuset
which is always present in all configurations and contains all CPUs. The
'system' cpuset which contains CPUs which are used for system tasks. These
are the normal tasks that are not "important," but which need to run on the
system. And finally, the 'user' cpuset which contains CPUs which are used for
"important" tasks. The 'user' cpuset is the shield. Only those tasks that
are somehow important, usually tasks whose performance determines the overall
rating for the machine, are run in the 'user' cpuset.
The +shield+ subcommand manages all of these cpusets and lets you define the
CPUs and Memory Nodes that are in the 'shielded' and 'unshielded' sets. The
subcommand automatically moves all movable tasks on the system into the
'unshielded' cpuset on shield activation, and back into the 'root' cpuset on
shield tear down. The subcommand then lets you move tasks into and out of the
shield. Additionally, you can move special tasks (kernel threads) which
normally run in the 'root' cpuset into the 'unshielded' set so that your
shield will have even less disturbance.
The +shield+ subcommand abstracts the management of these cpusets away from
you and provides options that drive how the shield is set up, which tasks are
to be shielded and which tasks are not, and status of the shield. In fact,
you need not be bothered with the naming of the required cpusets or even where
the cpuset filesystem is mounted. *Cset* and the +shield+ subcommand takes
care of all that.
If you find yourself needing to define more cpusets for your application, then
it is likely that this simple shielding is not a rich enough model for you.
In this case, you should transition to using the +set+ and +proc+ subcommands
described in a later section.
A Simple Shielding Example
--------------------------
Assume that we have a 4-way machine that is not NUMA. This means there are 4
CPUs at our disposal and there is only one Memory Node available. On such
machines, we do not need to specify any memory node parameters to *cset*, it
sets up the only available memory node by default.
Usually, one wants to dedicate as many CPUs to the shield as possible and
leave a minimal set of CPUs for normal system processing. The reasoning for
this is because the performance of the important tasks will rule the
performance of the installation as a whole and these important tasks need as
many resources available to them as possible, exclusive of other, unimportant
tasks that are running on the system.
NOTE: I use the word "task" to represent either a process or a thread that is
running on the system.
Setup and Teardown of the Shield
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To set up a shield of 3 CPUs with 1 CPU left for low priority system
processing, issue the following command.
-------------------------------------------------------------------
[zuul:cpuset-trunk]# cset shield -c 1-3
cset: --> activating shielding:
cset: moving 176 tasks from root into system cpuset...
[==================================================]%
cset: "system" cpuset of CPUSPEC(0) with 176 tasks running
cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running
-------------------------------------------------------------------
This command does a number of things. First, a 'user' cpuset is created with
what's called a CPUSPEC (CPU specification) from the +-c/\--cpu+ option. This
CPUSPEC specifies to use CPUs 1 through 3 inclusively. Next, the command
creates a 'system' cpuset with a CPUSPEC that is the inverse of the +-c+
option for the current machine. On this machine that cpuset will only contain
the first CPU, CPU0. Next, all userspace processes running in the 'root'
cpuset are transfered to the 'system' cpuset. This makes all those processes
run only on CPU0. The effect of this is that the shield consists of CPUs 1
through 3 and they are now idling.
Note that the command did not move the kernel threads that are running in the
'root' cpuset to the 'system' cpuset. This is because you may want these
kernel threads to use all available CPUs. If you do not, the you can use the
+-k/\--kthread+ option as described below.
The shield setup command above outputs the information of which cpusets were
created and how many tasks are running on each. If you want to see the
current status of the shield again, issue this command:
-------------------------------------------------------------------
[zuul:cpuset-trunk]# cset shield
cset: --> shielding system active with
cset: "system" cpuset of CPUSPEC(0) with 176 tasks running
cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running
-------------------------------------------------------------------
Which shows us that the shield is set up and that 176 tasks are running in the
'system' cpuset--the "unshielded" cpuset.
It is important to move all possible tasks from the 'root' cpuset to the
unshielded 'system' cpuset because a task's cpuset property is inherited by
its children. Since we've moved all running tasks (including init) to the
unshielded 'system' cpuset, that means that any new tasks that are spawned
will *also* run in the unshielded 'system' cpuset.
Some kernel threads can be moved into the unshielded 'system' cpuset as well.
These are the threads that are not bound to specific CPUs. If a kernel thread
is bound to a specific CPU, then it is generally not a good idea to move that
thread to the 'system' set because at worst it may hang the system and at best
it will slow the system down significantly. These threads are usually the IRQ
threads on a real time Linux kernel, for example, and you may want to not move
these kernel threads into 'system'. If you leave them in the 'root' cpuset,
then they will have access to all CPUs.
However, if your application demands an even "quieter" shield, then you can
move all movable kernel threads into the unshielded 'system' set with the
following command.
--------------------------------------------------------------------
[zuul:cpuset-trunk]# cset shield -k on
cset: --> activating kthread shielding
cset: kthread shield activated, moving 70 tasks into system cpuset...
[==================================================]%
cset: done
--------------------------------------------------------------------
You can see that this moved an additional 70 tasks to the unshielded 'system'
cpuset. Note that the +-k/\--kthread on+ parameter can be given at the shield
creation time as well and you do not need to perform these two steps
separately if you know that you will want kernel thread shielding as well.
Executing *cset shield* again shows us the current state of the shield.
--------------------------------------------------------------------
[zuul:cpuset-trunk]# cset shield
cset: --> shielding system active with
cset: "system" cpuset of CPUSPEC(0) with 246 tasks running
cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running
--------------------------------------------------------------------
You can get a detailed listing of what is running in the shield by specifying
either +-s/\--shield+ or +-u/\--unshield+ to the +shield+ subcommand and using
the verbose flag. You will get output similar to the following.
--------------------------------------------------------------------
[zuul:cpuset-trunk]# cset shield --unshield -v
cset: "system" cpuset of CPUSPEC(0) with 251 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root 1 0 Soth init [5]
root 2 0 Soth [kthreadd]
root 84 2 Sf50 [IRQ-9]
...
alext 31796 31789 Soth less
root 32653 25222 Roth python ./cset shield --unshield -v
--------------------------------------------------------------------
Note that I abbreviated the listing; we do have 251 tasks running in the
'system' set. The output is self-explanatory; however, the "SPPr" field may
need a little explanation. "SPPr" stands for State, Policy and Priority. You
can see that the initial two tasks are Stopped and running in timeshare
priority, marked as "oth" (for "other"). The [IRQ-9] task is also stopped,
but marked at real time FIFO policy with a priority of 50. The last task in
the listing is the *cset* command itself and is marked as running. Also note
that adding a second +-v/\--verbose+ option will not restrict the output to
fit into an 80 character screen.
Tear down of the shield, stopping the shield in other words, is done with the
+-r/\--reset+ option to the +shield+ subcommand. When this command is issued,
both the 'system' and 'user' cpusets are deleted and any tasks that are
running in both of those cpusets are moved to the 'root' cpuset. Once so
moved, all tasks will have access to all resources on the system. For
example:
--------------------------------------------------------------------
[zuul:cpuset-trunk]# cset shield --reset
cset: --> deactivating/reseting shielding
cset: moving 0 tasks from "/user" user set to root set...
cset: moving 250 tasks from "/system" system set to root set...
[==================================================]%
cset: deleting "/user" and "/system" sets
cset: done
--------------------------------------------------------------------
Moving Interesting Tasks Into and Out of the Shield
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Now that we have a shield running, the objective is to run our "important"
processes in that shield. These processes can be anything, but usually they
are directly related to the purpose of the machine. There are two ways to run
tasks in the shield:
. Exec a process into the shield
. Move an already running task into the shield
Execing a Process into the Shield
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Running a new process in the shield can be done with the +-e/\--exec+ option
to the +shield+ subcommand. This is the simplest way to get a task to run in
the shield. For this example, let's exec a new bash shell into the shield
with the following commands.
--------------------------------------------------------------------
[zuul:cpuset-trunk]# cset shield -s
cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running
cset: done
[zuul:cpuset-trunk]# cset shield -e bash
cset: --> last message, executed args into cpuset "/user", new pid is: 13300
[zuul:cpuset-trunk]# cset shield -s -v
cset: "user" cpuset of CPUSPEC(1-3) with 2 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root 13300 8583 Soth bash
root 13329 13300 Roth python ./cset shield -s -v
[zuul:cpuset-trunk]# exit
[zuul:cpuset-trunk]# cset shield -s
cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running
cset: done
--------------------------------------------------------------------
The first command above lists the status of the shield. We see that the
shield is defined as CPUs 1 through 3 inclusive and currently there are no
tasks running in it.
The second command execs the bash shell into the shield with the +-e+
option. The last message of cset lists the PID of the new process.
NOTE: *cset* follows the tradition of separating the tool options from the
command to be execed options with a double dash (+\--+). This is not shown in
this simple example, but if the command you want to exec also takes options,
separate them with the double dash like so: +# cset shield -e mycommand \-- -v+
The +-v+ will be passed to +mycommand+, and not to *cset*.
The next command lists the status of the shield again. You will note that
there are actually two tasks running shielded: our new shell and the *cset*
status command itself. Remember that the cpuset property of a task is
inherited by its children. Since we ran the new shell in the shield, its
child, which is the status command, also ran in the shield.
TIP: Execing a shell into the shield is a useful way to experiment with
running tasks in the shield since all children of the shell will also run in
the shield.
The last command exits the shell after which we request a shield status again
and see that once again, it does not contain any tasks.
You may have noticed in the output above that both the new shell and the
status command are running as the root user. This is because *cset* needs to
run as root and so all it's children will also run as root. If you need to
run a process under a different user and or group, you may use the +\--user+
and +\--group+ options for +exec+ as follows.
--------------------------------------------------------------------
[zuul:cpuset-trunk]# cset shield --user=alext --group=users -e bash
cset: --> last message, executed args into cpuset "/user", new pid is: 14212
alext@zuul> cset shield -s -v
cset: "user" cpuset of CPUSPEC(1-3) with 2 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
alext 14212 8583 Soth bash
alext 14241 14212 Roth python ./cset shield -s -v
--------------------------------------------------------------------
Moving a Running Task into and out of the Shield
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
While execing a process into the shield is undoubtably useful, most of the
time, you'll want to move already running tasks into and out of the shield.
The *cset* shield subcommand includes two options for doing this:
+-s/\--shield+ and +-u/\--unshield+. These options require what's called a
PIDSPEC (process specification) to also be specified with the +-p/\--pid+
option. The PIDSPEC defines which tasks get operated on. The PIDSPEC can be
a single process ID, a list of process IDs separated by commas, and a list of
process ID ranges separated by dashes, groups of which are separated by
commas. For example:
+\--shield --pid 1234+::
This PIDSPEC argument specifies that PID 1234 be shielded.
+\--shield --pid 1234,42,1934,15000,15001,15002+::
This PIDSPEC argument specifies that this list of PIDs only be moved into the
shield.
+\--unshield -p 5000,5100,6010-7000,9232+::
This PIDSPEC argument specifies that PIDs 5000,5100 and 9232 be unshielded
(moved out of the shield) along with any existing PID that is in the range
6010 through 7000 inclusive.
NOTE: A range in a PIDSPEC does not have to have tasks running for every
number in that range. In fact, it is not even an error if there are no tasks
running in that range; none will be moved in that case. The range simply
specifies to act on any tasks that have a PID or TID that is within that
range.
Use of the appropriate PIDSPEC can thus be handy to move tasks and groups of
tasks into and out of the shield. Additionally, there is one more option that
can help with multi-threaded processes, and that is the +\--threads+ flag. If
this flag is present in a +shield+ or +unshield+ command with a PIDSPEC and if
any of the task IDs in the PIDSPEC belong to a thread in a process container,
then *all* the sibling threads in that process container will get shielded or
unshielded as well. This flag provides an easy mechanism to shield/unshield
all threads of a process by simply specifying one thread in that process.
In the following example, we move the current shell into the shield with a
range PIDSPEC and back out with the bash variable for the current PID.
-----------------------------------------------------------------------
[zuul:cpuset-trunk]# echo $$
22018
[zuul:cpuset-trunk]# cset shield -s -p 22010-22020
cset: --> shielding following pidspec: 22010-22020
cset: done
[zuul:cpuset-trunk]# cset shield -s -v
cset: "user" cpuset of CPUSPEC(1-3) with 2 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root 3770 22018 Roth python ./cset shield -s -v
root 22018 5034 Soth bash
cset: done
[zuul:cpuset-trunk]# cset shield -u -p $$
cset: --> unshielding following pidspec: 22018
cset: done
[zuul:cpuset-trunk]# cset shield -s
cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running
cset: done
-----------------------------------------------------------------------
NOTE: Ordinarily, the +shield+ option will shield a PIDSPEC only if it is
currently running in the 'system' set--the unshielded set. The +unshield+
option will unshield a PIDSPEC only if it is currently running in the 'user'
set--the shielded set. If you want to +shield/unshield+ a process that
happens to be running in the 'root' set (not common), then use the +\--force+
option for these commands.
Full Featured Cpuset Manipulation Commands
==========================================
While basic shielding as described above is useful and a common use model for
*cset*, there comes a time when more functionality will be desired to
implement your strategy. To implement this, *cset* provides two subcommands:
+set+, which allows you to manipulate cpusets; and +proc+, which allows you to
manipulate processes within those cpusets.
The Set Subcommand
------------------
In order to do anything with cpusets, you must be able to create, adjust,
rename, move and destroy them. The +set+ subcommand allows the management of
cpusets in such a manner.
Creating and Destroying Cpusets with Set
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The basic syntax of +set+ for cpuset creation is:
----------------------------------------------------------
[zuul:cpuset-trunk]# cset set -c 1-3 -s my_cpuset1
cset: --> created cpuset "my_cpuset1"
----------------------------------------------------------
This creates a cpuset named "my_cpuset1" with a CPUSPEC of CPU1, CPU2 and
CPU3. The CPUSPEC is the same concept as described in the "Setup and
Teardown of the Shield" section above. The +set+ subcommand also takes a
+-m/\--mem+ option that lets you specify the memory nodes the set will use as
well as flags to make the CPUs and MEMs exclusive to the cpuset. If you are
on a non-NUMA machine, just leave the +-m+ option out and the default memory
node 0 will be used.
Just like with +shield+, you can adjust the CPUs and MEMs with subsequent
calls to +set+. If, for example, you wish to adjust the "my_cpuset1" cpuset
to only use CPUs 1 and 3 (and omit CPU2), then issue the following command.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset set -c 1,3 -s my_cpuset1
cset: --> modified cpuset "my_cpuset
----------------------------------------------------------
*cset* will then adjust the CPUs that are assigned to the "my_cpuset1" set to
only use CPU1 and CPU3.
To rename a cpuset, use the +-n/\--newname+ option. For example:
----------------------------------------------------------
[zuul:cpuset-trunk]# cset set -s my_cpuset1 -n super_set
cset: --> renaming "/cpusets/my_cpuset1" to "super_set"
----------------------------------------------------------
Renames the cpuset called "my_cpuset1" to "super_set".
To destroy a cpuset, use the +-d/\--destroy+ option as follows.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset set -d super_set
cset: --> processing cpuset "super_set", moving 0 tasks to parent "/"...
cset: --> deleting cpuset "/super_set"
cset: done
----------------------------------------------------------
This command destroys the newly created cpuset called "super_set". When a
cpuset is destroyed, all the tasks running in it are moved to the parent
cpuset. The 'root' cpuset, which always exists and always contains all CPUs,
can not be destroyed. You may also give the +\--destroy+ option a list of
cpusets to destroy.
NOTE: The *cset* subcommand creates the cpusets based on a mounted cpuset
filesystem. You do not need to know where that filesystem is mounted,
although it is easy to figure out (by default it's on '/cpusets'). When you
give the +set+ subcommand a name for a new cpuset, it is created wherever the
cpuset filesystem is mounted at.
If you want to create a cpuset hierarchy, then you must give a path to the
*cset* +set+ subcommand. This path will always begin with the 'root' cpuset,
for which the path is '/'. For example.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset set -c 1,3 -s top_set
cset: --> created cpuset "top_set"
[zuul:cpuset-trunk]# cset set -c 3 -s /top_set/sub_set
cset: --> created cpuset "/top_set/sub_set"
----------------------------------------------------------
These commands created two cpusets: 'top_set' and 'sub_set'. The 'top_set'
uses CPU1 and CPU3. It has a subset of 'sub_set' which only uses CPU3. Once
you have created a subset with a path, then if the name is unique, you do not
have to specify the path in order to affect it. If the name is not unique,
then *cset* will complain and ask you to use the path. For example:
----------------------------------------------------------
[zuul:cpuset-trunk]# cset set -c 1,3 -s sub_set
cset: --> modified cpuset "sub_set
----------------------------------------------------------
This command adds CPU1 to the 'sub_set' cpuset for it's use. Note that using
the path in this case is optional.
If you attempt to destroy a cpuset which has sub-cpusets, *cset* will complain
and not do it unless you use the +-r/\--recurse+ and the +\--force+ options.
If you do use +\--force+, then all the tasks running in all subsets of the
deletion target cpuset will be moved to the target's parent cpuset and all
cpusets.
Moving a cpuset from under a certain cpuset to a different location is
currently not implemented and is slated for a later release of *cset*.
Listing Cpusets with Set
~~~~~~~~~~~~~~~~~~~~~~~~
To list cpusets, use the +set+ subcommand with the '-l/\--list' option. For
example:
----------------------------------------------------------
[zuul:cpuset-trunk]# cset set -l
cset:
Name CPUs-X MEMs-X Tasks Subs Path
------------ ---------- - ------- - ----- ---- ----------
root 0-3 y 0 y 320 1 /
one 3 n 0 n 0 1 /one
----------------------------------------------------------
This shows that there is currently one cpuset present called 'one'. (Of course
that there is also the 'root' set, which is always present.) The output shows
that the 'one' cpuset has no tasks running in it. The 'root' cpuset has 320
tasks running. The "-X" for "CPUs" and "MEMs" fields denotes whether the CPUs
and MEMs in the cpusets are marked exclusive to those cpusets. Note that the
'one' cpuset has subsets as indicated by a 1 in the 'Subs' field. You can
specify a cpuset to list with the +set+ subcommand as follows.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset set -l -s one
cset:
Name CPUs-X MEMs-X Tasks Subs Path
------------ ---------- - ------- - ----- ---- ----------
one 3 n 0 n 0 1 /one
two 3 n 0 n 0 1 /one/two
----------------------------------------------------------
This output shows that there is a cpuset called 'two' in cpuset 'one' and it
also has subset. You can also ask for a recursive listing as follows.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset set -l -r
cset:
Name CPUs-X MEMs-X Tasks Subs Path
------------ ---------- - ------- - ----- ---- ----------
root 0-3 y 0 y 320 1 /
one 3 n 0 n 0 1 /one
two 3 n 0 n 0 1 /one/two
three 3 n 0 n 0 0 /one/two/three
----------------------------------------------------------
This command lists all cpusets existing on the system since it asks for a
recursive listing beginning at the 'root' cpuset. Incidentally, should you
need to specify the 'root' cpuset you can use either +root+ or +/+ to specify it
explicitely--just remember that the 'root' cpuset cannot be deleted or modified.
The Proc Subcommand
-------------------
Now that we know how to create, rename and destroy cpusets with the +set+
subcommand, the next step is to manage threads and processes in those
cpusets. The subcommand to do this is called +proc+ and it allows you to exec
processes into a cpuset, move existing tasks around existing cpusets, and list
tasks running in specified cpusets. For the following examples, let us
assume a cpuset setup of two sets as follows:
----------------------------------------------------------
[zuul:cpuset-trunk]# cset set -l
cset:
Name CPUs-X MEMs-X Tasks Subs Path
------------ ---------- - ------- - ----- ---- ----------
root 0-3 y 0 y 309 2 /
two 2 n 0 n 3 0 /two
three 3 n 0 n 10 0 /three
----------------------------------------------------------
Listing Tasks with Proc
~~~~~~~~~~~~~~~~~~~~~~~
Operation of the +proc+ subcommand follows the same model as the +set+
subcommand. For example, to list tasks in a cpuset, you need to use the
+-l/\--list+ option and specify the cpuset by name or, if the name exists
multiple times in the cpuset hierarchy, by path. For example:
----------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -l -s two
cset: "two" cpuset of CPUSPEC(2) with 3 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root 16141 4300 Soth bash
root 16171 16141 Soth bash
root 16703 16171 Roth python ./cset proc -l two
----------------------------------------------------------
This output shows us that the cpuset called 'two' has CPU2 only attached to it
and is running three tasks: two shells and the python command to list it.
Note that cpusets are inherited so that if a process is contained in a cpuset,
then any children it spawns also run within that set. In this case, the
python command to list set 'two' was run from a shell already running in set
'two'. This can be seen by the PPID (parent process ID) of the python
command matching the PID of the shell.
Additionally, the "SPPr" field needs explanation. "SPPr" stands for State,
Policy and Priority. You can see that the initial two tasks are Stopped and
running in timeshare priority, marked as "oth" (for "other"). The last task
is marked as running, "R" and also at timeshare priority, "oth." If any of
these tasks would have been at real time priority, then the policy would be
shown as "f" for FIFO or "r" for round robin, and the priority would be a
number from 1 to 99. See below for an example.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -l -s root | head -7
cset: "root" cpuset of CPUSPEC(0-3) with 309 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root 1 0 Soth init [5]
root 2 0 Soth [kthreadd]
root 3 2 Sf99 [migration/0]
root 4 2 Sf99 [posix_cpu_timer]
----------------------------------------------------------
This output shows the first few tasks in the 'root' cpuset. Note that both
'init' and '[kthread]' are running at timeshare; however, the '[migration/0]'
and '[posix_cpu_timer]' kernel threads are running at real time policy of FIFO
and priority of 99. Incidentally, this output is from a system running the
real time Linux kernel which runs some kernel threads at real time
priorities. And finally, note that you can of course use *cset* as any other
Linux tool and include it in pipelines as in the example above.
Taking a peek into the third cpuset called 'three', we see:
----------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -l -s three
cset: "three" cpuset of CPUSPEC(3) with 10 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
alext 16165 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16169 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16170 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16237 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16491 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16492 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16493 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 17243 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 17244 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 17265 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
----------------------------------------------------------
This output shows that a lot of 'beagled' tasks are running in this cpuset and
it also shows an ellipsis (...) at the end of their listings. If you see this
ellipsis, that means that the command was too long to fit onto an 80 character
screen. To see the entire commandline, use the +-v/\--verbose+ flag, as per
following.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -l -s three -v | head -4
cset: "three" cpuset of CPUSPEC(3) with 10 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
alext 16165 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg --autostarted --indexing-delay 300
----------------------------------------------------------
Execing Tasks with Proc
~~~~~~~~~~~~~~~~~~~~~~~
To exec a task into a cpuset, the +proc+ subcommand needs to be employed with
the +-e/\--exec+ option. Let's exec a shell into the cpuset named 'two' in
our set. First we check to see what is running that set:
----------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -l -s two
cset: "two" cpuset of CPUSPEC(2) with 0 tasks running
[zuul:cpuset-trunk]# cset proc -s two -e bash
cset: --> last message, executed args into cpuset "/two", new pid is: 20955
[zuul:cpuset-trunk]# cset proc -l -s two
cset: "two" cpuset of CPUSPEC(2) with 2 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root 20955 19253 Soth bash
root 20981 20955 Roth python ./cset proc -l two
----------------------------------------------------------
You can see that initially, 'two' had nothing running in it. After the
completion of the second command, we list 'two' again and see that there are
two tasks running: the shell which we execed and the python *cset* command
that is listing the cpuset. The reason for the second task is that the cpuset
property of a running task is inherited by all its children. Since we
executed the listing command from the new shell which was bound to cpuset
'two', the resulting process for the listing is also bound to cpuset 'two'.
Let's test that by just running a new shell with no prefixed *cset* command.
----------------------------------------------------------
[zuul:cpuset-trunk]# bash
[zuul:cpuset-trunk]# cset proc -l -s two
cset: "two" cpuset of CPUSPEC(2) with 3 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root 20955 19253 Soth bash
root 21118 20955 Soth bash
root 21147 21118 Roth python ./cset proc -l two
----------------------------------------------------------
Here again we see that the second shell, PID 21118, has a parent PID of 20955
which is the first shell. Both shells as well as the listing command are
running in the 'two' cpuset.
NOTE: *cset* follows the tradition of separating the tool options from the
command to be execed options with a double dash (+\--+). This is not shown in
this simple example, but if the command you want to exec also takes options,
separate them with the double dash like so: +# cset proc -s myset -e mycommand
\-- -v+ The +-v+ will be passed to +mycommand+, and not to *cset*.
TIP: Execing a shell into a cpuset is a useful way to experiment with running
tasks in that cpuset since all children of the shell will also run in the same
cpuset.
Finally, if you misspell the command to be execed, the result may be
puzzling. For example:
----------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -s two -e blah-blah
cset: --> last message, executed args into cpuset "/two", new pid is: 21655
cset: **> [Errno 2] No such file or directory
----------------------------------------------------------
The result is no new process even though a new PID is output. The reason for
the message is of course that the *cset* process forked in preparation for
exec, but the command +blah-blah+ was not found in order to exec it.
Moving Tasks with Proc
~~~~~~~~~~~~~~~~~~~~~~
Although the ability to exec a task into a cpuset is fundamental, you will
most likely be moving tasks between cpusets more often. Moving tasks is
accomplished with the +-m/\--move+ and +-p/\--pid+ options to the +proc+
subcommand of *cset*. The move option tells the +proc+ subcommand that a task
move is requested. The +-p/\--pid+ option takes an argument called a PIDSPEC
(PID Specification). The PIDSPEC defines which tasks get operated on.
The PIDSPEC can be a single process ID, a list of process IDs separated by
commas, and a list of process ID ranges also separated by commas. For
example:
+\--move --pid 1234+::
This PIDSPEC argument specifies that task 1234 be moved.
+\--move --pid 1234,42,1934,15000,15001,15002+::
This PIDSPEC argument specifies that this list of tasks only be moved.
+\--move --pid 5000,5100,6010-7000,9232+::
This PIDSPEC argument specifies that tasks 5000,5100 and 9232 be moved along
with any existing task that is in the range 6010 through 7000 inclusive.
NOTE: A range in a PIDSPEC does not have to have running tasks for every
number in that range. In fact, it is not even an error if there are no tasks
running in that range; none will be moved in that case. The range simply
specifies to act on any tasks that have a PID or TID that is within that
range.
In the following example, we move the current shell into the cpuset named
'two' with a range PIDSPEC and back out to the 'root' cpuset with the bash
variable for the current PID.
-----------------------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -l -s two
cset: "two" cpuset of CPUSPEC(2) with 0 tasks running
[zuul:cpuset-trunk]# echo $$
19253
[zuul:cpuset-trunk]# cset proc -m -p 19250-19260 -t two
cset: moving following pidspec: 19253
cset: moving 1 userspace tasks to /two
cset: done
[zuul:cpuset-trunk]# cset proc -l -s two
cset: "two" cpuset of CPUSPEC(2) with 2 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root 19253 16447 Roth bash
root 29456 19253 Roth python ./cset proc -l -s two
[zuul:cpuset-trunk]# cset proc -m -p $$ -t root
cset: moving following pidspec: 19253
cset: moving 1 userspace tasks to /
cset: done
[zuul:cpuset-trunk]# cset proc -l -s two
cset: "two" cpuset of CPUSPEC(2) with 0 tasks running
-----------------------------------------------------------------------
Use of the appropriate PIDSPEC can thus be handy to move tasks and groups of
tasks. Additionally, there is one more option that can help with
multi-threaded processes, and that is the +\--threads+ flag. If this flag is
present in a +proc+ move command with a PIDSPEC and if any of the task IDs in
the PIDSPEC belongs to a thread in a process container, then *all* the sibling
threads in that process container will also get moved. This flag provides an
easy mechanism to move all threads of a process by simply specifying one
thread in that process. In the following example, we move all the threads
running in cpuset 'three' to cpuset 'two' by using the +\--threads+ flag.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset set two three
cset:
Name CPUs-X MEMs-X Tasks Subs Path
------------ ---------- - ------- - ----- ---- ----------
two 2 n 0 n 0 0 /two
three 3 n 0 n 10 0 /three
[zuul:cpuset-trunk]# cset proc -l -s three
cset: "three" cpuset of CPUSPEC(3) with 10 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
alext 16165 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16169 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16170 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16237 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16491 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16492 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16493 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 17243 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 17244 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 27133 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
[zuul:cpuset-trunk]# cset proc -m -p 16165 --threads -t two
cset: moving following pidspec: 16491,16493,16492,16170,16165,16169,27133,17244,17243,16237
cset: moving 10 userspace tasks to /two
[==================================================]%
cset: done
[zuul:cpuset-trunk]# cset set two three
cset:
Name CPUs-X MEMs-X Tasks Subs Path
------------ ---------- - ------- - ----- ---- ----------
two 2 n 0 n 10 0 /two
three 3 n 0 n 0 0 /three
----------------------------------------------------------
Moving All Tasks from one Cpuset to Another
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There is a special case for moving all tasks currently running in one cpuset
to another. This can be a common use case, and when you need to do it,
specifying a PIDSPEC with +-p+ is not necessary 'so long as' you use the
+-f/\--fromset+ *and* the +-t/\--toset+ options.
In the following example, we move all 10 +beagled+ threads back to cpuset
'three' with this method.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -l two three
cset: "two" cpuset of CPUSPEC(2) with 10 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
alext 16165 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16169 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16170 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16237 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16491 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16492 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 16493 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 17243 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 17244 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
alext 27133 1 Soth beagled /usr/lib64/beagle/BeagleDaemon.exe --bg -...
cset: "three" cpuset of CPUSPEC(3) with 0 tasks running
[zuul:cpuset-trunk]# cset proc -m -f two -t three
cset: moving all tasks from two to /three
cset: moving 10 userspace tasks to /three
[==================================================]%
cset: done
[zuul:cpuset-trunk]# cset set two three
cset:
Name CPUs-X MEMs-X Tasks Subs Path
------------ ---------- - ------- - ----- ---- ----------
two 2 n 0 n 0 0 /two
three 3 n 0 n 10 0 /three
----------------------------------------------------------
Moving Kernel Threads with Proc
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Kernel threads are special and *cset* detects tasks that are kernel threads
and will refuse to move them unless you also add a +-k/\--kthread+ option to
your +proc+ move command. Even if you include +-k+, *cset* will 'still'
refuse to move the kernel thread if they are bound to specific CPUs. The
reason for this is system protection.
A number of kernel threads, especially on the real time Linux kernel, are
bound to specific CPUs and depend on per-CPU kernel variables. If you move
these threads to a different CPU than what they are bound to, you risk at best
that the system will become horribly slow, and at worst a system hang. If you
still insist to move those threads (after all, *cset* needs to give the
knowledgeable user access to the keys), then you need to use the +--force+
option additionally.
WARNING: Overriding a task move command with +--force+ can have dire
consequences for the system. Please be sure of the command before you force
it.
In the following example, we move all unbound kernel threads running in the
'root' cpuset to the cpuset named 'two' by using the +-k+ option.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -k -f root -t two
cset: moving all kernel threads from / to /two
cset: moving 70 kernel threads to: /two
cset: --> not moving 76 threads (not unbound, use --force)
[==================================================]%
cset: done
----------------------------------------------------------
You will note that we used the fromset->toset facility of the +proc+
subcommand and we only specified the +-k+ option (not the +-m+ option). This
has the effect of moving all kernel threads only.
Note that only 70 actual kernel threads were moved and 76 were not. The
reason that 76 kernel threads were not moved was because they are bound to
specific CPUs. Now, let's move those kernel threads back to 'root'.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -k -f two -t root
cset: moving all kernel threads from /two to /
cset: ** no task matched move criteria
cset: **> kernel tasks are bound, use --force if ok
[zuul:cpuset-trunk]# cset set -l -s two
cset:
Name CPUs-X MEMs-X Tasks Subs Path
------------ ---------- - ------- - ----- ---- ----------
two 2 n 0 n 70 0 /two
----------------------------------------------------------
Ah! What's this? *Cset* refused to move the kernel threads back to 'root'
because it says that they are "bound." Let's check this with the Linux
taskset command.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -l -s two | head -5
cset: "two" cpuset of CPUSPEC(2) with 70 tasks running
USER PID PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root 2 0 Soth [kthreadd]
root 55 2 Soth [khelper]
[zuul:cpuset-trunk]# taskset -p 2
pid 2's current affinity mask: 4
[zuul:cpuset-trunk]# cset set -l -s two
cset:
Name CPUs-X MEMs-X Tasks Subs Path
------------ ---------- - ------- - ----- ---- ----------
two 2 n 0 n 70 0 /two
----------------------------------------------------------
Of course, since the cpuset named 'two' only has CPU2 assigned to it, once we
moved the unbound kernel threads from 'root' to 'two', their affinity masks
got automatically changed to only use CPU2. This is evident from the
+taskset+ output which is a hex value. To really move these threads back to
'root', we need to force the move as follows.
----------------------------------------------------------
[zuul:cpuset-trunk]# cset proc -k -f two -t root --force
cset: moving all kernel threads from /two to /
cset: moving 70 kernel threads to: /
[==================================================]%
cset: done
----------------------------------------------------------
Destroying Tasks
~~~~~~~~~~~~~~~~
There actually is no *cset* subcommand or option to destroy tasks--it's not
really needed. Tasks exist and are accessible on the system as normal, even
if they happen to be running in one cpuset or another. To destroy tasks, use
the usual ^C method or by using the *kill(1)* command.
Implementing "Shielding" with Set and Proc
------------------------------------------
With the preceding material on the +set+ and +proc+ subcommands, we now have
the background to implement the basic shielding model, just like the +shield+
subcommand.
One may pose the question as to why we want to do this, especially since
+shield+ already does it? The answer is that sometimes one needs more
functionality than +shield+ has to implement one's shielding strategy. In
those case you need to first stop using +shield+ since that subcommand will
interfere with the further application of +set+ and +proc+; however, you will
still need to implement the functionality of +shield+ in order to implement