/
crun.1.md
710 lines (494 loc) · 19.2 KB
/
crun.1.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
crun 1 "User Commands"
==================================================
# NAME
crun - a fast and lightweight OCI runtime
# SYNOPSIS
crun [global options] command [command options] [arguments...]
# DESCRIPTION
crun is a command line program for running Linux containers that
follow the Open Container Initiative (OCI) format.
# COMMANDS
**create**
Create a container. The runtime detaches from the container process
once the container environment is created. It is necessary to
successively use `start` for starting the container.
**delete**
Remove definition for a container.
**exec**
Exec a command in a running container.
**list**
List known containers.
**kill**
Send the specified signal to the container init process. If no signal
is specified, SIGTERM is used.
**ps**
Show the processes running in a container.
**run**
Create and immediately start a container.
**spec**
Generate a configuration file.
**start**
Start a container that was previously created. A container cannot be
started multiple times.
**state**
Output the state of a container.
**pause**
Pause all the processes in the container.
**resume**
Resume the processes in the container.
**update**
Update container resource constraints.
**checkpoint**
Checkpoint a running container using CRIU
**restore**
Restore a container from a checkpoint
# STATE
By default, when running as root user, crun saves its state under the
**/run/crun** directory. As unprivileged user, instead the
*XDG_RUNTIME_DIR* environment variable is honored, and the directory
**$XDG_RUNTIME_DIR/crun** is used. The global option **--root**
overrides this setting.
# GLOBAL OPTIONS
**--debug**
Produce verbose output.
**--log**=_LOG-DESTINATION_
Define the destination for the error and warning messages generated by
crun.
If the error happens late in the container init process, when crun
already stopped watching it, then it will be printed to the container
stderr.
It is specified in the form *BACKEND:SPECIFIER*.
These following backends are supported:
- file:PATH
- journald:IDENTIFIER
- syslog:IDENTIFIER
If no backend is specified, then *file:* is used by default.
**--log-format**=_FORMAT_
Define the format of the log messages. It can either be **text**, or
**json**. The default is **text**.
**--no-pivot**
Use `chroot(2)` instead of `pivot_root(2)` when creating the container.
This option is not safe, and should be avoided.
**--root**=_DIR_
Defines where to store the state for crun containers.
**--systemd-cgroup**
Use systemd for configuring cgroups. If not specified, the cgroup is
created directly using the cgroupfs backend.
**--cgroup-manager**=_MANAGER_
Specify what cgroup manager must be used. Permitted values are **cgroupfs**,
**systemd** and **disabled**.
**-?**, **--help**
Print a help list.
**--usage**
Print a short usage message.
**-V**, **--version**
Print program version
## CREATE OPTIONS
crun [global options] create [options] CONTAINER
**--bundle**=_PATH_
Path to the OCI bundle, by default it is the current directory.
**--config**=_FILE_
Override the configuration file to use. The default value is **config.json**.
**--console-socket**=_SOCKET_
Path to a UNIX socket that will receive the ptmx end of the tty for
the container.
**--no-new-keyring**
Keep the same session key
**--preserve-fds**=_N_
Additional number of FDs to pass into the container.
**--pid-file**=_PATH_
Path to the file that will contain the container process PID.
## RUN OPTIONS
crun [global options] run [options] CONTAINER
**--bundle**=_BUNDLE_
Path to the OCI bundle, by default it is the current directory.
**--config**=_FILE_
Override the configuration file to use. The default value is **config.json**.
**--console-socket**=_SOCKET_
Path to a UNIX socket that will receive the ptmx end of the tty for
the container.
**--no-new-keyring**
Keep the same session key.
**--preserve-fds**=_N_
Additional number of FDs to pass into the container.
**--pid-file**=_PATH_
Path to the file that will contain the container process PID.
**--detach**
Detach the container process from the current session.
## DELETE OPTIONS
crun [global options] delete [options] CONTAINER
**--force**
Delete the container even if it is still running.
**--regex**=_REGEX_
Delete all the containers that satisfy the specified regex.
## EXEC OPTIONS
crun [global options] exec [options] CONTAINER CMD
**--apparmor**=_PROFILE_
Set the apparmor profile for the process.
**--console-socket**=_SOCKET_
Path to a UNIX socket that will receive the ptmx end of the tty for
the container.
**--cwd**=_PATH_
Set the working directory for the process to PATH.
**--cap**=_CAP_
Specify an additional capability to add to the process.
**--detach**
Detach the container process from the current session.
**--cgroup**=_PATH_
Specify a sub-cgroup path inside the container cgroup. The path must
already exist in the container cgroup.
**--env**=_ENV_
Specify an environment variable.
**--no-new-privs**
Set the no new privileges value for the process.
**--preserve-fds**=_N_
Additional number of FDs to pass into the container.
**--process**=_FILE_
Path to a file containing the process JSON configuration.
**--process-label**=_VALUE_
Set the asm process label for the process commonly used with selinux.
**--pid-file**=_PATH_
Path to the file that will contain the new process PID.
**-t** **--tty**
Allocate a pseudo TTY.
**-u _USERSPEC_ **--user**=_USERSPEC_
Specify the user in the form UID[:GID].
## LIST OPTIONS
crun [global options] list [options]
**-q** **--quiet**
Show only the container ID.
## KILL OPTIONS
crun [global options] kill [options] CONTAINER SIGNAL
**--all**
Kill all the processes in the container.
**--regex**=_REGEX_
Kill all the containers that satisfy the specified regex.
## PS OPTIONS
crun [global options] ps [options]
**--format**=_FORMAT_
Specify the output format. It must be either `table` or `json`.
By default `table` is used.
## SPEC OPTIONS
crun [global options] spec [options]
**-b** _DIR_ **--bundle**=_DIR_
Path to the root of the bundle dir (default ".").
**--rootless**
Generate a config.json file that is usable by an unprivileged user.
## UPDATE OPTIONS
crun [global options] update [options] CONTAINER
**--blkio-weight**=_VALUE_
Specifies per cgroup weight.
**--cpu-period**=_VALUE_
CPU CFS period to be used for hardcapping.
**--cpu-quota**=_VALUE_
CPU CFS hardcap limit.
**--cpu-rt-period**=_VALUE_
CPU realtime period to be used for hardcapping.
**--cpu-rt-runtime**=_VALUE_
CPU realtime hardcap limit.
**--cpu-share**=_VALUE_
CPU shares.
**--cpuset-cpus**=_VALUE_
CPU(s) to use.
**--cpuset-mems**=_VALUE_
Memory node(s) to use.
**--kernel-memory**=_VALUE_
Kernel memory limit.
**--kernel-memory-tcp**=_VALUE_
Kernel memory limit for TCP buffer.
**--memory**=_VALUE_
Memory limit.
**--memory-reservation**=_VALUE_
Memory reservation or soft_limit.
**--memory-swap**=_VALUE_
Total memory usage.
**--pids-limit**=_VALUE_
Maximum number of pids allowed in the container.
**-r**, **--resources**=_FILE_
Path to the file containing the resources to update.
## CHECKPOINT OPTIONS
crun [global options] checkpoint [options] CONTAINER
**--image-path**=_DIR_
Path for saving CRIU image files
**--work-path**=_DIR_
Path for saving work files and logs
**--leave-running**
Leave the process running after checkpointing
**--tcp-established**
Allow open TCP connections
**--ext-unix-sk**
Allow external UNIX sockets
**--shell-job**
Allow shell jobs
**--pre-dump**
Only checkpoint the container's memory without stopping the container.
It is not possible to restore a container from a pre-dump. A pre-dump
always needs a final checkpoint (without **--pre-dump**). It is possible
to make as many pre-dumps as necessary. For a second pre-dump or for
a final checkpoint it is necessary to use **--parent-path** to point
crun (and thus CRIU) to the pre-dump.
**--parent-path**=_DIR_
Doing multiple pre-dumps or the final checkpoint after one or multiple
pre-dumps requires that crun (and thus CRIU) knows the location of
the pre-dump. It is important to use a relative path from the actual
checkpoint directory specified via **--image-path**. It will fail
if an absolute path is used.
**--manage-cgroups-mode**=_MODE_
Specify which CRIU manage cgroup mode should be used. Permitted values are
**soft**, **ignore**, **full** or **strict**. Default is **soft**.
## RESTORE OPTIONS
crun [global options] restore [options] CONTAINER
**-b DIR** **--bundle**=_DIR_
Container bundle directory (default ".")
**--image-path**=_DIR_
Path for saving CRIU image files
**--work-path**=_DIR_
Path for saving work files and logs
**--tcp-established**
Allow open TCP connections
**--ext-unix**
Allow external UNIX sockets
**--shell-job**
Allow shell jobs
**--detach**
Detach from the container's process
**--pid-file**=_FILE_
Where to write the PID of the container
**--manage-cgroups-mode**=_MODE_
Specify which CRIU manage cgroup mode should be used. Permitted values are
**soft**, **ignore**, **full** or **strict**. Default is **soft**.
# Extensions to OCI
## `run.oci.mount_context_type=context`
Set the mount context type on volumes mounted with SELinux labels.
Valid context types are:
context (default)
fscontext
defcontext
rootcontext
More information on how the context mount flags works see the `mount(8)` man page.
## `run.oci.seccomp.receiver=PATH`
If the annotation `run.oci.seccomp.receiver=PATH` is specified, the
seccomp listener is sent to the UNIX socket listening on the specified
path. It can also set with the `RUN_OCI_SECCOMP_RECEIVER` environment variable.
It is an experimental feature, and the annotation will be removed once
it is supported in the OCI runtime specs. It must be an absolute path.
## `run.oci.seccomp.plugins=PATH`
If the annotation `run.oci.seccomp.plugins=PLUGIN1[:PLUGIN2]...` is specified, the
seccomp listener fd is handled through the specified plugins. The
plugin must either be an absolute path or a file name that is looked
up by `dlopen(3)`. More information on how the lookup is performed
are available on the `ld.so(8)` man page.
## `run.oci.seccomp_fail_unknown_syscall=1`
If the annotation `run.oci.seccomp_fail_unknown_syscall` is present, then crun
will fail when an unknown syscall is encountered in the seccomp configuration.
## `run.oci.seccomp_bpf_data=PATH`
If the annotation `run.oci.seccomp_bpf_data` is present, then crun
ignores the seccomp section in the OCI configuration file and use the specified data
as the raw data to the `seccomp(SECCOMP_SET_MODE_FILTER)` syscall.
The data must be encoded in base64.
It is an experimental feature, and the annotation will be removed once
it is supported in the OCI runtime specs.
## `run.oci.keep_original_groups=1`
If the annotation `run.oci.keep_original_groups` is present, then crun
will skip the `setgroups` syscall that is used to either set the
additional groups specified in the OCI configuration, or to reset the
list of additional groups if none is specified.
## `run.oci.pidfd_receiver=PATH`
It is an experimental feature and will be removed once the feature is in the
OCI runtime specs.
If present, specify the path to the UNIX socket that will receive the
pidfd for the container process.
## `run.oci.systemd.force_cgroup_v1=/PATH`
If the annotation `run.oci.systemd.force_cgroup_v1=/PATH` is present, then crun
will override the specified mount point `/PATH` with a cgroup v1 mount
made of a single hierarchy `none,name=systemd`.
It is useful to run on a cgroup v2 system containers using older
versions of systemd that lack support for cgroup v2.
**Note**: Your container host has to have the cgroup v1 mount already present, otherwise
this will not work. If you want to run the container rootless, the user it runs under
has to have permissions to this mountpoint.
For example, as root:
```
mkdir /sys/fs/cgroup/systemd
mount cgroup -t cgroup /sys/fs/cgroup/systemd -o none,name=systemd,xattr
chown -R the_user.the_user /sys/fs/cgroup/systemd
```
## `run.oci.systemd.subgroup=SUBGROUP`
Override the name for the systemd sub cgroup created under the systemd
scope, so the final cgroup will be like:
```
/sys/fs/cgroup/$PATH/$SUBGROUP
```
When it is set to the empty string, a sub cgroup is not created.
If not specified, it defaults to `container` on cgroup v2, and to `""`
on cgroup v1.
e.g.
```
/sys/fs/cgroup//system.slice/foo-352700.scope/container
```
## `run.oci.delegate-cgroup=DELEGATED-CGROUP`
If the `run.oci.systemd.subgroup` annotation is specified, yet another
sub-cgroup is created and the container process is moved here.
If a cgroup namespace is used, the cgroup namespace is created before
moving the container to the delegated cgroup.
```
/sys/fs/cgroup/$PATH/$SUBGROUP/$DELEGATED-CGROUP
```
The runtime doesn't apply any limit to the `$DELEGATED-CGROUP`
sub-cgroup, the runtime uses only `$PATH/$SUBGROUP`.
The container payload fully manages `$DELEGATE-CGROUP`, the limits
applied to `$PATH/$SUBGROUP` still applies to `$DELEGATE-CGROUP`.
Since cgroup delegation is not safe on cgroup v1, this option is
supported only on cgroup v2.
## `run.oci.hooks.stdout=FILE`
If the annotation `run.oci.hooks.stdout` is present, then crun
will open the specified file and use it as the stdout for the hook
processes. The file is opened in append mode and it is created if it
doesn't already exist.
## `run.oci.hooks.stderr=FILE`
If the annotation `run.oci.hooks.stderr` is present, then crun
will open the specified file and use it as the stderr for the hook
processes. The file is opened in append mode and it is created if it
doesn't already exist.
## `run.oci.handler=HANDLER`
It is an experimental feature.
If specified, run the specified handler for execing the container.
The only supported values are `krun` and `wasm`.
- `krun`: When `krun` is specified, the `libkrun.so` shared object is loaded
and it is used to launch the container using libkrun.
- `wasm`: If specified, run the wasm handler for container. Allows running wasm
workload natively. Accepts a `.wasm` binary as input and if `.wat` is
provided it will be automatically compiled into a wasm module. Stdout of
wasm module is relayed back via crun.
## tmpcopyup mount options
If the `tmpcopyup` option is specified for a tmpfs, then the path that
is shadowed by the tmpfs mount is recursively copied up to the tmpfs
itself.
## copy-symlink mount options
If the `copy-symlink` option is specified, if the source of a bind
mount is a symlink, the symlink is recreated at the specified
destination instead of attempting a mount that would resolve the
symlink itself. If the destination already exists and it is not a
symlink with the expected content, crun will return an error.
## r$FLAG mount options
If a `r$FLAG` mount option is specified then the flag `$FLAG` is set
recursively for each children mount.
These flags are supported:
- "rro"
- "rrw"
- "rsuid"
- "rnosuid"
- "rdev"
- "rnodev"
- "rexec"
- "rnoexec"
- "rsync"
- "rasync"
- "rdirsync"
- "rmand"
- "rnomand"
- "ratime"
- "rnoatime"
- "rdiratime"
- "rnodiratime"
- "rrelatime"
- "rnorelatime"
- "rstrictatime"
- "rnostrictatime"
## idmap mount options
If the `idmap` option is specified then the mount is ID mapped using
the container target user namespace. This is an experimental feature
and can change at any time without notice.
The `idmap` option supports a custom mapping that can be different
than the user namespace used by the container.
The mapping can be specified after the `idmap` option like:
`idmap=uids=0-1-10#10-11-10;gids=0-100-10`.
For each triplet, the first value is the start of the backing
file system IDs that are mapped to the second value on the host. The
length of this mapping is given in the third value.
Multiple ranges are separated with `#`.
These values are written to the `/proc/$PID/uid_map` and
`/proc/$PID/gid_map` files to create the user namespace for the
idmapped mount.
The only two options that are currently supported after `idmap` are
`uids` and `gids`.
When a custom mapping is specified, a new user namespace is created
for the idmapped mount.
If no option is specified, then the container user namespace is used.
If the specified mapping is prepended with a '@' then the mapping is
considered relative to the container user namespace. The host ID for
the mapping is changed to account for the relative position of the
container user in the container user namespace.
For example, the mapping: `uids=@1-3-10`, given a configuration like
```
"uidMappings": [
{
"containerID": 0,
"hostID": 0,
"size": 1
},
{
"containerID": 1,
"hostID": 2,
"size": 1000
}
]
```
will be converted to the absolute value `uids=1-4-10`, where 4 is
calculated by adding 3 (container ID in the `uids=` mapping) and 1
(`hostID - containerID` for the user namespace mapping where
`containerID = 1` is found).
The current implementation doesn't take into account multiple
user namespace ranges, so it is the caller's responsibility to split a
mapping if it overlaps multiple ranges in the user namespace. In such
a case, there won't be any error reported.
## Automatically create user namespace
When running as user different than root, an user namespace is
automatically created even if it is not specified in the config file.
The current user is mapped to the ID 0 in the container, and any
additional id specified in the files `/etc/subuid` and `/etc/subgid`
is automatically added starting with ID 1.
# CGROUP v2
**Note**: cgroup v2 does not yet support control of realtime processes and
the cpu controller can only be enabled when all RT processes are in the root
cgroup. This will make crun fail while running alongside RT processes.
If the cgroup configuration found is for cgroup v1, crun attempts a
conversion when running on a cgroup v2 system.
These are the OCI resources currently supported with cgroup v2 and how
they are converted when needed from the cgroup v1 configuration.
## Memory controller
| OCI (x) | cgroup 2 value (y) | conversion | comment |
|---|---|---|---|
| limit | memory.max | y = x ||
| swap | memory.swap.max | y = x - memory_limit | the swap limit on cgroup v1 includes the memory usage too |
| reservation | memory.low | y = x ||
## PIDs controller
| OCI (x) | cgroup 2 value (y) | conversion | comment |
|---|---|---|---|
| limit | pids.max | y = x ||
## CPU controller
| OCI (x) | cgroup 2 value (y) | conversion | comment |
|---|---|---|---|
| shares | cpu.weight | y = (1 + ((x - 2) * 9999) / 262142) | convert from [2-262144] to [1-10000]|
| period | cpu.max | y = x| period and quota are written together|
| quota | cpu.max | y = x| period and quota are written together|
## blkio controller
| OCI (x) | cgroup 2 value (y) | conversion | comment |
|---|---|---|---|
| weight | io.bfq.weight | y = x ||
| weight_device | io.bfq.weight | y = x ||
| weight | io.weight (fallback) | y = 1 + (x-10)*9999/990 | convert linearly from [10-1000] to [1-10000] |
| weight_device | io.weight (fallback) | y = 1 + (x-10)*9999/990 | convert linearly from [10-1000] to [1-10000] |
| rbps | io.max | y=x ||
| wbps | io.max | y=x ||
| riops | io.max |y=x ||
| wiops | io.max |y=x ||
## cpuset controller
| OCI (x) | cgroup 2 value (y) | conversion | comment |
|---|---|---|---|
| cpus | cpuset.cpus | y = x ||
| mems | cpuset.mems | y = x ||
## hugetlb controller
| OCI (x) | cgroup 2 value (y) | conversion | comment |
|---|---|---|---|
| <PAGE_SIZE>.limit_in_bytes | hugetlb.<PAGE_SIZE>.max | y = x ||