IO Chaos Document
This document helps you to build IO chaos experiments.
IO chaos allows you to simulate file system faults such as IO delay,
read/write errors, etc. It can inject delay and errno when you use the IO system calls such as open, read and write.
Note: IO Chaos can only be used if the relevant labels and annotations are set before the application is created. More info refer here
Prerequisites
Admission Controller
IO chaos needs to inject a sidecar container to user pods and the sidecar container can be added to applicable Kubernetes pods using a mutating webhook admission controller provided by Chaos Mesh.
While admission controllers are enabled by default, some Kubernetes distributions may disable them. If this is the case, follow the instructions to turn on admission controllers.
ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks are required by IO chaos.
Data directory
The data directory of the application in the target pod should be a subdirectory of PersistentVolumes.
example:
# the config about tikv PersistentVolumes
volumeMounts:
- name: datadir
mountPath: /var/lib/tikv
# the arguments to start tikv
ARGS="--pd=${CLUSTER_NAME}-pd:2379 \
--advertise-addr=${HOSTNAME}.${HEADLESS_SERVICE_NAME}.${NAMESPACE}.svc:20160 \
--addr=0.0.0.0:20160 \
--data-dir=/var/lib/tikv/data \ # data directory
--capacity=${CAPACITY} \
--config=/etc/tikv/tikv.tomlNode: The default data directory of TiKV is not a subdirectory of
PersistentVolumes. If your application is TiDB cluster, you need to modify it at _start_tikv.sh.tpl. PD has the same issue with TiKV, you need to modity the data directory of pd at _start_pd.sh.tpl.
Usage
Configure a ConfigMap
Chaos Mesh uses sidecar container to inject IO chaos, to fulfill this chaos you need to configure this sidecar container using a ConfigMap You can refer this document to define a specify ConfigMap for your application before starting your chaos experiment.
You can apply the ConfigMap defined for your application to Kubernetes cluster by using this command:
kubectl apply -f app-configmap.yaml # app-configmap.yaml is the ConfigMap file Define the Chaos YAML file
Below is a sample YAML file of IO chaos:
apiVersion: pingcap.com/v1alpha1
kind: IoChaos
metadata:
name: io-delay-example
namespace: chaos-testing
spec:
action: mixed
mode: one
duration: "400s"
configName: "chaosfs-tikv"
path: ""
selector:
namespaces:
- tidb-cluster-demo
labelSelectors:
"app.kubernetes.io/component": "tikv"
layer: "fs"
percent: "50"
delay: "1ms"
scheduler:
cron: "@every 10m" For more sample files, see examples. You can edit them as needed.
Description:
-
selector: is used to select pods that are used to inject chaos actions.
-
action: represents the IO chaos actions. Currently the delay, errno, and mixed actions are supported. You can go to IO chaos available actions for more details.
-
mode: defines the mode to run chaos actions. Supported mode:
one/all/fixed/fixed-percent/random-max-percent. -
duration: represents the duration of a chaos action. The duration might be a string with the signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as
"300ms","-1.5h"or"2h45m". -
delay: defines the value of IO chaos action delay. The duration might be a string with the signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as
"300ms","-1.5h"or”2h45m”. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", and "h". IfDelayis empty, the operator will generate a value for it randomly. -
errno: defines the error code that is returned by an IO action. It and the errno defined by Linux system are consistent. It is an int32 string like
"2","2"meansNo such file or directory. This field need to be set when you choose anerrnoormixedaction. Iferrnois empty, the operator will randomly generate an error code for it. See the common Linux system errors for more Linux system error codes. -
percent: defines the percentage of injection errors and provides a number from 0-100. The default value is
100. -
path: defines the path of files for injecting IO chaos actions. It should be a regular expression for the path you want to inject errno or delay. If the path is
""or not defined, IO chaos actions will be injected into all files. -
methods: defines the IO methods for injecting IO chaos actions. It’s an array of string, which sets the IO syscalls such as
openandread. See the available methods for more details. -
addr: defines the sidecar HTTP server address for a sidecar container, such as
":8080". -
configName: defines the config name which is used to inject chaos action into pods. You can refer to examples/tikv-configmap.yaml to define your configuration.
-
layer: represents the layer of the IO action. Supported value:
fs(by default).
Create a chaos experiment
Before the application created, you need to make admission-webhook enable by label add an annotation to the application namespace:
admission-webhook.pingcap.com/init-request:chaosfs-tikvYou can use the following commands to set labels and annotations of the application namespace:
# If the application namespace does not exist. you can exec this command to create one,
# otherwise ignore this command.
kubectl create ns app-ns # "app-ns" is the application namespace
# enable admission-webhook
kubectl label ns app-ns admission-webhook=enabled
# set annotation
kubectl annotate ns app-ns admission-webhook.pingcap.com/init-request=chaosfs-tikv
# create your application
...Then, you can start your application and define YAML file to start your chaos experiment.
Start a chaos experiment
Assume that you are using examples/io-mixed-example.yaml, you can run the following command to create a chaos experiment:
kubectl apply -f examples/io-mixed-example.yamlIO chaos available actions
IO chaos currently supports the following actions:
- delay: IO delay action. You can specify the latency before the IO operation returns a result.
- errno: IO errno action. In this mode, read/write IO operations will return an error.
- mixed: Both delay and errno actions.
delay
If you are using the delay mode, you can edit spec as below:
spec:
action: delay
delay: "1ms"If delay is not specified, it will be generated randomly on runtime.
errno
If you are using the errno mode, you can edit spec as below:
spec:
action: errno
errno: "32"If errno is not specified, it will be generated randomly on runtime.
mixed
If you are using the mixed mode, you can edit spec as below:
spec:
action: mixed
delay: "1ms"
errno: "32"The mix mode defines the delay and errno actions in one spec.
Common Linux system errors
1: Operation not permitted2: No such file or directory5: I/O error6: No such device or address12: Out of memory16: Device or resource busy17: File exists20: Not a directory22: Invalid argument24: Too many open files28: No space left on device
The number represents the errno the Linux system error. More Linux system errors refer to Errors: Linux System Errors.
Available methods
Available methods are as below:
openreadwritemkdirrmdiropendirfsyncflushreleasetruncategetattrchownchmodutimensallocategetlksetlksetlkwstatfsreadlinksymlinkcreateaccesslinkmknodrenameunlinkgetxattrlistxattrremovexattrsetxattr