blockchaos: impl blockchaos in chaosdaemon #2907

YangKeao · 2022-02-17T10:40:04Z

What problem does this PR solve?

Implement blockchaos in chaosdaemon

Test Steps

This functions is relatively hard to automatically test. I have tested manually with the following steps in a centos virtual machine:

Install chaos-driver
Start up a k3s kubernetes cluster.
Install local static provisioner https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner
Create a Pod and PVC

Create the BlockChaos with following spec:

 apiVersion: chaos-mesh.org/v1alpha1
 kind: BlockChaos
 metadata:
   name: disk-example
 spec:
   selector:
     labelSelectors:
       app: disk-example
   mode: all
   volumeName: disk-example
   action: delay
   delay:
     latency: 1s

kubectl exec to enter the pod, and do some operation on the disk.

Detailed Steps

These steps can be concluded in the following scripts, but please execute them one by one.

Bootstraping virtual machine

Firstly we need a virtual machine to test this feature, I used vagrant to help me setup the machine. In an empty directory:

vagrant init centos/7
vagrant up --provider virtualbox

vagrant ssh

Then update the kernel and kernel-devel for virtual machine

sudo yum update -y
sudo yum install kernel-devel -y
sudo reboot

You'll also need to install docker/go/python3 to build chaos-mesh

sudo yum install -y yum-utils python3 gcc
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo usermod -aG docker vagrant
sudo systemctl enable docker

wget https://go.dev/dl/go1.18.3.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.18.3.linux-amd64.tar.gz

sudo reboot

Install chaos driver

Then, inside the virtual machine, we'll need to build the chaos-driver

# clone chaos-driver
git clone https://github.com/chaos-mesh/chaos-driver.git
cd chaos-driver
make all

sudo insmod ./driver/chaos_driver.ko

Now, you have installed the chaos-driver. In the dmesg you can see:

[ 1552.172408] chaos_driver is loading 
[ 1552.172539] io scheduler ioem-mq registered
[ 1552.172541] io scheduler ioem registered

Install k3s and helm

curl -sfL https://get.k3s.io | sh -
# after some time
sudo chmod 666 /etc/rancher/k3s/k3s.yaml
k3s kubectl get pods -n kube-system

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

You can see every pod is running or completed.

Build and install chaos mesh

git clone https://github.com/YangKeao/chaos-mesh.git
cd chaos-mesh
git checkout --track origin/impl-blockchaos-chaosdaemon

make image

docker image save ghcr.io/chaos-mesh/chaos-mesh:latest > ./chaos-mesh.tar
docker image save ghcr.io/chaos-mesh/chaos-daemon:latest > ./chaos-daemon.tar

sudo /usr/local/bin/k3s ctr images import ./chaos-mesh.tar
sudo /usr/local/bin/k3s ctr images import ./chaos-daemon.tar

k3s kubectl create ns chaos-testing
KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm install chaos-mesh ./helm/chaos-mesh -n chaos-testing --set dashboard.securityMode=false --set controllerManager.replicaCount=1 --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/k3s/containerd/containerd.sock

Then you can check whether chaos mesh is running through k3s kubectl get pods -n chaos-testing

Setup Injectee Pod

I have prepared one in the examples

kubectl apply -f examples/blockchaos/hostpath-persistent-volume.yaml
kubectl apply -f examples/blockchaos/pod.yaml

Inject BlockChaos

kubectl apply -f examples/blockchaos/blockchaos-delay-example.yaml

Then you can verify the injection through read or write inside the /usr/share/nginx/html

kubectl exec -it hostpath-example /bin/bash
# Then execute the following codes in the container

apt update -y && apt install fio -y

cd /usr/share/nginx/html
dd if=

fio --filename=./some --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=16 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=1000000000

You can see:

Jobs: 16 (f=16): [r(16)][21.7%][r=639KiB/s][r=159 IOPS][eta 01m:34s]
Jobs: 16 (f=16): [r(16)][22.5%][r=510KiB/s][r=127 IOPS][eta 01m:33s]
Jobs: 16 (f=16): [r(16)][24.2%][r=511KiB/s][r=127 IOPS][eta 01m:31s] 
Jobs: 16 (f=16): [r(16)][25.8%][r=512KiB/s][r=128 IOPS][eta 01m:29s] 
^Cbs: 16 (f=16): [r(16)][26.7%][r=512KiB/s][r=128 IOPS][eta 01m:28s]

# following code will recover the chaos
kubectl delete -f ./examples/blockchaos/blockchaos-delay-example.yaml 

# Then execute the following codes in the container
cd /usr/share/nginx/html
fio --filename=./some --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=16 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=1000000000

You can see:

Jobs: 16 (f=16): [r(16)][2.5%][r=49.9MiB/s][r=12.8k IOPS][eta 01m:57s]
Jobs: 16 (f=16): [r(16)][3.3%][r=49.2MiB/s][r=12.6k IOPS][eta 01m:56s]
Jobs: 16 (f=16): [r(16)][4.2%][r=50.2MiB/s][r=12.9k IOPS][eta 01m:55s]
Jobs: 16 (f=16): [r(16)][5.0%][r=51.2MiB/s][r=13.1k IOPS][eta 01m:54s]
Jobs: 16 (f=16): [r(16)][5.8%][r=47.6MiB/s][r=12.2k IOPS][eta 01m:53s]
Jobs: 16 (f=16): [r(16)][6.7%][r=49.2MiB/s][r=12.6k IOPS][eta 01m:52s]
Jobs: 16 (f=16): [r(16)][7.5%][r=48.8MiB/s][r=12.5k IOPS][eta 01m:51s]
Jobs: 16 (f=16): [r(16)][9.2%][r=48.3MiB/s][r=12.4k IOPS][eta 01m:49s]

Further Tests

Test with hostPath PV
Test with multiple volumes

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

ti-chi-bot · 2022-02-17T10:40:04Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Andrewmatilde
Hexilee

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

codecov · 2022-02-17T10:52:25Z

Codecov Report

Merging #2907 (c1f86b3) into master (6a379cf) will decrease coverage by 0.26%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #2907      +/-   ##
==========================================
- Coverage   41.09%   40.82%   -0.27%     
==========================================
  Files         165      166       +1     
  Lines       13851    13969     +118     
==========================================
+ Hits         5692     5703      +11     
- Misses       7726     7835     +109     
+ Partials      433      431       -2

Impacted Files	Coverage Δ
api/v1alpha1/blockchaos_types.go	`0.00% <ø> (ø)`
controllers/utils/controller/key.go	`0.00% <0.00%> (ø)`
pkg/chaosdaemon/blockchaos_server.go	`0.00% <0.00%> (ø)`
pkg/workflow/controllers/chaos_node_reconciler.go	`61.31% <0.00%> (+3.15%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fdd4bb7...c1f86b3. Read the comment docs.

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

…haosdaemon

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

YangKeao · 2022-02-23T09:58:25Z

/run-e2e-tests

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

YangKeao · 2022-03-02T04:02:37Z

/build-image

…haosdaemon Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

github-actions · 2022-03-02T04:14:54Z

You can download and import the image with following commands:

./hack/download-image.sh -r chaos-mesh/chaos-mesh -i 1920188139

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

…o impl-blockchaos-chaosdaemon

Andrewmatilde · 2022-06-29T17:07:09Z

controllers/chaosimpl/blockchaos/impl.go

+	}
+
+	blockchaos := obj.(*v1alpha1.BlockChaos)
+	if blockchaos.Status.InjectionIds == nil {


I think will not happen or this is a situation with some unknown problem.

Andrewmatilde · 2022-06-29T17:10:59Z

pkg/chaosdaemon/blockchaos_server.go

+func (s *DaemonServer) ApplyBlockChaos(ctx context.Context, req *pb.ApplyBlockChaosRequest) (*pb.ApplyBlockChaosResponse, error) {
+	log := s.getLoggerFromContext(ctx)
+
+	volumeName, err := normalizeVolumeName(ctx, req.VolumePath)


This is wired , but you have to do it. What a pity.

I think you can access the mnt ns of process in /proc/$PID/root/.

@Andrewmatilde I have considered this problem, the difficult part is the symbolic path resolving. If there is a symlink points to an absolute address, e.g. /home/vagrant/test -> /mnt, then the /proc/1/root/home/vagrant/test in container also points to /mnt, which doesn't exist in the container. To resolve this problem we have to implement a symlink walking by ourselves, which is somehow disappointing, as it may contain relative paths, absolute paths...

I agree with it.

Andrewmatilde · 2022-06-29T17:36:29Z

It seems the example have some problem.

YangKeao · 2022-06-30T04:25:06Z

It seems the example have some problem.

@Andrewmatilde
It prints ioem scheduler not found, which means you haven't installed the chaos-driver? Did you see the io scheduler ioem registered in the dmesg?

Andrewmatilde · 2022-06-30T05:16:42Z

@Andrewmatilde It prints ioem scheduler not found, which means you haven't installed the chaos-driver? Did you see the io scheduler ioem registered in the dmesg?

It fixed. THX

Andrewmatilde

LGTM

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

…haosdaemon Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

cwen0 · 2022-07-05T06:23:26Z

controllers/chaosimpl/blockchaos/impl.go

+	if pbClient != nil {
+		defer pbClient.Close()
+	}
+	if err != nil {


Why not check this error as soon as it is returned at L52?

This is the tricky part (as all of other container-level injection). It'll need to check whether the pbClient should be closed, before returning the error.

I thought there are some problems which makes it hard to clean the pbClient inside DecodeContainerRecord, but I'm not sure. I'll open an issue to optimize this function (DecodeContainerRecord) further. But in this PR, I think we can keep the same with all other implementations.

Hexilee

Rest LGTM

controllers/chaosimpl/blockchaos/impl.go

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

YangKeao · 2022-07-05T07:28:01Z

/merge

ti-chi-bot · 2022-07-05T07:28:04Z

This pull request has been accepted and is ready to merge.

Commit hash: c1f86b3

YangKeao · 2022-07-05T07:28:18Z

After nearly five months 😆

* implement blockchaos on chaosdaemon Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * make check Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * implement blockchaos helper Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * auto enable ioem scheduler Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * make check Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * fix decode container record Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * fix joining the volume path Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * enable local mount and mount namespace for cdh Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * use host-sys rather than sys in chaos-daemon Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * enable CGO for cdh Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * fine tune the protobuf Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * don't override scheduler if it's already set Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * make check Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * remount sysfs with rw permission Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * modify BlockChaos API Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * regenerate something Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * remove blockchaos limit Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * clean go.sum in e2e-test Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * add CHANGELOG.md of blockchaos Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * remove example of limit Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * use float64 correlation Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * close chaos-driver client Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * make tidy Signed-off-by: YangKeao <yangkeao@chunibyo.icu> * return error after recovering Signed-off-by: YangKeao <yangkeao@chunibyo.icu> Signed-off-by: STRRL <im@strrl.dev>

implement blockchaos on chaosdaemon

37c6d2c

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

ti-chi-bot added the do-not-merge/work-in-progress label Feb 17, 2022

ti-chi-bot requested a review from cwen0 February 17, 2022 10:40

ti-chi-bot added the size/XXL label Feb 17, 2022

YangKeao force-pushed the impl-blockchaos-chaosdaemon branch 2 times, most recently from f61d729 to 7db00da Compare February 17, 2022 11:10

YangKeao added 10 commits February 17, 2022 19:10

make check

7db00da

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

Merge remote-tracking branch 'upstream/master' into impl-blockchaos-c…

e4e0d39

…haosdaemon

implement blockchaos helper

19da2c2

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

auto enable ioem scheduler

a575905

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

make check

cdabc2b

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

fix decode container record

4344fcb

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

fix joining the volume path

a324959

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

enable local mount and mount namespace for cdh

ea3a35d

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

use host-sys rather than sys in chaos-daemon

9d709d0

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

enable CGO for cdh

b8b689b

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

YangKeao marked this pull request as ready for review February 23, 2022 09:49

ti-chi-bot removed the do-not-merge/work-in-progress label Feb 23, 2022

YangKeao requested a review from a team February 23, 2022 10:01

fine tune the protobuf

91469ed

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

STRRL self-requested a review February 28, 2022 10:04

ti-chi-bot added needs-rebase and removed needs-rebase labels Feb 28, 2022

Merge remote-tracking branch 'upstream/master' into impl-blockchaos-c…

d68774d

…haosdaemon Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

STRRL mentioned this pull request Mar 2, 2022

Release v2.2.0 #2608

Closed

51 tasks

YangKeao added 5 commits June 27, 2022 19:47

add CHANGELOG.md of blockchaos

970dc7f

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

Merge branch 'master' into impl-blockchaos-chaosdaemon

fc71a2c

Merge branch 'master' into impl-blockchaos-chaosdaemon

605d822

remove example of limit

1b7b1d8

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

Merge remote-tracking branch 'origin/impl-blockchaos-chaosdaemon' int…

bcc784b

…o impl-blockchaos-chaosdaemon

Andrewmatilde reviewed Jun 29, 2022

View reviewed changes

Andrewmatilde approved these changes Jun 30, 2022

View reviewed changes

ti-chi-bot added status/LGT1 needs-rebase labels Jun 30, 2022

YangKeao added 4 commits July 1, 2022 18:52

use float64 correlation

bde80a6

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

Merge remote-tracking branch 'upstream/master' into impl-blockchaos-c…

df71842

…haosdaemon Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

close chaos-driver client

50b2cbd

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

Merge branch 'master' into impl-blockchaos-chaosdaemon

b894ad6

ti-chi-bot removed the needs-rebase label Jul 5, 2022

make tidy

1b5ec68

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

cwen0 reviewed Jul 5, 2022

View reviewed changes

YangKeao mentioned this pull request Jul 5, 2022

DecodeContainerRecord and the pattern to use its result should be optimized #3423

Open

Hexilee approved these changes Jul 5, 2022

View reviewed changes

controllers/chaosimpl/blockchaos/impl.go Outdated Show resolved Hide resolved

ti-chi-bot added status/LGT2 and removed status/LGT1 labels Jul 5, 2022

return error after recovering

c1f86b3

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

ti-chi-bot added the status/can-merge label Jul 5, 2022

ti-chi-bot merged commit a49c43d into chaos-mesh:master Jul 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

blockchaos: impl blockchaos in chaosdaemon #2907

blockchaos: impl blockchaos in chaosdaemon #2907

YangKeao commented Feb 17, 2022 •

edited

ti-chi-bot commented Feb 17, 2022 •

edited

codecov bot commented Feb 17, 2022 •

edited

YangKeao commented Feb 23, 2022

YangKeao commented Mar 2, 2022

github-actions bot commented Mar 2, 2022

Andrewmatilde Jun 29, 2022

Andrewmatilde Jun 29, 2022

Andrewmatilde Jun 29, 2022

YangKeao Jun 30, 2022

Andrewmatilde Jun 30, 2022

Andrewmatilde commented Jun 29, 2022

YangKeao commented Jun 30, 2022 •

edited

Andrewmatilde commented Jun 30, 2022 •

edited

Andrewmatilde left a comment

cwen0 Jul 5, 2022

YangKeao Jul 5, 2022

Hexilee left a comment

YangKeao commented Jul 5, 2022

ti-chi-bot commented Jul 5, 2022

YangKeao commented Jul 5, 2022

blockchaos: impl blockchaos in chaosdaemon #2907

blockchaos: impl blockchaos in chaosdaemon #2907

Conversation

YangKeao commented Feb 17, 2022 • edited

What problem does this PR solve?

Test Steps

Detailed Steps

Bootstraping virtual machine

Install chaos driver

Install k3s and helm

Build and install chaos mesh

Setup Injectee Pod

Inject BlockChaos

Further Tests

ti-chi-bot commented Feb 17, 2022 • edited

codecov bot commented Feb 17, 2022 • edited

Codecov Report

YangKeao commented Feb 23, 2022

YangKeao commented Mar 2, 2022

github-actions bot commented Mar 2, 2022

Andrewmatilde Jun 29, 2022

Choose a reason for hiding this comment

Andrewmatilde Jun 29, 2022

Choose a reason for hiding this comment

Andrewmatilde Jun 29, 2022

Choose a reason for hiding this comment

YangKeao Jun 30, 2022

Choose a reason for hiding this comment

Andrewmatilde Jun 30, 2022

Choose a reason for hiding this comment

Andrewmatilde commented Jun 29, 2022

YangKeao commented Jun 30, 2022 • edited

Andrewmatilde commented Jun 30, 2022 • edited

Andrewmatilde left a comment

Choose a reason for hiding this comment

cwen0 Jul 5, 2022

Choose a reason for hiding this comment

YangKeao Jul 5, 2022

Choose a reason for hiding this comment

Hexilee left a comment

Choose a reason for hiding this comment

YangKeao commented Jul 5, 2022

ti-chi-bot commented Jul 5, 2022

YangKeao commented Jul 5, 2022

YangKeao commented Feb 17, 2022 •

edited

ti-chi-bot commented Feb 17, 2022 •

edited

codecov bot commented Feb 17, 2022 •

edited

YangKeao commented Jun 30, 2022 •

edited

Andrewmatilde commented Jun 30, 2022 •

edited