Implement the JVM chaos controller #1264

Gallardot · 2020-12-09T11:30:55Z

What problem does this PR solve?

What is changed and how does it work?

We use chaosblade-exec-jvm to implement JVM chaos. We use admission-webhook to inject the required files and associated configurations. We use JAVA_TOOL_OPTIONS to start a java agent with application.

ps: It is currently only a technical preview, and we need to do more for the documentation and E2E test. 😂

Here are the steps to perform JVM chaos. Have fun.🚀

# create app namespace
kubectl create namespace app

// enable chaos mesh admission-webhook
kubectl label namespace app admission-webhook=enabled

// add the required sidecar profile
kubectl apply -f  examples/jvm/sidecar.yaml  
kubectl apply -f  examples/jvm/sidecar-template.yaml

Preparation done
At this point, we need to prepare a Java application pod, adding the following information to the annotations of pod

admission-webhook.chaos-mesh.org/request: jvmchaos-sidecar

As an example, you can use our demo program.

// deploy a demo app
kubectl apply -f examples/jvm/app.yaml

// wait until the pod is running
kubectl get pods -n app

JVM Agent injection validation

// Open two new terminals to perform the local service
kubectl port-forward -n app deployment/springboot-jvmchaos-demo 12345:8080
kubectl port-forward -n app deployment/springboot-jvmchaos-demo 10086:10086

// Check that the JVM Agent has been injected
curl -N -s http://127.0.0.1:10086/sandbox/default/module/http/sandbox-module-mgr/list

// response information is as follows
sandbox-info        	ACTIVE  	LOADED  	0    	0    	0.0.4          	luanjia@taobao.com
sandbox-module-mgr  	ACTIVE  	LOADED  	0    	0    	0.0.2          	luanjia@taobao.com
chaosblade          	FROZEN  	LOADED  	0    	0    	0.9.0          	Changjun Xiao
sandbox-control     	ACTIVE  	LOADED  	0    	0    	0.0.3          	luanjia@taobao.com
total=4

make a delayed fault injection test

// Very fast response
curl -w "time_total:%{time_total}s" http://127.0.0.1:12345/hello

// Starts a delayed fault injection for all path in the servlet
kubectl apply -f examples/jvm/jvm-delay4servlet-example.yaml 

// about 10s delay
curl -w "time_total:%{time_total}s" http://127.0.0.1:12345/hello

// clean 
 kubectl delete -f examples/jvm/jvm-delay4servlet-example.yaml 

// Very fast response
curl -w "time_total:%{time_total}s" http://127.0.0.1:12345/hello

Do a test to modify the method's return value

// return `hello world!`
curl http://127.0.0.1:12345/hello

// change the return value to 'hello chaos mesh!'
kubectl apply -f examples/jvm/jvm-return-example.yaml 

// return `hello chaos mesh!`
curl http://127.0.0.1:12345/hello

// clean 
kubectl delete -f examples/jvm/jvm-return-example.yaml 

// return `hello world!`
curl http://127.0.0.1:12345/hello

Checklist

Tests

Unit test
E2E test
Manual test (add detailed scripts or steps below)
No code

Side effects

Breaking backward compatibility

Related changes

Need to update the documentation

Does this PR introduce a user-facing change?

NONE

Signed-off-by: Gallardot <tttick@163.com>

codecov-io · 2020-12-11T02:26:07Z

Codecov Report

Merging #1264 (092141d) into master (7e9ff3f) will decrease coverage by 4.40%.
The diff coverage is 54.34%.

@@            Coverage Diff             @@
##           master    #1264      +/-   ##
==========================================
- Coverage   55.78%   51.37%   -4.41%     
==========================================
  Files          68       79      +11     
  Lines        4383     5014     +631     
==========================================
+ Hits         2445     2576     +131     
- Misses       1768     2174     +406     
- Partials      170      264      +94

Impacted Files	Coverage Δ
api/v1alpha1/common_types.go	`0.00% <0.00%> (ø)`
api/v1alpha1/common_webhook.go	`100.00% <ø> (ø)`
api/v1alpha1/dnschaos_type.go	`0.00% <0.00%> (ø)`
api/v1alpha1/dnschaos_webhook.go	`0.00% <0.00%> (ø)`
api/v1alpha1/httpchaos_types.go	`0.00% <0.00%> (ø)`
api/v1alpha1/iochaos_types.go	`0.00% <ø> (-40.00%)`	⬇️
api/v1alpha1/jvmchaos_webhook.go	`0.00% <0.00%> (ø)`
api/v1alpha1/kernelchaos_types.go	`0.00% <ø> (-20.00%)`	⬇️
api/v1alpha1/kernelchaos_webhook.go	`100.00% <ø> (+14.81%)`	⬆️
api/v1alpha1/kinds.go	`27.27% <ø> (+0.60%)`	⬆️
... and 128 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d4852fb...092141d. Read the comment docs.

Signed-off-by: Gallardot <tttick@163.com>

STRRL · 2020-12-21T06:37:33Z

Hi @Gallardot, is there any documents about the web API http://%s:%d/sandbox/default/module/http/chaosblade/create and docker image gallardot/chaos-jvm?😁

Gallardot · 2020-12-21T06:54:49Z

Hi @Gallardot, is there any documents about the web API http://%s:%d/sandbox/default/module/http/chaosblade/create and docker image gallardot/chaos-jvm?😁

Sorry, the web API has no documentation. But we can find the implementation code.

The docker image gallardot/chaos-jvm is our demo program.

Signed-off-by: Gallardot <tttick@163.com>

STRRL · 2021-01-11T09:38:01Z

Hi @Gallardot , sorry for my late reply, as you mentioned before, copy "jvm-sandbox" then attach JVM in the container works well.

I think injecting chaos at runtime is better than manually changing workload spec, although the first one is more complex.

How about making JAVA_HOME and USER (and other necessary things) as fields in the JVMChaosSpec? As we have "chaos-daemon", we could implement "copy file" and "change user" without kubernetes API.

Multi-container and more than one java process should also be considered, we should enrich out chaos spec.

Gallardot · 2021-01-13T02:49:15Z

As we discussed, executing JVM Chaos by Chaos-daemon is desirable, but difficult to handle all cases. With Sidecar we can handle most cases.
Therefore, we plan to implement both ways, and the user can choose one or the other depending on the actual situation. @STRRL is going to help us implement the Chaos-daemon way. Thanks to @STRRL .❤️

We want to merge the current PR first, and then we create other PR to implement the Chaos-daemon way and supplement the E2E use cases and documentation.

ps: I will fix the conflicting files later.

@cwen0 @YangKeao PTAL

Signed-off-by: Gallardot <tttick@163.com>

…haos-mesh into jvm-chaos-controller

STRRL

LGTM

cwen0

Rest LGTM

cwen0 · 2021-01-19T08:43:16Z

controllers/jvmchaos/types.go

+	r.Log.Info("Try to apply jvm chaos", "namespace",
+		pod.Namespace, "name", pod.Name)
+
+	err := jvm.ActiveSandbox(pod.Status.PodIP, 10086)


Use a const to defein 10086?

Done! PTAL!

Signed-off-by: Gallardot <tttick@163.com>

STRRL

LGTM

cwen0

LGTM

cwen0 · 2021-01-26T09:35:06Z

/merge

ti-srebot · 2021-01-26T09:42:33Z

/run-all-tests

ti-srebot · 2021-01-26T09:57:34Z

@Gallardot merge failed.

Signed-off-by: Gallardot <tttick@163.com>

STRRL · 2021-01-26T10:21:22Z

/merge

ti-srebot · 2021-01-26T10:21:33Z

/run-all-tests

Gallardot added 2 commits December 9, 2020 19:32

implement the JVM chaos controller

dc7eeb5

Signed-off-by: Gallardot <tttick@163.com>

implement the JVM chaos controller

c8b006c

Signed-off-by: Gallardot <tttick@163.com>

Gallardot force-pushed the jvm-chaos-controller branch from ba5ac74 to c8b006c Compare December 9, 2020 11:32

Gallardot added 2 commits December 9, 2020 19:33

Merge branch 'master' into jvm-chaos-controller

993249a

fix ci

8a6c577

Signed-off-by: Gallardot <tttick@163.com>

Gallardot added 2 commits December 11, 2020 10:31

Update to version 0.9.0 of chaosblade-exec-jvm

ac8f978

Signed-off-by: Gallardot <tttick@163.com>

Fix CI

7c7f7b3

Signed-off-by: Gallardot <tttick@163.com>

Gallardot force-pushed the jvm-chaos-controller branch from 8477efe to 7c7f7b3 Compare December 11, 2020 02:32

clean code and add example

61e46e2

Signed-off-by: Gallardot <tttick@163.com>

Gallardot force-pushed the jvm-chaos-controller branch from 1469b34 to 61e46e2 Compare December 16, 2020 05:55

Merge branch 'master' into jvm-chaos-controller

a562aa4

Gallardot marked this pull request as ready for review December 16, 2020 06:11

Gallardot requested review from cwen0, 70data, Colstuwjx, STRRL, WangXiangUSTC and YangKeao December 16, 2020 06:12

Gallardot added chaos/jvm component/operator status/PTAL labels Dec 16, 2020

Merge branch 'master' into jvm-chaos-controller

4644db8

Gallardot force-pushed the jvm-chaos-controller branch from d1ee870 to 79e1812 Compare January 2, 2021 09:29

fix CI

97857ce

Signed-off-by: Gallardot <tttick@163.com>

Gallardot force-pushed the jvm-chaos-controller branch from 79e1812 to 97857ce Compare January 2, 2021 09:50

Gallardot added 2 commits January 14, 2021 17:34

address the comments

8cc8c8f

Signed-off-by: Gallardot <tttick@163.com>

Merge branch 'jvm-chaos-controller' of https://github.com/Gallardot/c…

64cc899

…haos-mesh into jvm-chaos-controller

Gallardot requested a review from STRRL January 14, 2021 09:38

STRRL previously approved these changes Jan 19, 2021

View reviewed changes

cwen0 reviewed Jan 19, 2021

View reviewed changes

address the comments

b4a659f

Signed-off-by: Gallardot <tttick@163.com>

Gallardot dismissed STRRL’s stale review via b4a659f January 20, 2021 08:31

Merge branch 'master' into jvm-chaos-controller

14ea15a

Gallardot requested review from STRRL and cwen0 January 20, 2021 08:38

STRRL previously approved these changes Jan 20, 2021

View reviewed changes

cwen0 previously approved these changes Jan 26, 2021

View reviewed changes

ti-srebot added the status/can-merge label Jan 26, 2021

Merge branch 'master' into jvm-chaos-controller

092141d

fix ci

ce2aba5

Signed-off-by: Gallardot <tttick@163.com>

Gallardot dismissed stale reviews from cwen0 and STRRL via ce2aba5 January 26, 2021 10:03

cwen0 requested review from STRRL and cwen0 January 26, 2021 10:19

cwen0 approved these changes Jan 26, 2021

View reviewed changes

STRRL approved these changes Jan 26, 2021

View reviewed changes

ti-srebot merged commit 65b7d25 into chaos-mesh:master Jan 26, 2021

Gallardot deleted the jvm-chaos-controller branch January 26, 2021 10:48

WangXiangUSTC mentioned this pull request Mar 1, 2021

support JVM chaos chaos-mesh/chaosd#21

Closed

cwen0 mentioned this pull request Mar 23, 2021

Remove unnecessary namespaceSelector from MutatingWebhookConfiguration #1580

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement the JVM chaos controller #1264

Implement the JVM chaos controller #1264

Gallardot commented Dec 9, 2020 •

edited

codecov-io commented Dec 11, 2020 •

edited

STRRL commented Dec 21, 2020

Gallardot commented Dec 21, 2020

STRRL commented Jan 11, 2021

Gallardot commented Jan 13, 2021

STRRL left a comment

cwen0 left a comment

cwen0 Jan 19, 2021

Gallardot Jan 20, 2021

STRRL left a comment

cwen0 left a comment

cwen0 commented Jan 26, 2021

ti-srebot commented Jan 26, 2021

ti-srebot commented Jan 26, 2021

STRRL commented Jan 26, 2021

ti-srebot commented Jan 26, 2021

Implement the JVM chaos controller #1264

Implement the JVM chaos controller #1264

Conversation

Gallardot commented Dec 9, 2020 • edited

What problem does this PR solve?

What is changed and how does it work?

Checklist

Does this PR introduce a user-facing change?

codecov-io commented Dec 11, 2020 • edited

Codecov Report

STRRL commented Dec 21, 2020

Gallardot commented Dec 21, 2020

STRRL commented Jan 11, 2021

Gallardot commented Jan 13, 2021

STRRL left a comment

Choose a reason for hiding this comment

cwen0 left a comment

Choose a reason for hiding this comment

cwen0 Jan 19, 2021

Choose a reason for hiding this comment

Gallardot Jan 20, 2021

Choose a reason for hiding this comment

STRRL left a comment

Choose a reason for hiding this comment

cwen0 left a comment

Choose a reason for hiding this comment

cwen0 commented Jan 26, 2021

ti-srebot commented Jan 26, 2021

ti-srebot commented Jan 26, 2021

STRRL commented Jan 26, 2021

ti-srebot commented Jan 26, 2021

Gallardot commented Dec 9, 2020 •

edited

codecov-io commented Dec 11, 2020 •

edited