Problem when running IOChaos #2305
Comments
|
@faraktingi Hi, could you follow these suggestions, we need more information about the issue, thx.
|
|
Thanks for your help @iguoyr .
|
|
@faraktingi It's strange that there is no chaos-daemon pod in
|
|
|
Perhaps we need to perform this command:
What do you think? |
Yes, as described in the FAQ |
|
now I have: |
|
@faraktingi Great! And try to run I/O latency chaos again, is there still having problems? |
|
yes i restarted it and got this error now: |
|
this command: |
|
any idea please about last error I got? |
@YangKeao It's seems like a issue with
@faraktingi Sorry, it's seems like that |
|
@faraktingi Have this pod opened a lot of file? If the process has opened a lot of files, it may take too long for the We are actively exploring a better way to inject |
|
Thanks for your message @YangKeao. Have this pod opened a lot of file? Here is the result of the following command:
|
|
As an additional information I can provide that I get the same error with several others pods in different namespaces:
So i'm not quite sure this error is related to the number of open files BTW... |
|
Could you provide the log of the |
|
hi |
|
Sorry I made several attempts but in my previous comment this is the last log with a /tmp folder.... But this is similar for every other pods I tried in my namespace. |
|
Hello @YangKeao, Let me know if the logs I sent Yesterday are the one you asked for and if I need to provide something else please. I appreciate your help on this problem. Best, |
Yes! Really Thanks for the log. I have located the error https://github.com/chaos-mesh/toda/blob/master/src/mount.rs#L40 . The chaos mesh is trying to execute Here are several possible situations for 1 . The source is not a volume. For example the # findmnt -o TARGET,PROPAGATION /
TARGET PROPAGATION
/ private,slaveThanks. |
|
here are results of the commands above: sh-4.4$
I do not have the root password. sh-4.4$
|
|
Can you provide the definition of the pod (after hiding the sensitive information, if you have any security concern)? Or could you try to deploy a simple application (e.g. a sleeping ubuntu image with It would be even better to deploy a priviledged sleeping ubuntu, so that you will be able to run the former |
|
@YangKeao thanks for your feedback. Here is the definition of the pod I'm trying to inject I/O chaos:
|
|
Hello, I have created a new namespace and then created the following Pod:
Then I create a new I/O experiment on Volume Path: /data/demo I only got this error: And I think that the experiment passed at the end of the duration (1 minute) I got: Regarding the mount command I tried:
My id is:
|
|
Additional information about MQ pod:
Hi @YangKeao here is some additional definition I did not send to you earlier but you could see the security context of the Pod. Just note that I have also performed the test with chaos-mesh pod (from chaos-mesh testing project) and I got same error. |
Thanks for your reply! The error "Failed to update conditions" doesn't matter. If there are not other error, and there are events like "Successfully apply chaos for NAMESPACE/PODNAME", the injection works well. You can check the latency by running During the injection, there is a file system (called |
|
@YangKeao What is the option we have in such case? Is it something no need to investigate deeply from your side? |
I don't know why the priviledge of target pod will affect the execution of Chaos Mesh, as Chaos Mesh runs all injection under its own user and container (by switching namespaces / cgroups) (and the I really want to investigate, write down all possible situations into the document and enable OpenShift users to use Chaos Mesh out of box, but sometimes I don't know the direction to investigate. Let me read more about the OpenShift and SELinux 🧠 . I need more knowledge to solve this issue. One more question, does other functions of Chaos Mesh (e.g. |
|
Sure @YangKeao Yes I was able to perform some NetworkChaos experiments successfully. Many thanks for your help again. |
|
hello @YangKeao How are you doing? Any news regarding this issue please? Thanks Fabien. |
|
Hello @YangKeao - Do you think I could have an update on this issue soon please? Thanks for your help, |
No. I don't think I could get a solution soon 😿 . |
|
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 21 days |
|
By default, the chaos mesh daemon set does not get scheduled on the control-plane/ master node, and I was getting this error In order to fix it, add a toleration for chaos daemon: |
|
add tolerations to chaos-daemon and redeploy |
I don't know where I should add tolerations in? I assume that I should add tolerations into values.yaml file and change:
From To:
Thank you in advance |
Yes, here is the correct place to put the tolerations. And please notice the intent things. |
|
I am going to close this issue because this issue was created years ago, and it does not active recently. feel free to create new issues/discussions if you still have problem. Thanks! |
Bug Report
What version of Kubernetes are you using?
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0+4c3480d", GitCommit:"4c3480dcd4299c3b3e9a75e28d643177600e7d72", GitTreeState:"clean", BuildDate:"2021-07-09T00:02:08Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
What version of Chaos Mesh are you using?
Controller manager Version: version.Info{GitVersion:"v2.0.1", GitCommit:"2989e66ed8c1a9815cba025664a72ed59422c73d", BuildDate:"2021-08-26T10:04:06Z", GoVersion:"go1.15.11", Compiler:"gc", Platform:"linux/amd64"}
What did you do?
chaos-mesh was installed by executing the following command:
helm install chaos-mesh chaos-mesh/chaos-mesh -n=chaos-testing --set chaosDaemon.runtime=crio --set chaosDaemon.socketPath=/var/run/crio/crio.sock
Trying to do an I/O experiment:
Here is the experiment yaml:
What did you expect to see?
I/O latency running without error.
What did you see instead?
Got this error:
Output of chaosctl
The text was updated successfully, but these errors were encountered: