StressChaos - certificate has expired or is not yet valid

[niroowns]

Bug Report

What version of Kubernetes are you using?
1.28

What version of Chaos Mesh are you using?
2.6.3

What did you do? / Minimal Reproducible Example
When applying a StressChaos in a particular cluster, we don't see that the chaos experiment has properly succeeded.

What did you expect to see?
Successful completion of the experiment.

What did you see instead?
Events:
Type Reason Age From Message
Normal FinalizerInited 67s initFinalizers Finalizer has been inited
Normal Updated 67s initFinalizers Successfully update finalizer of resource
Normal Started 67s desiredphase Experiment has started
Normal Updated 67s desiredphase Successfully update desiredPhase of resource
Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z"
Normal Updated 67s records Successfully update records of resource
Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z"
Normal Updated 67s records Successfully update records of resource
Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z"
Normal Updated 67s records Successfully update records of resource
Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z"
Normal Updated 67s records Successfully update records of resource
Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z"
Normal Updated 67s records Successfully update records of resource
Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z"
Normal Updated 67s records Successfully update records of resource
Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z"
Normal Updated 67s records Successfully update records of resource
Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z"
Normal Updated 67s records Successfully update records of resource
Warning Failed 66s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:59Z is after 2024-01-04T08:54:33Z"
Normal Updated 66s records Successfully update records of resource
Warning Failed 66s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:59Z is after 2024-01-04T08:54:33Z"
Normal Updated 66s records Successfully update records of resource
Warning Failed 65s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:00Z is after 2024-01-04T08:54:33Z"
Normal Updated 65s records Successfully update records of resource
Warning Failed 65s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:00Z is after 2024-01-04T08:54:33Z"
Normal Updated 65s records Successfully update records of resource
Warning Failed 59s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:06Z is after 2024-01-04T08:54:33Z"
Normal Updated 59s records Successfully update records of resource
Warning Failed 59s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:06Z is after 2024-01-04T08:54:33Z"
Normal Updated 59s records Successfully update records of resource
Warning Failed 39s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:26Z is after 2024-01-04T08:54:33Z"
Normal Updated 39s records Successfully update records of resource
Warning Failed 39s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:26Z is after 2024-01-04T08:54:33Z"
Normal Updated 39s records Successfully update records of resource

chaosctl debug output is the same:
./chaosctl debug stresschaos chaosmesh-cpu-stress -N cluster-addons -n cluster-addons
failed to execute cmd:

failed get pid from pod xxxxxxx/xxxxx-xxxxxx-7c6b8fd5c-rj2zm: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T14:49:21Z is after 2024-01-04T08:54:33Z"

I am really perplexed at the root of where the "after" (2024-01-04T08:54:33Z) timestamp is coming from. We use cert-manager and I have validated that all the certificates being used by the controller and the agent (including the corresponding Issuers) are all fine.

Any help is greatly appreciated.

[STRRL]

it seems that the certificates used by webhook have been outdated, by default, chaos mesh would generate certificates which would be valid for 5 years, ref: https://github.com/chaos-mesh/chaos-mesh/blob/master/helm/chaos-mesh/templates/_certs.tpl

so there might be other tools that changed certificates, maybe contact your kubernetes admin to address that.

If you could provide more information it would be helpful to resolve this issue.