Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rfc: Network Chaos on Kubernetes Service #36

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

rfc: Network Chaos on Kubernetes Service #36

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions text/2022-01-22-inject-network-chaos-based-on-service.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Title
STRRL marked this conversation as resolved.
Show resolved Hide resolved

## Summary

<!-- One para explanation of the proposal. -->
Network chaos injection based on Kubernetes service is urgently needed.
We did some research about ChaosMesh for a few weeks, also comparing with some competitor tools. The first impression is that ChaosMesh product is good, but not perfect, but we think it can do better.

## Motivation

<!-- Why are we doing this? What use cases does it support? What is the expected outcome? -->
The IaaS of our company is completedly based on domestic multi-clouds, just like Ali Cloud, Tencent Cloud, and may have more in future. The target is to make our SaaS & PaaS deployed on Ali Cloud and Tencent Cloud at the same time, and keep both active and active, if one cloud is outage, and the other can be completedly hot backup so that we can reduce the MTTR as most as possible.
STRRL marked this conversation as resolved.
Show resolved Hide resolved

The architecture is briefly described as followed, the two clouds can be connected by fiber lines, and the key idea is to make SaaS & IaaS completely isolated, only the PaaS can be mutual accessed to exchange/sync data.

```
┌┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┐
┆ DNS ┆
└┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┘
∧ ∧
| |
∨ ∨
┌┈┈┈┈┐ ┌┈┈┈┈┐
┆SaaS┆ ┆SaaS┆
└┈┈┈┈┘ └┈┈┈┈┘
∧ ∧
| |
∨ ∨
┌┈┈┈┈┐ ┌┈┈┈┈┐
┆PaaS┆ <---- fiber lines ----> ┆PaaS┆
└┈┈┈┈┘ └┈┈┈┈┘
∧ ∧
| |
∨ ∨
┌┈┈┈┈┐ ┌┈┈┈┈┐
┆IaaS┆ ┆IaaS┆
└┈┈┈┈┘ └┈┈┈┈┘
(Ali Cloud. cidr:172.25.0.0/16) (Tencent Cloud. cidr:10.32.0.0/16)
```

To make above great idea happened, we need to do more network chaos experiments routinely to check if the unitization deployment is still working or broken by any unexpected RD user changes. In Tencent Cloud K8S enviroment (172.25.0.0/16), like based on a specific namespace and service, we expect to inject network 100% loss for 10.32.0.0/16 as external targets, well, it works if the two pods ping directly, but it does not work while accessing Ali Cloud appname using service name. In fact, the service name is resolved to an IPVS LB, which is a proxy for the target appname.
STRRL marked this conversation as resolved.
Show resolved Hide resolved

```
Actualy we expect this process working:


(inject network 100% loss for external target 10.32.0.0/16)
|
∨
Tencent Cloud Namespace/Appname -- ╳ --> {appname}-svc.{namespace} --> Ali Cloud appname
(172.25.0.0/16) (can't ping 10.32.0.0/16 ) (10.32.0.0/16)


But current ChaosMesh does not support, it is still blind in service mechanism, seems like it is a little fall behind Cloud Native Concept.
```


## Detailed design

<!--This is the bulk of the RFC. Explain the design in enough detail that:

- It is reasonably clear how the feature would be implemented.
- Corner cases are dissected by example.
- How the feature is used. -->
I still have no specific detailed design for this feature, but have a rough proposal like this: moving the network chaos experiment from Pods to Nodes, transparently transmit the Pods's info(like ip, namespace, appname, etc) to the Nodes, and add some tc/iptable rules based on Nodes according to the specified namespace/appname.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could apply tc/iptables on node network namespace for Service NetworkChaos.

The Linux network namespace is the basic way to keep isolation and control the blast radius, I prefer to keep using pods' network namespace for Pod NetworkChaos.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This way for ClusterIP and in-cluster network traffic sounds great. Another concerning thing is the networking path for a LoadBalancer service and ClusterIP might be not exactly the same.

I have no idea about how cloud providers treat the LoadBalancer service. I am very glad if you could provide more introduction about that @dengruilong . ❤️

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@STRRL As we have discussed on the last weekly meeting, the Service should only be allowed for the destination of the connection (e.g. the target selector for direction `to). The network path doesn't matter. Because in the view of the source pod, it only sees the target IP, no matter how it reaches the other end.

The only task for us is to get the IP of the service. For LoadBalancer, it seems that the .status.loadBalancer.ingress[].ip and port is faithful, and we could also add the .spec.clusterIP (if they are not nil).

Another question, I wonder whether the service discovering works for the LoadBalancer service and which IP it will return.

Copy link
Member

@STRRL STRRL Jan 24, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@STRRL As we have discussed on the last weekly meeting, the Service should only be allowed for the destination of the connection (e.g. the target selector for direction `to).

That could be the first stage for the feature "NetworkChoas on Kubernetes Service".

I wonder if we could do much more to inject chaos into the "Kubernetes Service" not only network traffic in the Kubernetes cluster. For example, an "external service"(not located in Kubernetes Cluster) could access this LoadBalancer service, also affected by Chaos Mesh NetworkChaos

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not only network traffic in the Kubernetes cluster

It seems we could do that if we modify the tc/iptable rules on Node.

Copy link
Member

@STRRL STRRL Jan 24, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another question, I wonder whether the service discovering works for the LoadBalancer service and which IP it will return.

"Normal" (not headless) Services are assigned a DNS A or AAAA record, depending on the IP family of the service, for a name of the form my-svc.my-namespace.svc.cluster-domain.example. This resolves to the cluster IP of the Service.

As the document says, only ClusterIP is introduced. And it does act as that document says(on my local minikube cluster). I did not dig into the codes of coredns kubernetes plugin, not sure about there is no "magic" in it about resolving external IP. :P

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This way for ClusterIP and in-cluster network traffic sounds great. Another concerning thing is the networking path for a LoadBalancer service and ClusterIP might be not exactly the same.

I have no idea about how cloud providers treat the LoadBalancer service. I am very glad if you could provide more introduction about that @dengruilong . ❤️

We didn't use cloud providers's LoadBalancer, because it usually depended on the provides' difference and hard to work as a universal solution. Instead, we used Ingress as north-to-south communication, and used service (in fact, this service worked as a IPVS, which has high performance in kernel level and highly used by the K8s industry) as horizon communication.

Hope above info can help you clear your concerns. Many thanks ❤️

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any update for the service injection design? @STRRL

Copy link
Member

@STRRL STRRL Feb 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no more new outcomes yet🤔

This feature is not in chaos-mesh/chaos-mesh#2608. Maybe it needs more time to consider.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you have interest in helping us with the design and implementation, PR is welcome! ❤️


Hopefully ChaosMesh community experts can have an universal design for K8s service network injection.

## Drawbacks

<!-- Why should we not do this? -->
- N/A

## Alternatives
<!--
- Why is this design the best in the space of possible designs?
- What other designs have been considered and what is the rationale for not
choosing them?
- What is the impact of not doing this?
-->
- N/A

## Unresolved questions
- N/A
<!-- What parts of the design are still to be determined? -->