This article explains how to implement yet another Service Mesh solution by extending Apache APISIX.
- Written for the first time on 2021/02/09
- Service Mesh
- Apache APISIX
- Sidecar in APISIX Mesh
- Communcation Bus Protocol
- The Selection of Control Plane
Service Mesh, is a technology to control how individual application talks with each other, applications get released from it since all essential functions like routing, service discovery, authenication, authorizatoin and etc are implemented in the Service Mesh solutions, just like what Istio and Linkerd does.
Typically, there're two components in a Service Mesh solution, the control plane and data plane. The former, as the brain of Service Mesh, discovering services from the service registry (Kubernetes, Consul and others), accepting configuration change requests from admin or CI/CD robots and delivering all configurations to the data plane; The data plane, usually composed by an sidecar process and an application process, is deployed distributed.
Just like the introduction in the website of Apache APISIX:
Apache APISIX is a dynamic, real-time, high-performance Cloud-Native API gateway, based on the Nginx library and etcd.
Apache APISIX software provides rich traffic management features such as load balancing, dynamic upstream, canary release, circuit breaking, authentication, observability, and more.
Apache APISIX is an excellent API Gateway solution, but not only for API Gateway, it owns all the necessary characters that a Service Mesh sidecar needs. It has fantastic resources utilization, flexible routing capabilities, rich plugins and can be extended easily.
In the APISIX Mesh solution, Apache APISIX is designed as the proxy in the data plane, rather than the whole the sidecar component since Apache APISIX is coupled with etcd v3 API tightly, while this protocol is not so common, for the sake of adjusting existing control plane solutions easily, it's not wise to use this protocol as the communication bus protocol between control plane and data plane.
That's why another program (named as apisix-mesh-agent) comes in, it uses a well designed protocol to talk to the control plane, receiving configurations from it, minicking etcd v3 API for the concomitant Apache APISIX.
Resort to this design, what Apache APISIX needs to change is only the value of etcd.host, just tweaking to the etcd v3 API address of concomitant apisix-mesh-agent (the blue arrow in the above diagram).
More importantly, apisix mesh agent will set up dozens of iptables rules, to intercept the inbound (pink arrow) and outbound (brown arrow) traffics of the application.
Of course, the above all are not all functions that the apisix mesh agent provides, it also has other auxiliary features such as delivering TLS/SSL certificates, uploading logs, tracing data, metrics for better observability and etc. But for the first stage, only core funcionalities (routing, inbound, outbound traffic interceptions) will be focused on, other features will be added gradually.
As above mentioned, the etcd v3 API protocol is not a good choice to as the communcation protocol between the data plane and control plane, instead, a well designed, service mesh dedicated protocol is required, and the Envoy xDS protocol is the best one (at least for now), not only because its rich data structures, underlying transport protocol, but also for its spread and adoption. With the help of xDS protocol, the selection of control plane is not force as long as it also supports the xDS protocol.
Therefore, the apisix-mesh-agent should implement the xDS client side protocol, fortunately, not much effort needs to be take, there is an existing SDK go-control-plane.
As the xDS protocol is used, the control plane selection is clear, any products once support xDS protocol can be used as the control plane of The APISIX Mesh, like Istio, Kuma. Use existing control plane products reduce the migration overheads since control planes are always incompatible with each other. It's difficult to ask users migrate from one control plane to another. Target of the first stage is the adoption of Apache APISIX as the data plane.
The architecture is followed. In the future, Custom control plane will be supported, it'll be designed flexible, easy to use/deploy and high available.
