New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple IP addresses for proxy in consul deployment #9441

Closed
zhaohuabing opened this Issue Oct 20, 2018 · 9 comments

Comments

Projects
None yet
3 participants
@zhaohuabing
Copy link
Contributor

zhaohuabing commented Oct 20, 2018

Describe the feature request
I ran into a multi-IP addresses problem when trying to integrate Istio with our product, here is the situation I'm facing:

  • The applications are deployed in K8S.
  • We don't use "K8S service" for service discovery, instead, we're using Consul as the service registry. The two main reasons are:
    1. We need multiple network Interfaces in the POD, so we developed our own CNI implementation, and multiple network interfaces can't work well with K8S service.
    2. Some of the legacy applications are not typical Microservices, they have multiple APIs which need to be registered independently.
  • We deploy Istio control plane components(Pilot, Mixer, Istio API Server) on the host, and deploy apps in K8S cluster, an Envoy proxy is deployed alongside the application in each pod.
  • Services are registered to the Consul, and Consul is integrated with Istio Pilot.
  • Each pod has two network interfaces(Let's say network1 and network2), for some reasons, some services are registered via network1, others are registered via network2.

The pilot is using the IP Address reported by Envoy proxy to build the inbound listener(The IP address is conveyed by the id in the node structure of the xDS request, such as sidecar~192.168.206.23~productpage-v1-54b8b9f55-bx2dq.default~default.svc.cluster.local).
In case that there're two IP Addresses in the pod, only the IP address of the first network interface is sent to Pilot, so when services in the pod are registered via the IP address of the second network interface, the Pilot doesn't know these services are located in the same pod with the proxy and don't build inbound listeners for them. This causes an infinite loop when the envoy receiving an inbound request and result in envoy crash because of running out of file descriptors.

Although multiple network interfaces in a pod may sound rare, this can also happen in VM/bare metal deployment of Istio(without K8S).

`

                  +------+         +------+
                  |Pilot +-------->+Consul|
                  ++----++         +---+--+
                   ^    |              ^
                   |    |Svc1 as       |
        xDS request|    |outbound      | Register Svc1
        (IP 1)     |    |listener      | (IP 2)
                   |    v              |
                 +-------------------------+
                 | +------+       +-----+  |
                 | |Envoy |       |Svc 1|  |
                 | +------+       +-----+  |
                 |         Pod/Host        |
                 +-------------------------+
                    |--|           |--|
                    +--+           +--+
                    IP 1           IP 2

`

Describe alternatives you've considered
Allows proxy to send multiple IP addresses to Pilot, Pilot use all the IP addresses to tell which services are located with the proxy and build the inbound listeners for that proxy.

Additional context
If this request does make sense, I'd be glad to contribute the codes. It's almost done in my current working project.

@zhaohuabing zhaohuabing changed the title Support multiple IP addresses for Envoy proxy at Consul deployment Support multiple IP addresses for Envoy proxy in Consul deployment Oct 20, 2018

@zhaohuabing zhaohuabing changed the title Support multiple IP addresses for Envoy proxy in Consul deployment Support multiple IP addresses for Envoy proxy in consul deployment Oct 20, 2018

@zhaohuabing zhaohuabing changed the title Support multiple IP addresses for Envoy proxy in consul deployment Support multiple IP addresses for proxy in consul deployment Oct 20, 2018

@baodongli

This comment has been minimized.

Copy link

baodongli commented Oct 26, 2018

Thanks for creating the issue. As it stands right now, only podIP is available through k8s API. And as you've found out, the podIP is passed to pilot from the proxy.

But it might be possible to do this with the help of istio CNI that we are developing since it's aware of all the IPs assigned to the POD.

@ayj ayj added the area/networking label Oct 26, 2018

@zhaohuabing

This comment has been minimized.

Copy link
Contributor Author

zhaohuabing commented Oct 29, 2018

Hi @baodongli , should we address this issue in the data plane API rather than in the Istio CNI? We know that Istio can run with or without kubernetes.

The workaround I'm using now (DexMesh@97417e8) is to pass all the IPs to Pilot via the Node Metadata of the discovery request. This approach is compatible with the current data plane API and it solves the issue in both the kubernetes and consul deployment of Istio.

@baodongli

This comment has been minimized.

Copy link

baodongli commented Oct 29, 2018

@zhaohuabing the change looks good to me. My question to you is how you passed the multiple IP addresses to pilot agent which builds the node metadata. I guess that it was manually entered in the manifest?

@zhaohuabing

This comment has been minimized.

Copy link
Contributor Author

zhaohuabing commented Oct 30, 2018

@baodongli We can directly get the IPs of the node in pilot-agent using go net package, this approach works both with and without kubernetes: DexMesh@e6ce750

@baodongli

This comment has been minimized.

Copy link

baodongli commented Oct 30, 2018

That's right, pilot agent can simply grab IPs from the interfaces.

@zhaohuabing

This comment has been minimized.

Copy link
Contributor Author

zhaohuabing commented Oct 31, 2018

Hi guys, could we merge this enhancement to the next release if it makes sense?

@baodongli

This comment has been minimized.

Copy link

baodongli commented Oct 31, 2018

Why not propose a PR for it?

@zhaohuabing

This comment has been minimized.

Copy link
Contributor Author

zhaohuabing commented Nov 1, 2018

Why not propose a PR for it?

This is my first time trying to submit codes to Istio, is there anything specific I need to do before proposing the PR?

Thanks,
Huabing

zhaohuabing added a commit to DexMesh/istio that referenced this issue Nov 1, 2018

Support multiple network interfaces(istio#9441)
Allows proxy to send multiple IP addresses to Pilot, Pilot use all the
IP addresses to tell which services are located with the proxy and build
the inbound listeners for that proxy.

Fixes istio#9441

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

zhaohuabing added a commit to DexMesh/istio that referenced this issue Nov 10, 2018

Support multiple network interfaces(istio#9441)
Allows proxy to send multiple IP addresses to Pilot, Pilot use all the
IP addresses to tell which services are located with the proxy and build
the inbound listeners for that proxy.

Fixes istio#9441

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

zhaohuabing added a commit to DexMesh/istio that referenced this issue Dec 3, 2018

Support multiple network interfaces(istio#9441)
Allows proxy to send multiple IP addresses to Pilot, Pilot use all the
IP addresses to tell which services are located with the proxy and build
the inbound listeners for that proxy.

Fixes istio#9441

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Fix unit test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Add IP Addresses to test data

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Refactory test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Convert IPAddress into an array

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Fix review issues

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Proxy have to carry valid IP addressess

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Address review comments

Fail back to ip from node id if ISTIO_META_INSTANCE_IPS set wrong format
Add unit test for multiple IP Addresses

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Fix unit tests

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

fix e2e test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

fix test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Remove duplicated codes

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

zhaohuabing added a commit to DexMesh/istio that referenced this issue Dec 10, 2018

Support multiple network interfaces(istio#9441)
Allows proxy to send multiple IP addresses to Pilot, Pilot use all the
IP addresses to tell which services are located with the proxy and build
the inbound listeners for that proxy.

Fixes istio#9441

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Fix unit test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Add IP Addresses to test data

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Refactory test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Convert IPAddress into an array

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Fix review issues

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Proxy have to carry valid IP addressess

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Address review comments

Fail back to ip from node id if ISTIO_META_INSTANCE_IPS set wrong format
Add unit test for multiple IP Addresses

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Fix unit tests

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

fix e2e test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

fix test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Remove duplicated codes

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

fix lint format

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

fix test failure

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

zhaohuabing added a commit to DexMesh/istio that referenced this issue Dec 11, 2018

Support multiple network interfaces(istio#9441)
Allows proxy to send multiple IP addresses to Pilot, Pilot use all the
IP addresses to tell which services are located with the proxy and build
the inbound listeners for that proxy.

Fixes istio#9441

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Fix unit test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Convert IPAddress into an array

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Proxy have to carry valid IP addressess

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Fail back to ip from node id if ISTIO_META_INSTANCE_IPS set wrong format
Add unit test for multiple IP Addresses

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

rshriram added a commit that referenced this issue Dec 14, 2018

Support multiple network interfaces(#9441) (#9688)
* Support multiple network interfaces(#9441)

Allows proxy to send multiple IP addresses to Pilot, Pilot use all the
IP addresses to tell which services are located with the proxy and build
the inbound listeners for that proxy.

Fixes #9441

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Fix unit test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Convert IPAddress into an array

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Proxy have to carry valid IP addressess

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

Fail back to ip from node id if ISTIO_META_INSTANCE_IPS set wrong format
Add unit test for multiple IP Addresses

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* Fix test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* Fix comments

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
@zhaohuabing

This comment has been minimized.

Copy link
Contributor Author

zhaohuabing commented Dec 15, 2018

This has been fixed by #9688

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment