Failed to disable source destination check due to: RequestLimitExceeded: Request limit exceeded.

[alebsack]

Running v0.1.0-rc2 When running a cluster in AWS with more than ~30 nodes across three AZs, I start getting this:

E0218 19:54:33.307172       1 network_routes_controller.go:907] Failed to disable source destination check due to: RequestLimitExceeded: Request limit exceeded.
	status code: 503, request id: 1e048252-93d2-4672-a3b2-d65163a78220

Looking at the code, every instance of kube-router simply tries to disable the SourceDestCheck on every other node in the cluster. This ends up ultimately leaving some nodes with SourceDestCheck never disabled.

I'm also getting a lot of these, though I'm not sure if related to the above error:

E0218 19:54:33.756882       1 health_controller.go:100] Network Routing Controller heartbeat missed

[murali-reddy]

@alebsack thanks for reporting the issue.

Are you using KOPS or custom installer?

I usually use KOPS to test large cluster. In kops only master node has necessary IAM to access EC2 api, so never ran into this issue. But I agree there is no point each node trying to disable src-dst check for rest of the nodes.

Health check failure is due to failure to setup crticial step (disabling src-dst check). I will work out a fix.

[alebsack]

@murali-reddy Thanks for the response. Yes, I am using KOPS. I did notice that this is not a problem in us-west-2, however a cluster of mine in eu-west-1 is acting differently -- the nodes are able to call ModifyInstanceAttribute (though the IAM policies in both clusters are identical). I'll have to open an issue with AWS on that one.

[murali-reddy]

@alebsack please use image cloudnativelabs/kube-router-git:master with the fix.

[alebsack]

@murali-reddy thanks for the quick turnaround... I will test this in the coming days and report back.