Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNAT allocated ports reset during scaling operation #1673

Closed
landro opened this issue Jun 15, 2020 · 4 comments
Closed

SNAT allocated ports reset during scaling operation #1673

landro opened this issue Jun 15, 2020 · 4 comments

Comments

@landro
Copy link

landro commented Jun 15, 2020

What happened:

I increased the minimum number of nodes in default pool (2->3 autoscaling pool), and that reset snat allocated ports back to default of 1024

What you expected to happen:

no change

Environment:

1.15.10

@jluk
Copy link
Contributor

jluk commented Jun 15, 2020

@landro do you have a support ticket tied to this issue we can inspect? cc @marwanad / @paulgmiller

@landro
Copy link
Author

landro commented Jun 15, 2020

No, not this exact case. I reproduced the issue on all of our 6 clusters.

We have a support ticket though for a reset that probably happened due to a reconciliation process triggered by AKS. See 120051822001429 for details.

We've had this value reset to the default several times now (either triggered by some management operation triggered by us, or some kind of management operation on the masters, triggered by you) - I'm surprised that this can happen in a GA product like this. Not impressed to say the least, especially given the fact that it apparently takes a long time to get this fixed and released. We've actually had to put in place dedicated monitoring to make sure we get alerted whenever the value is reset.

@jluk
Copy link
Contributor

jluk commented Jun 15, 2020

Thanks for sharing the SR @landro, it looks like this was a bug on your initial report which already has a fix captured for release. The acknowledgement of the issue and a notice of this fix should have come back to you on the support ticket, but I can patch in the info for you here.

It is rolling out on the active service release, which should be global in the next 8 days. This fix will be notated on the aks release notes publishing this week as well. I will leave this open until the fix is global and you confirm the issue is resolved.

@jluk
Copy link
Contributor

jluk commented Aug 7, 2020

This should be resolved now, closing as a result. If you still hit this specific issue just let us know and I will revisit.

@jluk jluk closed this as completed Aug 7, 2020
@ghost ghost locked as resolved and limited conversation to collaborators Sep 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants