Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure route table conflicted updating because of concurrent API calls #88151

Closed
feiskyer opened this issue Feb 14, 2020 · 2 comments · Fixed by #88209
Closed

Azure route table conflicted updating because of concurrent API calls #88151

feiskyer opened this issue Feb 14, 2020 · 2 comments · Fixed by #88209

Comments

@feiskyer
Copy link
Member

@feiskyer feiskyer commented Feb 14, 2020

What happened:

Azure NRP doesn't support concurrent updates on route table, while route controller updates route for each node concurrently. Because of this, there would be a lot of conflicted errors from Azure cloud provider and a lot of retries are required to complete route update operations.

Related logs:

E0213 13:10:19.567482       1 route_controller.go:221] Could not delete route nodeName 10.43.0.0/24 after 87.546627ms: timed out waiting for the condition
E0213 13:10:19.567441       1 azure_backoff.go:610] processHTTPRetryResponse: backoff failure, will retry, err=network.RoutesClient#Delete: Failure sending request: StatusCode=0 -- Original Error: Code="RetryableError" Message="A retryable error occurred." Details=[{"code":"RetryableErrorDueToAnotherOperation","message":"Operation DeleteRouteOperation is updating resource <route-table-resource-id>. The call can be retried in 11 seconds."}]

What you expected to happen:

No conflicted errors and hence fewer retries would be needed to update route table.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:

/kind bug
/priority critical-urgent
/sig cloud-provider
/area provider/azure

@feiskyer

This comment has been minimized.

Copy link
Member Author

@feiskyer feiskyer commented Feb 14, 2020

/assign

@feiskyer

This comment has been minimized.

Copy link
Member Author

@feiskyer feiskyer commented Feb 14, 2020

/milestone v1.18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants
You can’t perform that action at this time.