Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a mutex lock to 'awsCloudInstances' map #15415

Merged
merged 3 commits into from
May 19, 2023
Merged

Add a mutex lock to 'awsCloudInstances' map #15415

merged 3 commits into from
May 19, 2023

Conversation

fcgravalos
Copy link
Contributor

@fcgravalos fcgravalos commented May 12, 2023 •
edited
Loading

We're using terraform kops provider to manage our AWS kops clusters. Time to time we hit a race condition which the stack trace points to awsup.NewAWSCloud function, when writing to a map that holds the relationship regions to AWSCloud objects.

This PR changes this map to a new variable so it belongs to a new type, that wraps the map into its own struct where access is controlled by a mutex lock.

Let me know if that makes sense to you all.

Thanks for building this awesome project!

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 12, 2023
@k8s-ci-robot
Copy link
Contributor

Welcome @fcgravalos!

It looks like this is your first PR to kubernetes/kops 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kops has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 12, 2023
@k8s-ci-robot
Copy link
Contributor

Hi @fcgravalos. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot requested a review from hakman May 12, 2023 19:25
@k8s-ci-robot k8s-ci-robot added the area/provider/aws Issues or PRs related to aws provider label May 12, 2023
@k8s-ci-robot k8s-ci-robot requested a review from zetaab May 12, 2023 19:25
@fcgravalos
Copy link
Contributor Author

This is a piece of the stack trace of our terraform run:

image

@fcgravalos fcgravalos force-pushed the fix_potential_race_condition_in_aws_cloud_builder branch 2 times, most recently from 66c9896 to 609ab46 Compare May 12, 2023 22:26
@fcgravalos
Copy link
Contributor Author

@rifelpet would you mind re-triggering tests again? Thanks!

@hakman
Copy link
Member

hakman commented May 13, 2023

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 13, 2023
@fcgravalos
Add a mutex lock to 'awsCloudInstances' map
3e03eea 
We're using terraform kops provider to manage our AWS kops clusters.
Time to time we hit a race condition with the stack trace points to
`awsup.NewAWSCloud` function, when writing to a concurrent map, that
maintains a map between regions and `AWSCloud` objects.

This PR changes this to variable so it belongs to a new type, that wraps
the map into its own struct where access is controlled by a mutex lock.

Let me know if that makes sense to you all.

Thanks for building this awesome project!
@fcgravalos fcgravalos force-pushed the fix_potential_race_condition_in_aws_cloud_builder branch from 609ab46 to 3e03eea Compare May 13, 2023 08:48
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 13, 2023
@fcgravalos
Copy link
Contributor Author

@hakman made another change to make sure concurrent access to the ResetAWSCloudInstances is safe too, I believe this now requires again the /ok-to-test command? sorry about that.

@fcgravalos
Copy link
Contributor Author

fcgravalos commented May 13, 2023 •
edited
Loading

All the ci checks passed! 🚀

@fcgravalos
Copy link
Contributor Author

Hey @hakman @zetaab ! wdyt? Does it look like a reasonable PR?

@hakman
Copy link
Member

hakman commented May 18, 2023

I think mutexes need to be used both in write and read places, not just write.

@fcgravalos
Copy link
Contributor Author

@hakman added locking on reads too

@justinsb
Copy link
Member

Thanks @fcgravalos

/label tide/merge-method-squash

/approve
/lgtm

@k8s-ci-robot k8s-ci-robot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label May 19, 2023
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 19, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 19, 2023
@k8s-ci-robot k8s-ci-robot merged commit 1e32145 into kubernetes:master May 19, 2023
zetaab pushed a commit to zetaab/kops that referenced this pull request Jun 10, 2023
@fcgravalos @zetaab
Add a mutex lock to 'awsCloudInstances' map (kubernetes#15415)
cb861dc 
* Add a mutex lock to 'awsCloudInstances' map

We're using terraform kops provider to manage our AWS kops clusters.
Time to time we hit a race condition with the stack trace points to
`awsup.NewAWSCloud` function, when writing to a concurrent map, that
maintains a map between regions and `AWSCloud` objects.

This PR changes this to variable so it belongs to a new type, that wraps
the map into its own struct where access is controlled by a mutex lock.

Let me know if that makes sense to you all.

Thanks for building this awesome project!

* lock on reads as well

* cosmetic change, removing empty line
Shimiazoulai pushed a commit to spotinst/kubernetes-kops that referenced this pull request Jul 13, 2023
@fcgravalos
Add a mutex lock to 'awsCloudInstances' map (kubernetes#15415)
12bee2b 
* Add a mutex lock to 'awsCloudInstances' map

We're using terraform kops provider to manage our AWS kops clusters.
Time to time we hit a race condition with the stack trace points to
`awsup.NewAWSCloud` function, when writing to a concurrent map, that
maintains a map between regions and `AWSCloud` objects.

This PR changes this to variable so it belongs to a new type, that wraps
the map into its own struct where access is controlled by a mutex lock.

Let me know if that makes sense to you all.

Thanks for building this awesome project!

* lock on reads as well

* cosmetic change, removing empty line
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hakman hakman Awaiting requested review from hakman

@zetaab zetaab Awaiting requested review from zetaab

Assignees

@justinsb justinsb

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/provider/aws Issues or PRs related to aws provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@fcgravalos @k8s-ci-robot @hakman @justinsb