Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lock access to the plugin loading channels #13682

Merged
merged 1 commit into from
Aug 10, 2023
Merged

Conversation

Frassle
Copy link
Member

@Frassle Frassle commented Aug 9, 2023

Description

Fixes #12371.

This locks access to the plugin request channels with a RWLock. Before trying to write to the channel we try to take a Read lock (yes this sounds the wrong way round, carry on). Many loaders are free to send to the loadRequest channel at once, but we use the read lock to atomiclly track if any are currently in progress.

When we go to close the plugin host the first thing we do is take a Write lock. Firstly this can't be taken until all the read locks are released indicating that no plugins are currently loading, but secondly while the write lock is taken no more read locks can be taken blocking any further plugin loads from starting.

We never release this write lock, thus permenatly blocking plugin loads once Close is called. So that we don't indefinently block inside load calls we do a TryRLock and return an error if the read lock can't be taken.

With this locking in place the rest of Close is then free to shut down all current plugins and close of the request channels, assured that they shouldn't be posted to again as the lock stays held.

Checklist

  • I have run make tidy to update any new dependencies
  • I have run make lint to verify my code passes the lint check
    • I have formatted my code using gofumpt
  • I have added tests that prove my fix is effective or that my feature works
  • I have run make changelog and committed the changelog/pending/<file> documenting my change
  • Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version

@Frassle Frassle requested review from AaronFriel and a team August 9, 2023 08:36
@pulumi-bot
Copy link
Contributor

pulumi-bot commented Aug 9, 2023

Changelog

[uncommitted] (2023-08-10)

Bug Fixes

  • [engine] Fixes some synchronization in plugin shutdown to prevent panics on Ctrl-C.
    #13682

Copy link
Contributor

@abhinav abhinav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this looks like it'll work.

I think we might be able to do a not-super-complicated test for this:
call loadPlugin with a no-op load and Close concurrently, and it'll probably panic some times without this fix.
(If it's more complicated than that, ignore this part of the comment.)

@Frassle
Copy link
Member Author

Frassle commented Aug 10, 2023

call loadPlugin with a no-op load and Close concurrently,

Yup that panic'd it.

@Frassle
Copy link
Member Author

Frassle commented Aug 10, 2023

bors merge

@abhinav
Copy link
Contributor

abhinav commented Aug 10, 2023

merge will fail. lint is missing t.Parallel. I can handle it in half an hour if you're done for the day.

@Frassle
Copy link
Member Author

Frassle commented Aug 10, 2023

🤦‍♂️ Fixing....

Fixes #12371.

This locks access to the plugin request channels with a RWLock.
Before trying to write to the channel we try to take a Read lock (yes
this sounds the wrong way round, carry on). Many loaders are free to
send to the loadRequest channel at once, but we use the read lock to
atomiclly track if any are currently in progress.

When we go to close the plugin host the first thing we do is take a
Write lock. Firstly this can't be taken until all the read locks are
released indicating that no plugins are currently loading, but secondly
while the write lock is taken no more read locks can be taken blocking
any further plugin loads from starting.

We never release this write lock, thus permenatly blocking plugin loads
once `Close` is called. So that we don't indefinently block inside load
calls we do a `TryRLock` and return an error if the read lock can't be
taken.

With this locking in place the rest of `Close` is then free to shut down
all current plugins and close of the request channels, assured that they
shouldn't be posted to again as the lock stays held.
@bors
Copy link
Contributor

bors bot commented Aug 10, 2023

Canceled.

@Frassle
Copy link
Member Author

Frassle commented Aug 10, 2023

bors merge

@bors
Copy link
Contributor

bors bot commented Aug 10, 2023

🕐 Waiting for PR status (GitHub check) to be set, probably by CI. Bors will automatically try to run when all required PR statuses are set.

@bors
Copy link
Contributor

bors bot commented Aug 10, 2023

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

  • bors-ok

@bors bors bot merged commit cb914ec into master Aug 10, 2023
46 checks passed
@bors bors bot deleted the fraser/pluginCloseSync branch August 10, 2023 23:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Panic when pressing Ctrl + C
3 participants