Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace TryLock and Wait with Lock, and check for idempotency #7

Merged
merged 12 commits into from
Dec 19, 2018

Conversation

mholt
Copy link
Member

@mholt mholt commented Dec 13, 2018

This replaces TryLock and Wait with a single Lock function, and we check after obtaining a lock to make sure Obtain and Renew remain idempotent.

See issue #5.

/cc @DisposaBoy - please help test and review!

@mholt mholt mentioned this pull request Dec 13, 2018
Copy link
Contributor

@DisposaBoy DisposaBoy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The couple inline comments are just context for the changes I made to my fork so I could continue testing.

I updated the test code here https://gist.github.com/DisposaBoy/4967b04ea5db2b662562e5b2fcfc5d76

Running without args, it doesn't complete as expected with this PR so I made some changes to fix that in my fork https://github.com/DisposaBoy/certmagic/tree/newlocker.

Running with -crash results in the child progresses hanging while they wait for the stale lock timeout.

filestorage.go Outdated Show resolved Hide resolved
filestorage.go Outdated Show resolved Hide resolved
storage.go Outdated Show resolved Hide resolved
@mholt
Copy link
Member Author

mholt commented Dec 14, 2018

@DisposaBoy Would you like collaborator privileges so you can push any fixes to my branch and be credited for it in the commit history? Or I can just make the corrections and update this PR. Let me know!

@DisposaBoy
Copy link
Contributor

@DisposaBoy Would you like collaborator privileges so you can push any fixes to my branch and be credited for it in the commit history? Or I can just make the corrections and update this PR. Let me know!

I don't mind; whichever option's easier for you.

filestorage.go Outdated Show resolved Hide resolved
filestorage.go Outdated Show resolved Hide resolved
filestorage.go Outdated Show resolved Hide resolved
filestorage.go Outdated Show resolved Hide resolved
filestorage.go Outdated Show resolved Hide resolved
@mholt
Copy link
Member Author

mholt commented Dec 15, 2018

At this point, I think it would be easier for both of us if I just make you a collaborator -- go ahead and push commits to my branch and then we can fix this together.

mholt and others added 3 commits December 15, 2018 10:37
* Create lockDir just before lock file creation to reduce the chance that another process calls Unlock() and removes lockDir while we were waiting, preventing us from creating the lock file.
* Use the same strategy that Wait() uses to avoid depending on internal state.
filestorage.go Outdated Show resolved Hide resolved
filestorage.go Outdated Show resolved Hide resolved
filestorage.go Outdated Show resolved Hide resolved
filestorage.go Outdated Show resolved Hide resolved
filestorage.go Outdated Show resolved Hide resolved
filestorage.go Outdated Show resolved Hide resolved
Copy link
Member Author

@mholt mholt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking much better, thank you! I just had a couple questions/comments which I left inline.

I'd be willing to push one or two final commits to wrap this thing up if you're ready for that.

filestorage.go Outdated Show resolved Hide resolved
@mholt
Copy link
Member Author

mholt commented Dec 19, 2018

Okay, I've pushed a commit that I think addresses all outstanding issues mentioned in our reviews up to this point.

I decided to get rid of the existing notion of the fileStorageWaiter (it was confusing and buggy I think) and replace it with a fileLockWaiter which is just a channel that is used to block until it is closed. Even this is not strictly necessary, but if there are many threads in the same process that are vying for this lock, this waiter reduces file system hits since only one thread is polling; the rest in this process just block on the waiter.

If that optimization is not valuable (i.e. if we can accept many threads polling disk to check the status of the lock), I think we can do away with the fileLockWaiter entirely (and thus most of the logic in the default case of our switch in the loop to create a lock). Would love to know what you think!

Anyway, ready for a "final" review, hopefully. :)

@DisposaBoy
Copy link
Contributor

Unfortunately I didn't have much time today so I didn't do a very thorough review but the new code is very similar to what I had in mind.

I also wondered if we needed the internal lock (lockWaiters) at all. It seemed to just add more complexity with possibly no big performance gain.

@mholt
Copy link
Member Author

mholt commented Dec 19, 2018

Yeah, like I was saying, it's just to reduce extra polling. If the occasional multiple polling is OK, then we can nuke it.

filestorage.go Outdated Show resolved Hide resolved
@DisposaBoy
Copy link
Contributor

The test app runs fine and nothing else stands out, so apart from my last comment it LGTM.

@DisposaBoy
Copy link
Contributor

Yeah, like I was saying, it's just to reduce extra polling. If the occasional multiple polling is OK, then we can nuke it.

I'd personally nuke it. It's always easy to bring it back later if there's a performance issue.

@mholt
Copy link
Member Author

mholt commented Dec 19, 2018

Sounds good -- I'll do that right now, stay tuned (just a few minutes) for the final commit. Then when you change your review to Approved I can merge this!

@mholt
Copy link
Member Author

mholt commented Dec 19, 2018

You're right, this is much better! Thanks for the iterative reviews. This PR also removes almost 30 lines of code! Ready to merge.

Copy link
Contributor

@DisposaBoy DisposaBoy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :shipit:

@mholt mholt merged commit a3b276a into master Dec 19, 2018
@mholt mholt deleted the newlocker branch December 19, 2018 21:25
pteich added a commit to pteich/caddy-tlsconsul that referenced this pull request Dec 20, 2018
Changes made to be in sync with new storage interface coming from caddyserver/certmagic#7
pteich added a commit to pteich/caddy-tlsconsul that referenced this pull request Jan 3, 2022
Changes made to be in sync with new storage interface coming from caddyserver/certmagic#7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants