pacman-helper: add a command to break a lease #488
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To avoid interference between concurrent Pacman packages uploads, Git
for Windows' helper scripts first acquire a lock ("lease" in Azure
speak). Ideally, this lease would time out after something like 30
minutes, or even an hour, which would be a quite conservative maximum
estimate how long a successful upload could possibly take.
However, we cannot choose such a time-out because the maximum
non-infinite lease you can acquire for Azure Blobs is 60 seconds, see:
https://learn.microsoft.com/en-us/rest/api/storageservices/lease-blob.
Therefore, we _must_ acquire an infinite lease.
The downside is that we sometimes need to break that lease manually,
e.g. when a workflow run failed after acquiring such a lease (and hence
also failing to release it).
Previously, there was no tooling to do so, the only way to do it was to
direct a web browser to the Azure Portal, navigate to the `x86-64` Azure
Blob container and manually clicking the "Break lease" menu item.
This obviously adds to the stress when things go wrong (and the lease
only ever has to be broken when things go wrong). So, following the
mantra "deployments should be uneventful", we need to put tooling in
place that makes this operation much more of a no-brainer.
With this commit, there is now a scripted way for which we will add a
new GitHub workflow in git-for-windows/git-for-windows-automation, and
probably also teach the GitForWindowsHelper GitHub App a new trick
(`/break pacman-upload-lock` maybe?) to trigger that workflow.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
rimrul
approved these changes
Mar 7, 2023
rimrul
added a commit
to rimrul/git-for-windows-automation
that referenced
this pull request
Jul 4, 2024
To prevent parallel uploads of pacman packages from interfering with each other Git for Windows uses Azure blobs leases without a timeout. Sometimes it's necessary to manually break such a lease if a job acquired a lease and failed to release it. We can do that via the pacman-helper script[1], but having it available as a GitHub workflow or eventually a slash command makes this easier to dispatch. This fixes git-for-windows#40 [1] git-for-windows/build-extra#488 Signed-off-by: Matthias Aßhauer <mha1993@live.de>
rimrul
added a commit
to rimrul/git-for-windows-automation
that referenced
this pull request
Jul 4, 2024
To prevent parallel uploads of pacman packages from interfering with each other Git for Windows uses Azure blobs leases without a timeout. Sometimes it's necessary to manually break such a lease if a job acquired a lease and failed to release it. We can do that via the pacman-helper script[1], but having it available as a GitHub workflow or eventually a slash command makes this easier to dispatch. This fixes git-for-windows#40 [1] git-for-windows/build-extra#488 Signed-off-by: Matthias Aßhauer <mha1993@live.de>
rimrul
added a commit
to rimrul/git-for-windows-automation
that referenced
this pull request
Jul 4, 2024
To prevent parallel uploads of pacman packages from interfering with each other Git for Windows uses Azure blobs leases without a timeout. Sometimes it's necessary to manually break such a lease if a job acquired a lease and failed to release it. We can do that via the pacman-helper script[1], but having it available as a GitHub workflow or eventually a slash command makes this easier to dispatch. This fixes git-for-windows#40 [1] git-for-windows/build-extra#488 Signed-off-by: Matthias Aßhauer <mha1993@live.de>
ammyk9
pushed a commit
to ammyk9/git-for-windows-automation
that referenced
this pull request
Aug 8, 2024
To prevent parallel uploads of pacman packages from interfering with each other Git for Windows uses Azure blobs leases without a timeout. Sometimes it's necessary to manually break such a lease if a job acquired a lease and failed to release it. We can do that via the pacman-helper script[1], but having it available as a GitHub workflow or eventually a slash command makes this easier to dispatch. This fixes git-for-windows#40 [1] git-for-windows/build-extra#488 Signed-off-by: Matthias Aßhauer <mha1993@live.de>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When uploading Pacman packages, we have to take care to stop concurrent uploads from interfering with each other. We do that by acquiring a "lease" to the
git-for-windows.dbfile: Once that lease is acquired by one of the uploading workflow runs, it cannot be acquired by another one until the first one is done and releases that lease.I noticed today that the shiny new
pacman-packagesworkflow had failed to upload the packages. When I edited the definition and then tried to re-run it, the run failed due to some mistake of mine, leaving the lease acquired. So when I edited the definition again and tried to re-re-run it, it could not continue because the lease was already acquired. Similar things happened in the past, and whenever that happens, I go to the Azure Portal and break the lease.However, today I realized that this is some institutionalized knowledge that is locked in my brain, and prevents the Git for Windows project to become more of a team sport than a project that relies solely on @dscho.
So here is the first step to make the "break the lease" task much more transparent (and implicitly documented), with the eventual goal to wire it through as a new slash command in Git for Windows' PRs handled by the GitForWindowsHelper GitHub App, say,
/break pacman-upload-lease(which would trigger a to-be-implemented GitHub workflow in https://github.com/git-for-windows/git-for-windows-automation/ that mirrors a Check Run back to the PR where it was triggered).