Refactor loop AttachFromFile #463
Assignees
Comments
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
Dec 7, 2021
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs#463
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
Dec 8, 2021
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs#463
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
Dec 8, 2021
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs#463
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
Dec 8, 2021
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs#463
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
Dec 8, 2021
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs#463
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
Dec 10, 2021
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs#463
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
Dec 10, 2021
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs#463
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
Dec 10, 2021
Squashed commit of the following: commit 90bbb38 Merge: dbe4f32 8c3143d Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Fri Dec 10 13:41:10 2021 -0600 Merge pull request sylabs#473 from dtrudg/392-docs docs: Update INSTALL.md for 3.9.2 commit dbe4f32 Merge: 0cea468 a26bc7d Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Fri Dec 10 13:40:47 2021 -0600 Merge pull request sylabs#472 from dtrudg/loopdev-39 Pick loop device discovery change sylabs#458 & sylabs#464 commit 8c3143d Author: David Trudgian <david.trudgian@sylabs.io> Date: Fri Dec 10 11:54:57 2021 -0600 docs: Update INSTALL.md for 3.9.2 Match CI to docs update for go 1.17.5 commit a26bc7d Author: David Trudgian <david.trudgian@sylabs.io> Date: Tue Dec 7 17:52:02 2021 -0600 Refactor loop.AttachFromFile The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs#463 commit a88d915 Author: Tom Schoonjans <Tom.Schoonjans@genomicsplc.com> Date: Tue Dec 7 20:52:25 2021 +0000 Apply @dtrudg's comments commit af42c8a Author: Tom Schoonjans <Tom.Schoonjans@genomicsplc.com> Date: Tue Nov 30 20:48:37 2021 +0000 AttachFromFile: rewrite retry loop When the CmdSetStatus64 syscall returns EAGAIN or EBUSY, move on to try the next loop device instead of aborting. Rename AttachFromFile to attachFromFile and run in a loop while no valid loop devices were found but EAGAIN or EBUSY was encountered. This fixes an intermittent issue encountered on Arvados Crunch (https://dev.arvados.org/issues/18489) commit 0cea468 Merge: 3908ad2 b871d84 Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Fri Dec 10 11:12:26 2021 -0600 Merge pull request sylabs#469 from dtrudg/keyidx-39 Clarify source of key index (3.9) commit b871d84 Author: Richard Hattersley <rhattersley@gmail.com> Date: Thu Nov 25 09:29:02 2021 +0000 Clarify source of key index (3.9) commit 3908ad2 Merge: 471b3fc 5c00c11 Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Fri Dec 10 10:11:16 2021 -0600 Merge pull request sylabs#467 from dtrudg/issue465-39 fix: do not require working keyserver config in local build (3.9) commit 5c00c11 Author: David Trudgian <david.trudgian@sylabs.io> Date: Thu Dec 9 12:59:48 2021 -0600 fix: do not require working keyserver config in local build (3.9) At the CLI level, local builds had a hard fail if a keyserver config could not be retrieved. 1) Warn instead of fail. For builds that perform verification of a SIF it may still be possbile to verify using the local keyring. 2) Don't attempt to get a keyserver config at all, unless build definition has a bootstrap that will, or may provide a SIF source image (library/localimage/oras/shub). Fixes sylabs#465 commit 471b3fc Merge: de3dbb7 9fd911d Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Thu Dec 9 16:17:53 2021 -0600 Merge pull request sylabs#470 from dtrudg/clog-shfix docs: Note mvdan.cc/sh related bug fixes in changelog commit 9fd911d Merge: 38b50cb fb84d7d Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Thu Dec 9 15:53:54 2021 -0600 Merge pull request sylabs#468 from dtrudg/godep39 fix: ensure gengodep uses vendor dir if present (3.9) commit de3dbb7 Merge: 38b50cb fb84d7d Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Thu Dec 9 15:53:54 2021 -0600 Merge pull request sylabs#468 from dtrudg/godep39 fix: ensure gengodep uses vendor dir if present (3.9) commit fb84d7d Author: David Trudgian <david.trudgian@sylabs.io> Date: Tue Dec 7 13:20:25 2021 -0600 fix: ensure gengodep uses vendor dir if present (3.9) In a previous PR, initialization of the `srcdir` varible was removed from `makeit/gengodep`. However, the `gengodep` script was still using `srcdir` when checking for a `${srcdir}/vendor/modules.txt`. sylabs@48e7fab As a result, the vendor dir is never found, and `go` was run with `-mod=readonly`, which would lead to downloads. Since go 1.14 the presence of a vendor dir infers `-mod=vendor` automatically, so we can just remove the handling here. Fixes sylabs#461
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
Dec 10, 2021
Squashed commit of the following: commit 90bbb38 Merge: dbe4f32 8c3143d Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Fri Dec 10 13:41:10 2021 -0600 Merge pull request sylabs#473 from dtrudg/392-docs docs: Update INSTALL.md for 3.9.2 commit dbe4f32 Merge: 0cea468 a26bc7d Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Fri Dec 10 13:40:47 2021 -0600 Merge pull request sylabs#472 from dtrudg/loopdev-39 Pick loop device discovery change sylabs#458 & sylabs#464 commit 8c3143d Author: David Trudgian <david.trudgian@sylabs.io> Date: Fri Dec 10 11:54:57 2021 -0600 docs: Update INSTALL.md for 3.9.2 Match CI to docs update for go 1.17.5 commit a26bc7d Author: David Trudgian <david.trudgian@sylabs.io> Date: Tue Dec 7 17:52:02 2021 -0600 Refactor loop.AttachFromFile The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs#463 commit a88d915 Author: Tom Schoonjans <Tom.Schoonjans@genomicsplc.com> Date: Tue Dec 7 20:52:25 2021 +0000 Apply @dtrudg's comments commit af42c8a Author: Tom Schoonjans <Tom.Schoonjans@genomicsplc.com> Date: Tue Nov 30 20:48:37 2021 +0000 AttachFromFile: rewrite retry loop When the CmdSetStatus64 syscall returns EAGAIN or EBUSY, move on to try the next loop device instead of aborting. Rename AttachFromFile to attachFromFile and run in a loop while no valid loop devices were found but EAGAIN or EBUSY was encountered. This fixes an intermittent issue encountered on Arvados Crunch (https://dev.arvados.org/issues/18489) commit 0cea468 Merge: 3908ad2 b871d84 Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Fri Dec 10 11:12:26 2021 -0600 Merge pull request sylabs#469 from dtrudg/keyidx-39 Clarify source of key index (3.9) commit b871d84 Author: Richard Hattersley <rhattersley@gmail.com> Date: Thu Nov 25 09:29:02 2021 +0000 Clarify source of key index (3.9) commit 3908ad2 Merge: 471b3fc 5c00c11 Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Fri Dec 10 10:11:16 2021 -0600 Merge pull request sylabs#467 from dtrudg/issue465-39 fix: do not require working keyserver config in local build (3.9) commit 5c00c11 Author: David Trudgian <david.trudgian@sylabs.io> Date: Thu Dec 9 12:59:48 2021 -0600 fix: do not require working keyserver config in local build (3.9) At the CLI level, local builds had a hard fail if a keyserver config could not be retrieved. 1) Warn instead of fail. For builds that perform verification of a SIF it may still be possbile to verify using the local keyring. 2) Don't attempt to get a keyserver config at all, unless build definition has a bootstrap that will, or may provide a SIF source image (library/localimage/oras/shub). Fixes sylabs#465 commit 471b3fc Merge: de3dbb7 9fd911d Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Thu Dec 9 16:17:53 2021 -0600 Merge pull request sylabs#470 from dtrudg/clog-shfix docs: Note mvdan.cc/sh related bug fixes in changelog commit 9fd911d Merge: 38b50cb fb84d7d Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Thu Dec 9 15:53:54 2021 -0600 Merge pull request sylabs#468 from dtrudg/godep39 fix: ensure gengodep uses vendor dir if present (3.9) commit de3dbb7 Merge: 38b50cb fb84d7d Author: Dave Trudgian <david.trudgian@sylabs.io> Date: Thu Dec 9 15:53:54 2021 -0600 Merge pull request sylabs#468 from dtrudg/godep39 fix: ensure gengodep uses vendor dir if present (3.9) commit fb84d7d Author: David Trudgian <david.trudgian@sylabs.io> Date: Tue Dec 7 13:20:25 2021 -0600 fix: ensure gengodep uses vendor dir if present (3.9) In a previous PR, initialization of the `srcdir` varible was removed from `makeit/gengodep`. However, the `gengodep` script was still using `srcdir` when checking for a `${srcdir}/vendor/modules.txt`. sylabs@48e7fab As a result, the vendor dir is never found, and `go` was run with `-mod=readonly`, which would lead to downloads. Since go 1.14 the presence of a vendor dir infers `-mod=vendor` automatically, so we can just remove the handling here. Fixes sylabs#461
edytuk
pushed a commit
to vzokay/apptainer
that referenced
this issue
Dec 14, 2021
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs/singularity#463 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
This was referenced Dec 14, 2021
cclerget
pushed a commit
to cclerget/apptainer
that referenced
this issue
Dec 16, 2021
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs/singularity#463 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
cclerget
pushed a commit
to cclerget/apptainer
that referenced
this issue
Dec 16, 2021
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs/singularity#463 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch> Signed-off-by: Cédric Clerget <cedric.clerget@gmail.com>
DrDaveD
pushed a commit
to apptainer/apptainer
that referenced
this issue
Dec 16, 2021
* AttachFromFile: rewrite retry loop When the CmdSetStatus64 syscall returns EAGAIN or EBUSY, move on to try the next loop device instead of aborting. Rename AttachFromFile to attachFromFile and run in a loop while no valid loop devices were found but EAGAIN or EBUSY was encountered. This fixes an intermittent issue encountered on Arvados Crunch (https://dev.arvados.org/issues/18489) Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch> Signed-off-by: Cédric Clerget <cedric.clerget@gmail.com> * Apply @dtrudg's comments Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch> Signed-off-by: Cédric Clerget <cedric.clerget@gmail.com> * Refactor loop.AttachFromFile The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs/singularity#463 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch> Signed-off-by: Cédric Clerget <cedric.clerget@gmail.com> * Fix loop device EAGAIN set status error Signed-off-by: Cédric Clerget <cedric.clerget@gmail.com> Co-authored-by: Edita Kizinevic <edita.kizinevic@cern.ch> Co-authored-by: David Trudgian <david.trudgian@sylabs.io>
DrDaveD
pushed a commit
to DrDaveD/singularity
that referenced
this issue
Mar 14, 2022
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs/singularity#463
DrDaveD
pushed a commit
to DrDaveD/singularity
that referenced
this issue
Mar 14, 2022
The loop.AttachFromFile code, used to attach an image to a free loop device, or to find a shared loop device has been refactored. Previously, a complex loop handled both finding shared loop devices and attaching to new free devices. The logic was difficult to follow, especially with regard to tracking which errors were fatal and would bubble up, vs result in loop continuation, or retries. The shared and 'fresh' attachment have been separated into two functions. Some additional logging / error detail has been added. The test code has been modified to prevent a leak of a loop device. Fixes sylabs/singularity#463
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Type of issue
technical debt
Description of issue
In #458 a fix was applied for an issue under arvados where a loop mount from a file can fail with EBUSY... and the most sensible strategy to solve is to move on quickly, rather than holding an exclusive lock on dev in a 'pause and retry' loop etc.
The code is somewhat messy (prior to the contribution)... and could do with a general refactoring.
The text was updated successfully, but these errors were encountered: