Singularity Image Support #125

tri-adam · 2022-04-27T20:20:20Z

This PR is based on the work in #123, and is proposed as a potential way forward based on the feedback there and the discussion in the last community call. @Poluect has unfortunately had to step away from the Sylabs team on short notice, so I'm picking up where he left off.

This PR uses github.com/CalebQ42/squashfs to read the root filesystem contents. This is a pure Go library, and doesn't require a mount.

Structurally, this PR takes a different approach than #123, exposing Singularity Images via the existing func GetImageFromSource rather than adding a SIF-specific func GetSifImageFromSource. This adds a bit more complexity in stereoscope, but keeps the exported API tighter and makes it a lot easier to support SIF in projects consuming this module such as syft (I'll open a PR there shortly that uses pulls in this work.)

Related to anchore/syft#937, sylabs/sif#190

tri-adam · 2022-05-12T17:01:39Z

Hi @wagoodman, just checking in to see if there's anything I can do to push this forward. Thanks!

jonasagx · 2022-05-18T17:10:25Z

Hey @tri-adam thank you for working on this feature, I ran CI validations and there are issues with licenses from these dependencies:

$ make static-analysis
Allow Rules: [BSD.* MIT.* Apache.* MPL.*]
Unallowable license (GPL-2.0) from "github.com/rasky/go-lzo"
Unallowable license (CC0-1.0) from "github.com/therootcompany/xz"

wagoodman · 2022-05-19T14:04:15Z

@tri-adam we're going to review these new licenses internally to ensure we can pull in these dependencies safely (from a legal perspective), I'll report back soon

tri-adam · 2022-05-24T19:56:15Z

Hey @tri-adam thank you for working on this feature, I ran CI validations and there are issues with licenses from these dependencies:
$ make static-analysis
Allow Rules: [BSD.* MIT.* Apache.* MPL.*]
Unallowable license (GPL-2.0) from "github.com/rasky/go-lzo"
Unallowable license (CC0-1.0) from "github.com/therootcompany/xz"

@tri-adam we're going to review these new licenses internally to ensure we can pull in these dependencies safely (from a legal perspective), I'll report back soon

@jonasagx @wagoodman thanks. It looks like both of those are pulled in by github.com/CalebQ42/squashfs:

$ go mod why github.com/rasky/go-lzo github.com/therootcompany/xz
# github.com/rasky/go-lzo
github.com/anchore/stereoscope/pkg/file
github.com/CalebQ42/squashfs
github.com/CalebQ42/squashfs/internal/compression
github.com/rasky/go-lzo

# github.com/therootcompany/xz
github.com/anchore/stereoscope/pkg/file
github.com/CalebQ42/squashfs
github.com/CalebQ42/squashfs/internal/compression
github.com/therootcompany/xz

Let me know the results of the legal review, thanks.

wagoodman · 2022-05-25T13:12:42Z

Hey @tri-adam , sorry to report that GPL-2.0 licensed dependencies cannot be introduced into the codebase at this time (CC0-1.0 is ok to introduce though). Is seems that the LZO license is not something that could be controlled by the repo owner too:

Being a straightforward port of the original source code, it shares the same license (GPLv2) as I can't possibly claim any copyright on it.

There are a couple of paths forward:

we can try to find an lzo alternative go implementation (I couldn't find one after a quick search)
roll our own implementation of LZO (this looks pretty hard, given that LZO is a collection of a dozen or so algorithms that are very poorly documented)
is LZO required for the SIF image functionality? if not, then we could fork the squashfs lib and remove the dependency? (haven't dug into this option at all to check feasibility)
we could head back down the mount PR and work more closely on an opt-in approach
insert-other-idea-here <-- open to more suggestions!

Thanks for bearing with us on this 🙏

LZO support requires the github.com/rasky/go-lzo module, which is GPLv2 licensed and thus not suitable for inclusion in some projects (see discussion in anchore/stereoscope#125).

LZO support requires the github.com/rasky/go-lzo module, which is GPLv2 licensed and thus not suitable for inclusion in some projects (see discussion in anchore/stereoscope#125). Signed-off-by: Adam Hughes <9903835+tri-adam@users.noreply.github.com>

tri-adam · 2022-05-26T22:49:26Z

There are a couple of paths forward:

we can try to find an lzo alternative go implementation (I couldn't find one after a quick search)

roll our own implementation of LZO (this looks pretty hard, given that LZO is a collection of a dozen or so algorithms that are very poorly documented)

is LZO required for the SIF image functionality? if not, then we could fork the squashfs lib and remove the dependency? (haven't dug into this option at all to check feasibility)

we could head back down the mount PR and work more closely on an opt-in approach

insert-other-idea-here <-- open to more suggestions!

Took a stab at option 3, removing LZO support from the underlying module, and it's reasonably easy (CalebQ42/squashfs@v0.5.4...sylabs:remove-lzo-v0.5.4).

SingularityCE does not appear to rely on LZO compression, so I don't foresee any negative impacts in terms of functionality.

Still some further testing to do, but it's a start!

wagoodman · 2022-06-09T13:06:03Z

SingularityCE does not appear to rely on LZO compression, so I don't foresee any negative impacts in terms of functionality.

great news!

go.mod

wagoodman · 2022-06-09T13:16:21Z

pkg/image/source.go

@@ -76,6 +79,8 @@ func ParseSourceScheme(source string) Source {
 		return OciTarballSource
 	case "oci-registry", "registry":
 		return OciRegistrySource
+	case "singularity-image":


I think we could use a shorter name here such as singularity or sif (and "image" is redundant here) -- what do you think @tri-adam ?

I went ahead and shorted it to singularity, however, I'll hold off on pulling the trigger on merging until I hear back from you on this particular change

sif would likely be my suggestion - as that's what has been adopted by support brought into containers/image and hence podman which are using sif as the 'transport' name.

wagoodman

FYI -- force pushed to resolve merge conflict in go.mod/go.sum

wagoodman · 2022-06-09T14:12:37Z

@tri-adam right now integration tests are failing due to lack of image types covered (since singularity was added). The next step is to add testing that covers:

symlinks:

stereoscope/test/integration/fixture_image_symlinks_test.go

Lines 30 to 53 in 9e3b40f

    
           	name   string 
        
           	source string 
        
           }{ 
        
           	{ 
        
           		name:   "FromTarball", 
        
           		source: "docker-archive", 
        
           	}, 
        
           	{ 
        
           		name:   "FromDocker", 
        
           		source: "docker", 
        
           	}, 
        
           	{ 
        
           		name:   "FromPodman", 
        
           		source: "podman", 
        
           	}, 
        
           	{ 
        
           		name:   "FromOciTarball", 
        
           		source: "oci-archive", 
        
           	}, 
        
           	{ 
        
           		name:   "FromOciDirectory", 
        
           		source: "oci-dir", 
        
           	}, 
        
           }

a parsing go case:

stereoscope/test/integration/fixture_image_simple_test.go

Lines 25 to 61 in 9e3b40f

    
           { 
        
           	name:           "FromTarball", 
        
           	source:         "docker-archive", 
        
           	imageMediaType: v1Types.DockerManifestSchema2, 
        
           	layerMediaType: v1Types.DockerLayer, 
        
           	tagCount:       1, 
        
           }, 
        
           { 
        
           	name:           "FromDocker", 
        
           	source:         "docker", 
        
           	imageMediaType: v1Types.DockerManifestSchema2, 
        
           	layerMediaType: v1Types.DockerLayer, 
        
           	// name:hash 
        
           	// name:latest 
        
           	tagCount: 2, 
        
           }, 
        
           { 
        
           	name:           "FromPodman", 
        
           	source:         "podman", 
        
           	imageMediaType: v1Types.DockerManifestSchema2, 
        
           	layerMediaType: v1Types.DockerLayer, 
        
           	tagCount:       2, 
        
           }, 
        
           { 
        
           	name:           "FromOciTarball", 
        
           	source:         "oci-archive", 
        
           	imageMediaType: v1Types.OCIManifestSchema1, 
        
           	layerMediaType: v1Types.OCILayer, 
        
           	tagCount:       0, 
        
           }, 
        
           { 
        
           	name:           "FromOciDirectory", 
        
           	source:         "oci-dir", 
        
           	imageMediaType: v1Types.OCIManifestSchema1, 
        
           	layerMediaType: v1Types.OCILayer, 
        
           	tagCount:       0, 
        
           },

I think the one thing I'm hung up on is that these tests depend on building the same image and obtaining it in different ways -- is there a way to nicely achieve that in testing with singularity images? If not, that's ok, we can make one off tests to assert this functionality (in integration or unit-level testing).

Shout out if there is anything I can do to clarify or help!

dtrudg · 2022-06-10T20:16:03Z

Hi @wagoodman. Thanks for the review and comments on this! I work with @tri-adam at Sylabs and might be picking this work up from him for a short time.

I'll look into the tests in detail as soon as possible. With regard to building a SIF, within the workflow, that would be possible from the docker image that is generated here with an install of Singularity. We do package SingularityCE ourselves for Ubuntu, with those packages hosted on GitHub releases ... but I'm not sure whether you'd want to add an install into the CI workflows?

A final thought is that we could get close enough to a 'real' Singularity built SIF by directly leveraging mksquashfs and calling the github.com/sylabs/sif SIF code we are importing to construct a simple SIF.... but perhaps this is a bit too far removed from the intent to test against a real Singularity container image?

wagoodman · 2022-06-21T13:03:13Z

hey @dtrudg 👋

a. [it] would be possible from the docker image that is generated here with an install of Singularity...I'm not sure whether you'd want to add an install into the CI workflows?

b. ... we could get close enough to a 'real' Singularity built SIF by directly leveraging mksquashfs and calling the github.com/sylabs/sif SIF code we are importing to construct a simple SIF.

I'm ok with either approach, my only hesitation with option a is if the install time is long enough to make PR validations much slower --but if that is the more kosher approach then that might be an OK trade off.

The goal of any approach is to make certain that the content API of stereoscope functions with the new image type in a realistic setting (querying files by a glob or path, getting file contents, resolving symlinks properly, etc).

What's the approach that you suggest / lean towards?

dtrudg · 2022-06-21T14:08:23Z

a. [it] would be possible from the docker image that is generated here with an install of Singularity...I'm not sure whether you'd want to add an install into the CI workflows?

b. ... we could get close enough to a 'real' Singularity built SIF by directly leveraging mksquashfs and calling the github.com/sylabs/sif SIF code we are importing to construct a simple SIF.

I'm ok with either approach, my only hesitation with option a is if the install time is long enough to make PR validations much slower --but if that is the more kosher approach then that might be an OK trade off.

The goal of any approach is to make certain that the content API of stereoscope functions with the new image (querying files by a glob or path, getting file contents, resolving symlinks properly, etc).

What's the approach that you suggest / lean towards?

Thanks @wagoodman - I can work it up with (a) and if the impact is too high we can switch to (b). It would be nicer to actually build the container with 'real' Singularity, than construct it. ourselves. Hopefully I'll have time to put something together for this during the week here.

dtrudg · 2022-06-27T16:16:18Z

Hi @wagoodman - I've taken a go at fitting some testing for singularity SIFs in with the existing image_simple / image_symlinks.

Would be grateful for any feedback on whether this approach looks acceptable, and worth tidying up further, or we should go for separate tests. Many thanks for your patience!

dtrudg · 2022-06-28T19:26:05Z

I'm sorry for the test failure due to the apt install... that was a 🤦

Signed-off-by: Dmitry Harkavyi <dmytro.harkavyi@sylabs.io>

Use github.com/CalebQ42/squashfs module to read the contents of Singularity Image Format (SIF) container images without mounting them. Expose Singularity Images via GetImageFromSource rather than adding a SIF-specific GetSifImageFromSource by providing a GGCR v1.Image implementation backed by a SIF. This avoids the requirement of separate code paths within consumers of stereoscope such as syft. Signed-off-by: Adam Hughes <stickmanica@gmail.com>

Switch to fork of github.com/CalebQ42/squashfs, which removes the GPLv2-licensed indirect github.com/rasky/go-lzo dependency. Signed-off-by: Adam Hughes <9903835+tri-adam@users.noreply.github.com>

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

Modify the TestSimpleImage and TestImageSymlinks code to incorporate testing of a singularity sif image source. This requires signficant adaptations as singularity squashes containers down to a single layer. Singularity is expected to be available, and is is now installed in the ci-bootstrap Makefile target (from a GitHub release). Signed-off-by: David Trudgian <david.trudgian@sylabs.io>

dtrudg · 2022-08-01T20:13:48Z

Hi - I've re-based again onto the latest main. Looks like the last CI run prior to the rebase did pass, so would be grateful for any feedback on the testing approach, and what cleanup might be needed here. Many thanks!

wagoodman · 2022-08-01T20:55:52Z

sorry for the delay @dtrudg , approved the workflow and are reviewing the changes now.

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

wagoodman

The only change I made was making the singularity call more portable to help development on macs -- nothing too major. Thanks for the enhancement and updating the tests @tri-adam @dtrudg !!

dtrudg · 2022-08-02T14:09:51Z

Many thanks @wagoodman - I'll take a look at the related syft PR next.

Thanks for your help, and apologies this was pretty slow from our side.

wagoodman · 2022-08-02T14:24:01Z

@dtrudg no need to apologize, this slid on my side as well 👍 I'm looking at anchore/syft#974 and just pulled in the latest stereoscope and ~~I think I'm seeing some integration issues, but too soon to tell really. I'll dig further and see what comes up.~~ it works great (I spoke too soon 😄 )

wagoodman · 2022-08-02T14:33:09Z

@dtrudg I updated anchore/syft#974 and it seems to be working great -- did you want to add anything or give it a shot before merging?

dtrudg · 2022-08-02T14:38:37Z

@dtrudg I updated anchore/syft#974 and it seems to be working great -- did you want to add anything or give it a shot before merging?

@wagoodman I'll take a quick check within the next couple of hours when I'm out of a couple of calls, and ping you over on that PR if that's okay? I think it should be good but don't want to overlook anything given I'm picking it up from tri-adam :-)

tri-adam marked this pull request as ready for review April 27, 2022 20:28

tri-adam mentioned this pull request Apr 28, 2022

Singularity Image Support anchore/syft#974

Merged

wagoodman self-requested a review May 19, 2022 19:11

wagoodman self-assigned this May 19, 2022

tri-adam force-pushed the singularity branch from e2db6df to 9b1acbe Compare May 26, 2022 23:30

wagoodman reviewed Jun 9, 2022

View reviewed changes

go.mod Show resolved Hide resolved

wagoodman reviewed Jun 9, 2022

View reviewed changes

wagoodman force-pushed the singularity branch from bd21d86 to 295170a Compare June 9, 2022 13:53

wagoodman reviewed Jun 9, 2022

View reviewed changes

wagoodman force-pushed the singularity branch from 295170a to 83a74e2 Compare June 9, 2022 13:58

This was referenced Jun 9, 2022

add sylabs SIF image support anchore/syft#940

Closed

Add client method to support sylabs SIF images #123

Closed

dtrudg force-pushed the singularity branch from 24c442d to bdf49c9 Compare June 27, 2022 16:14

dtrudg force-pushed the singularity branch from bdf49c9 to f015448 Compare June 28, 2022 19:24

Add client method to support SIF images

376f18f

Signed-off-by: Dmitry Harkavyi <dmytro.harkavyi@sylabs.io>

Adam Hughes and others added 6 commits August 1, 2022 15:10

fix: remove indirect GPLv2 dependency

4d0db84

Switch to fork of github.com/CalebQ42/squashfs, which removes the GPLv2-licensed indirect github.com/rasky/go-lzo dependency. Signed-off-by: Adam Hughes <9903835+tri-adam@users.noreply.github.com>

update docs to reflect support for singularity

017492d

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

shorten singularity scheme name

3d9a9d9

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

allow CC0-1.0 license

64c7547

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

dtrudg force-pushed the singularity branch from f015448 to 72e0ece Compare August 1, 2022 20:11

move sigularity to a container for portability

881a056

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

wagoodman force-pushed the singularity branch from 4679c3c to 881a056 Compare August 2, 2022 13:48

wagoodman approved these changes Aug 2, 2022

View reviewed changes

wagoodman merged commit 49d33a1 into anchore:main Aug 2, 2022

tri-adam deleted the singularity branch August 8, 2022 14:05

tri-adam mentioned this pull request Aug 23, 2022

Syft failed to parse Singularity image anchore/syft#1150

Closed

tri-adam mentioned this pull request Sep 14, 2023

Deprecate pkg/user package? sylabs/sif#323

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Singularity Image Support #125

Singularity Image Support #125

tri-adam commented Apr 27, 2022 •

edited

tri-adam commented May 12, 2022

jonasagx commented May 18, 2022 •

edited

wagoodman commented May 19, 2022

tri-adam commented May 24, 2022 •

edited

wagoodman commented May 25, 2022 •

edited

tri-adam commented May 26, 2022 •

edited

wagoodman commented Jun 9, 2022

wagoodman Jun 9, 2022

wagoodman Jun 9, 2022

dtrudg Jun 10, 2022

wagoodman left a comment

wagoodman commented Jun 9, 2022 •

edited

dtrudg commented Jun 10, 2022

wagoodman commented Jun 21, 2022 •

edited

dtrudg commented Jun 21, 2022

dtrudg commented Jun 27, 2022

dtrudg commented Jun 28, 2022

dtrudg commented Aug 1, 2022

wagoodman commented Aug 1, 2022

wagoodman left a comment

dtrudg commented Aug 2, 2022 •

edited

wagoodman commented Aug 2, 2022 •

edited

wagoodman commented Aug 2, 2022

dtrudg commented Aug 2, 2022

Singularity Image Support #125

Singularity Image Support #125

Conversation

tri-adam commented Apr 27, 2022 • edited

tri-adam commented May 12, 2022

jonasagx commented May 18, 2022 • edited

wagoodman commented May 19, 2022

tri-adam commented May 24, 2022 • edited

wagoodman commented May 25, 2022 • edited

tri-adam commented May 26, 2022 • edited

wagoodman commented Jun 9, 2022

wagoodman Jun 9, 2022

Choose a reason for hiding this comment

wagoodman Jun 9, 2022

Choose a reason for hiding this comment

dtrudg Jun 10, 2022

Choose a reason for hiding this comment

wagoodman left a comment

Choose a reason for hiding this comment

wagoodman commented Jun 9, 2022 • edited

dtrudg commented Jun 10, 2022

wagoodman commented Jun 21, 2022 • edited

dtrudg commented Jun 21, 2022

dtrudg commented Jun 27, 2022

dtrudg commented Jun 28, 2022

dtrudg commented Aug 1, 2022

wagoodman commented Aug 1, 2022

wagoodman left a comment

Choose a reason for hiding this comment

dtrudg commented Aug 2, 2022 • edited

wagoodman commented Aug 2, 2022 • edited

wagoodman commented Aug 2, 2022

dtrudg commented Aug 2, 2022

tri-adam commented Apr 27, 2022 •

edited

jonasagx commented May 18, 2022 •

edited

tri-adam commented May 24, 2022 •

edited

wagoodman commented May 25, 2022 •

edited

tri-adam commented May 26, 2022 •

edited

wagoodman commented Jun 9, 2022 •

edited

wagoodman commented Jun 21, 2022 •

edited

dtrudg commented Aug 2, 2022 •

edited

wagoodman commented Aug 2, 2022 •

edited