Singularity Image Support (#125)

* Add client method to support SIF images

Signed-off-by: Dmitry Harkavyi <dmytro.harkavyi@sylabs.io>

* refactor: read Singularity images without mount

Use github.com/CalebQ42/squashfs module to read the contents of
Singularity Image Format (SIF) container images without mounting them.

Expose Singularity Images via GetImageFromSource rather than adding a
SIF-specific GetSifImageFromSource by providing a GGCR v1.Image
implementation backed by a SIF. This avoids the requirement of separate
code paths within consumers of stereoscope such as syft.

Signed-off-by: Adam Hughes <stickmanica@gmail.com>

* fix: remove indirect GPLv2 dependency

Switch to fork of github.com/CalebQ42/squashfs, which removes the
GPLv2-licensed indirect github.com/rasky/go-lzo dependency.

Signed-off-by: Adam Hughes <9903835+tri-adam@users.noreply.github.com>

* update docs to reflect support for singularity

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* shorten singularity scheme name

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* allow CC0-1.0 license

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* test: sif image_simple / image_symlinks adaptations

Modify the TestSimpleImage and TestImageSymlinks code to incorporate
testing of a singularity sif image source.

This requires signficant adaptations as singularity squashes
containers down to a single layer.

Singularity is expected to be available, and is is now installed in
the ci-bootstrap Makefile target (from a GitHub release).

Signed-off-by: David Trudgian <david.trudgian@sylabs.io>

* move sigularity to a container for portability

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

Co-authored-by: Dmitry Harkavyi <dmytro.harkavyi@sylabs.io>
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
Co-authored-by: David Trudgian <david.trudgian@sylabs.io>

.bouncer.yaml

@@ -2,4 +2,5 @@ permit:
   - BSD.*
   - MIT.*
   - Apache.*
-  - MPL.*
+  - MPL.*
+  - CC0-1.0

.github/workflows/validations.yaml

@@ -151,15 +151,29 @@ jobs:
       - name: Bootstrap CI environment dependencies
         run: make ci-bootstrap
 
-      - name: Build key for tar cache
+      - name: Build key for test-fixture cache
         run: make integration-fingerprint
 
-      - name: Restore integration test cache
+      - name: Restore integration test-fixture cache
         uses: actions/cache@v2.1.3
         with:
           path: ${{ github.workspace }}/test/integration/test-fixtures/cache
           key: ${{ runner.os }}-integration-test-cache-${{ hashFiles('test/integration/test-fixtures/cache.fingerprint') }}
-
+
+      - name: Build key for tool cache
+        run: make integration-tools-fingerprint
+
+      - name: Restore integration tool cache
+        id: integration-tool-cache
+        uses: actions/cache@v2.1.3
+        with:
+          path: ${{ github.workspace }}/test/integration/tools/cache
+          key: ${{ runner.os }}-integration-tools-cache-${{ hashFiles('test/integration/tools/cache.fingerprint') }}
+
+      - name: (cache-hit) Load integration tool cache
+        if: steps.integration-tool-cache.outputs.cache-hit == 'true'
+        run: make integration-tools-load
+
       - name: Run integration tests
         run: make integration
 

.gitignore

@@ -8,6 +8,7 @@
 *.DS_Store
 coverage.txt
 **/test-fixtures/cache/
+**/*.fingerprint
 
 # Binaries for programs and plugins
 *.exe

Makefile

@@ -44,6 +44,7 @@ help:
 .PHONY: ci-bootstrap
 ci-bootstrap: bootstrap
 	sudo apt install -y bc
+	curl -sLO https://github.com/sylabs/singularity/releases/download/v3.10.0/singularity-ce_3.10.0-focal_amd64.deb && sudo apt-get install -y -f ./singularity-ce_3.10.0-focal_amd64.deb
 
 $(RESULTSDIR):
 	mkdir -p $(RESULTSDIR)
@@ -110,8 +111,24 @@ show-benchstat:
 integration-fingerprint:
 	find test/integration/test-fixtures/image-* -type f -exec md5sum {} + | awk '{print $1}' | sort | md5sum | tee test/integration/test-fixtures/cache.fingerprint
 
+.PHONY: integration-tools-fingerprint
+integration-tools-fingerprint:
+	@cd test/integration/tools && make fingerprint
+
+.PHONY: integration-tools
+integration-tools:
+	@cd test/integration/tools && make
+
+.PHONY: integration-tools
+integration-tools-load:
+	@cd test/integration/tools && make load-cache
+
+.PHONY: integration-tools
+integration-tools-save:
+	@cd test/integration/tools && make save-cache
+
 .PHONY: integration
-integration: ## Run integration tests
+integration: integration-tools ## Run integration tests
 	$(call title,Running integration tests)
 	go test -v ./test/integration
 

README.md

@@ -21,7 +21,10 @@ Note: To run tests you will need `skopeo` installed.
 ## Overview
 
 This library provides the means to:
-- parse and read images from multiple sources (currently docker V2 schema images read from the docker daemon and from an archive on disk)
+- parse and read images from multiple sources, supporting:
+  - docker V2 schema images from the docker daemon, podman, or archive
+  - OCI images from disk, directory, or registry
+  - singularity formatted image files
 - build a file tree representing each layer blob
 - create a squashed file tree representation for each layer
 - search one or more file trees for selected paths

client.go

@@ -12,6 +12,7 @@ import (
 	"github.com/anchore/stereoscope/pkg/image"
 	"github.com/anchore/stereoscope/pkg/image/docker"
 	"github.com/anchore/stereoscope/pkg/image/oci"
+	"github.com/anchore/stereoscope/pkg/image/sif"
 	"github.com/anchore/stereoscope/pkg/logger"
 	"github.com/wagoodman/go-partybus"
 )
@@ -132,6 +133,11 @@ func selectImageProvider(imgStr string, source image.Source, cfg config) (image.
 		provider = oci.NewProviderFromTarball(imgStr, tempDirGenerator)
 	case image.OciRegistrySource:
 		provider = oci.NewProviderFromRegistry(imgStr, tempDirGenerator, cfg.Registry, cfg.Platform)
+	case image.SingularitySource:
+		if cfg.Platform != nil {
+			return nil, platformSelectionUnsupported
+		}
+		provider = sif.NewProviderFromPath(imgStr, tempDirGenerator)
 	default:
 		return nil, fmt.Errorf("unable determine image source")
 	}

go.mod

@@ -3,6 +3,7 @@ module github.com/anchore/stereoscope
 go 1.16
 
 require (
+	github.com/CalebQ42/squashfs v0.5.4
 	github.com/GoogleCloudPlatform/docker-credential-gcr v2.0.5+incompatible
 	github.com/anchore/go-testutils v0.0.0-20200925183923-d5f45b0d3c04
 	github.com/awslabs/amazon-ecr-credential-helper/ecr-login v0.0.0-20220517224237-e6f29200ae04
@@ -22,11 +23,15 @@ require (
 	github.com/pkg/errors v0.9.1
 	// pinned to pull in 386 arch fix: https://github.com/scylladb/go-set/commit/cc7b2070d91ebf40d233207b633e28f5bd8f03a5
 	github.com/scylladb/go-set v1.0.3-0.20200225121959-cc7b2070d91e
-	github.com/sergi/go-diff v1.1.0
+	github.com/sergi/go-diff v1.2.0
 	github.com/sirupsen/logrus v1.8.1
 	github.com/spf13/afero v1.6.0
 	github.com/stretchr/testify v1.7.0
+	github.com/sylabs/sif/v2 v2.7.0
 	github.com/wagoodman/go-partybus v0.0.0-20200526224238-eb215533f07d
 	github.com/wagoodman/go-progress v0.0.0-20200621122631-1a2120f0695a
-	golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
+	golang.org/x/crypto v0.0.0-20210921155107-089bfa567519
 )
+
+// Forked to remove https://github.com/rasky/go-lzo dependency, which is GPLv2 licensed.
+replace github.com/CalebQ42/squashfs => github.com/sylabs/squashfs v0.5.5-0.20220526223455-67e0f4cd95c5