Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: error on create if an index sha is used #2429

Merged
merged 52 commits into from
Apr 25, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
7d1988d
core of what I'm looking for working
AustinAbro321 Apr 9, 2024
c7eae77
simplify
AustinAbro321 Apr 9, 2024
5626a12
Merge branch 'main' into band-aid-index-sha
AustinAbro321 Apr 9, 2024
5dfe4dc
whitespace
AustinAbro321 Apr 9, 2024
505e4c8
Merge branch 'band-aid-index-sha' of github.com:defenseunicorns/zarf …
AustinAbro321 Apr 9, 2024
48ab849
platform has string() already
AustinAbro321 Apr 9, 2024
98c1a07
whitespace
AustinAbro321 Apr 9, 2024
f8aaf32
remove unneeded comment
AustinAbro321 Apr 9, 2024
7477c1a
merge
AustinAbro321 Apr 11, 2024
074b56c
retry with cancel implemented
AustinAbro321 Apr 11, 2024
a50fdda
Merge branch 'main' into band-aid-index-sha
AustinAbro321 Apr 15, 2024
d451143
Merge branch 'main' into band-aid-index-sha
AustinAbro321 Apr 15, 2024
5a1a114
Merge branch 'band-aid-index-sha' of github.com:defenseunicorns/zarf …
AustinAbro321 Apr 15, 2024
948dd21
Merge branch 'main' into band-aid-index-sha
AustinAbro321 Apr 16, 2024
4c862c9
changed error message
AustinAbro321 Apr 16, 2024
9e29b74
deleting unnecessary check
AustinAbro321 Apr 16, 2024
1346af6
fix test
AustinAbro321 Apr 16, 2024
ab508c2
changing how we determine if an image is an index list
AustinAbro321 Apr 16, 2024
1f5fe0d
test tablify
AustinAbro321 Apr 16, 2024
d31acd6
change format
AustinAbro321 Apr 17, 2024
d83f159
switching to refinfo
AustinAbro321 Apr 17, 2024
efd6a80
fix yq test
AustinAbro321 Apr 17, 2024
e16927f
mod tidy
AustinAbro321 Apr 17, 2024
d18b3d8
yq test cleanup
AustinAbro321 Apr 17, 2024
b61cbde
comment
AustinAbro321 Apr 17, 2024
c7550a3
Merge branch 'main' into band-aid-index-sha
Noxsios Apr 18, 2024
b538c55
use helpers release
AustinAbro321 Apr 19, 2024
a3aa14d
new compare workflow
AustinAbro321 Apr 19, 2024
22b0817
script header
AustinAbro321 Apr 19, 2024
feec052
checkout main
AustinAbro321 Apr 19, 2024
4b50f6b
fetch-depth 0
AustinAbro321 Apr 19, 2024
91f5401
fetch main
AustinAbro321 Apr 19, 2024
8131656
whoops
AustinAbro321 Apr 19, 2024
6f9346b
check
AustinAbro321 Apr 19, 2024
66a2e52
check
AustinAbro321 Apr 19, 2024
c6230b7
check vulnerabilities
AustinAbro321 Apr 19, 2024
355fd63
main-syft.json
AustinAbro321 Apr 19, 2024
b14a4c2
check
AustinAbro321 Apr 19, 2024
57088c5
check
AustinAbro321 Apr 19, 2024
e86f8e5
check
AustinAbro321 Apr 19, 2024
6cfd036
check
AustinAbro321 Apr 19, 2024
4d25896
check
AustinAbro321 Apr 19, 2024
35f189b
fix comment
AustinAbro321 Apr 22, 2024
74d7f52
removing comapre cve things
AustinAbro321 Apr 22, 2024
4889566
Merge branch 'main' into band-aid-index-sha
Noxsios Apr 22, 2024
cb33490
Merge branch 'main' into band-aid-index-sha
AustinAbro321 Apr 23, 2024
d4d5121
yq whitespace
AustinAbro321 Apr 23, 2024
a735e16
Merge branch 'main' into band-aid-index-sha
AustinAbro321 Apr 23, 2024
b1336ae
merge
AustinAbro321 Apr 25, 2024
d862a03
Merge branch 'main' into band-aid-index-sha
AustinAbro321 Apr 25, 2024
675ff5c
Merge branch 'main' into band-aid-index-sha
AustinAbro321 Apr 25, 2024
4420e1d
mod tidy
AustinAbro321 Apr 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ require (
github.com/anchore/clio v0.0.0-20240307182142-fb5fc4c9db3c
github.com/anchore/stereoscope v0.0.1
github.com/anchore/syft v0.100.0
github.com/defenseunicorns/pkg/helpers v1.0.0
github.com/defenseunicorns/pkg/helpers v1.0.1-0.20240411154501-d8e8145af112
github.com/defenseunicorns/pkg/oci v0.0.1
github.com/derailed/k9s v0.31.7
github.com/distribution/reference v0.5.0
Expand Down
4 changes: 4 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -595,6 +595,10 @@ github.com/defenseunicorns/gojsonschema v0.0.0-20231116163348-e00f069122d6 h1:gw
github.com/defenseunicorns/gojsonschema v0.0.0-20231116163348-e00f069122d6/go.mod h1:StKLYMmPj1R5yIs6CK49EkcW1TvUYuw5Vri+LRk7Dy8=
github.com/defenseunicorns/pkg/helpers v1.0.0 h1:0o3Rs+J/g0UemZHcENBS1Z2Qw2y4FIUUrGs75iEyPb4=
github.com/defenseunicorns/pkg/helpers v1.0.0/go.mod h1:F4S5VZLDrlNWQKklzv4v9tFWjjZNhxJ1gT79j4XiLwk=
github.com/defenseunicorns/pkg/helpers v1.0.1-0.20240411133953-c27224059905 h1:x4KxchKd18SoYFAb/UmqpsJVhOVMB1VYJsVxt7w3s9s=
github.com/defenseunicorns/pkg/helpers v1.0.1-0.20240411133953-c27224059905/go.mod h1:F4S5VZLDrlNWQKklzv4v9tFWjjZNhxJ1gT79j4XiLwk=
github.com/defenseunicorns/pkg/helpers v1.0.1-0.20240411154501-d8e8145af112 h1:Z6MQCjpKBA1CQfr3XqaRQsjXt+LYtNzfT4Adm53mfyE=
github.com/defenseunicorns/pkg/helpers v1.0.1-0.20240411154501-d8e8145af112/go.mod h1:F4S5VZLDrlNWQKklzv4v9tFWjjZNhxJ1gT79j4XiLwk=
github.com/defenseunicorns/pkg/oci v0.0.1 h1:EFRp3NeiwzhOWKpQ6mAxi0l9chnrAvDcIgjMr0o0fkM=
github.com/defenseunicorns/pkg/oci v0.0.1/go.mod h1:zVBgRjckEAhfdvbnQrnfOP/3M/GYJkIgWtJtY7pjYdo=
github.com/deitch/magic v0.0.0-20230404182410-1ff89d7342da h1:ZOjWpVsFZ06eIhnh4mkaceTiVoktdU67+M7KDHJ268M=
Expand Down
15 changes: 7 additions & 8 deletions site/src/content/docs/commands/zarf_tools_yq_eval.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@ tableOfContents: false

### Synopsis

yq is a portable command-line data file processor (https://github.com/mikefarah/yq/)
yq is a portable command-line data file processor (https://github.com/mikefarah/yq/)
See https://mikefarah.gitbook.io/yq/ for detailed documentation and examples.

## Evaluate Sequence ##
This command iterates over each yaml document from each given file, applies the
This command iterates over each yaml document from each given file, applies the
expression and prints the result in sequence.

```
Expand All @@ -28,21 +28,21 @@ zarf tools yq eval [expression] [yaml_file1]... [flags]
```

# Reads field under the given path for each file
zarf tools yq e '.a.b' f1.yml f2.yml
zarf tools yq e '.a.b' f1.yml f2.yml

# Prints out the file
zarf tools yq e sample.yaml
zarf tools yq e sample.yaml

# Pipe from STDIN
## use '-' as a filename to pipe from STDIN
cat file2.yml | zarf tools yq e '.a.b' file1.yml - file3.yml

# Creates a new yaml document
## Note that editing an empty file does not work.
zarf tools yq e -n '.a.b.c = "cat"'
zarf tools yq e -n '.a.b.c = "cat"'

# Update a file inplace
zarf tools yq e '.a.b = "cool"' -i file.yaml
# Update a file in place
zarf tools yq e '.a.b = "cool"' -i file.yaml

```

Expand Down Expand Up @@ -98,4 +98,3 @@ zarf tools yq e '.a.b = "cool"' -i file.yaml
### SEE ALSO

* [zarf tools yq](/commands/zarf_tools_yq/) - yq is a lightweight and portable command-line data file processor.

9 changes: 4 additions & 5 deletions src/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -134,12 +134,11 @@ func GetCraneOptions(insecure bool, archs ...string) []crane.Option {
options = append(options, crane.Insecure, crane.WithTransport(roundTripper))
}

// Add the image platform info
if archs != nil {
options = append(options, crane.WithPlatform(&v1.Platform{OS: "linux", Architecture: GetArch(archs...)}))
}

options = append(options,
crane.WithPlatform(&v1.Platform{
OS: "linux",
Architecture: GetArch(archs...),
}),
crane.WithUserAgent("zarf"),
crane.WithNoClobber(true),
// TODO: (@WSTARR) this is set to limit pushes to registry pods and reduce the likelihood that crane will get stuck.
Expand Down
19 changes: 10 additions & 9 deletions src/config/lang/english.go
Original file line number Diff line number Diff line change
Expand Up @@ -506,21 +506,21 @@ cat file2.yml | zarf tools yq ea '.a.b' file1.yml - file3.yml
`
CmdToolsYqEvalExample = `
# Reads field under the given path for each file
zarf tools yq e '.a.b' f1.yml f2.yml
zarf tools yq e '.a.b' f1.yml f2.yml

# Prints out the file
zarf tools yq e sample.yaml
zarf tools yq e sample.yaml

# Pipe from STDIN
## use '-' as a filename to pipe from STDIN
cat file2.yml | zarf tools yq e '.a.b' file1.yml - file3.yml

# Creates a new yaml document
## Note that editing an empty file does not work.
zarf tools yq e -n '.a.b.c = "cat"'
zarf tools yq e -n '.a.b.c = "cat"'

# Update a file inplace
zarf tools yq e '.a.b = "cool"' -i file.yaml
# Update a file in place
zarf tools yq e '.a.b = "cool"' -i file.yaml
`
CmdToolsMonitorShort = "Launches a terminal UI to monitor the connected cluster using K9s."

Expand Down Expand Up @@ -728,10 +728,11 @@ const (

// Collection of reusable error messages.
var (
ErrInitNotFound = errors.New("this command requires a zarf-init package, but one was not found on the local system. Re-run the last command again without '--confirm' to download the package")
ErrUnableToCheckArch = errors.New("unable to get the configured cluster's architecture")
ErrInterrupt = errors.New("execution cancelled due to an interrupt")
ErrUnableToGetPackages = errors.New("unable to load the Zarf Package data from the cluster")
ErrInitNotFound = errors.New("this command requires a zarf-init package, but one was not found on the local system. Re-run the last command again without '--confirm' to download the package")
ErrUnableToCheckArch = errors.New("unable to get the configured cluster's architecture")
ErrInterrupt = errors.New("execution cancelled due to an interrupt")
ErrUnableToGetPackages = errors.New("unable to load the Zarf Package data from the cluster")
ErrUnsupportedImageType = errors.New("zarf does not currently support image indexes or docker manifest lists")
)

// Collection of reusable warn messages.
Expand Down
19 changes: 18 additions & 1 deletion src/internal/packager/images/pull.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,14 @@ package images

import (
"context"
"encoding/json"
"fmt"
"path/filepath"
"strings"

"github.com/defenseunicorns/pkg/helpers"
"github.com/defenseunicorns/zarf/src/config"
"github.com/defenseunicorns/zarf/src/config/lang"
"github.com/defenseunicorns/zarf/src/pkg/layout"
"github.com/defenseunicorns/zarf/src/pkg/message"
"github.com/defenseunicorns/zarf/src/pkg/transform"
Expand All @@ -25,6 +27,7 @@ import (
"github.com/google/go-containerregistry/pkg/v1/empty"
clayout "github.com/google/go-containerregistry/pkg/v1/layout"
"github.com/google/go-containerregistry/pkg/v1/partial"
ctypes "github.com/google/go-containerregistry/pkg/v1/types"
"github.com/moby/moby/client"
)

Expand Down Expand Up @@ -271,7 +274,7 @@ func (i *ImageConfig) PullImage(src string, spinner *message.Spinner) (img v1.Im
if err != nil {
return nil, false, err
}
} else if _, err := crane.Manifest(src, config.GetCraneOptions(i.Insecure, i.Architectures...)...); err != nil {
} else if manifest, err := crane.Manifest(src, config.GetCraneOptions(i.Insecure)...); err != nil {
// If crane is unable to pull the image, try to load it from the local docker daemon.
message.Notef("Falling back to local 'docker' images, failed to find the manifest on a remote: %s", err.Error())

Expand Down Expand Up @@ -308,6 +311,20 @@ func (i *ImageConfig) PullImage(src string, spinner *message.Spinner) (img v1.Im
return nil, false, fmt.Errorf("failed to load image from docker daemon: %w", err)
}
} else {
var idx v1.IndexManifest
if err := json.Unmarshal(manifest, &idx); err != nil {
AustinAbro321 marked this conversation as resolved.
Show resolved Hide resolved
return nil, false, lang.ErrUnsupportedImageType
}

if strings.Contains(src, "@") && (idx.MediaType == ctypes.OCIImageIndex || idx.MediaType == ctypes.DockerManifestList) {
imageOptions := "please select one of the images below based on your platform"
lucasrod16 marked this conversation as resolved.
Show resolved Hide resolved
imageBaseName := strings.Split(src, "@")[0]
for _, manifest := range idx.Manifests {
imageOptions = fmt.Sprintf("%s\n %s@%s for platform %s", imageOptions, imageBaseName, manifest.Digest, manifest.Platform)
}
return nil, false, fmt.Errorf("%w: %s", lang.ErrUnsupportedImageType, imageOptions)
}

// Manifest was found, so use crane to pull the image.
if img, err = crane.Pull(src, config.GetCraneOptions(i.Insecure, i.Architectures...)...); err != nil {
return nil, false, fmt.Errorf("failed to pull image: %w", err)
Expand Down
9 changes: 7 additions & 2 deletions src/pkg/packager/creator/normal.go
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@ func (pc *PackageCreator) Assemble(dst *layout.PackagePaths, components []types.
var pulled []images.ImgInfo
var err error

ctx, cancel := context.WithCancel(context.TODO())
doPull := func() error {
imgConfig := images.ImageConfig{
ImagesPath: dst.Images.Base,
Expand All @@ -186,11 +187,15 @@ func (pc *PackageCreator) Assemble(dst *layout.PackagePaths, components []types.
}

pulled, err = imgConfig.PullAll()
if errors.Is(err, lang.ErrUnsupportedImageType) {
schristoff marked this conversation as resolved.
Show resolved Hide resolved
cancel()
}

return err
}

if err := helpers.Retry(doPull, 3, 5*time.Second, message.Warnf); err != nil {
return fmt.Errorf("unable to pull images after 3 attempts: %w", err)
if err := helpers.RetryWithContext(ctx, doPull, 3, 5*time.Second, message.Warnf); err != nil {
return fmt.Errorf("unable to pull images: %w", err)
}

for _, imgInfo := range pulled {
Expand Down
24 changes: 24 additions & 0 deletions src/test/e2e/14_create_sha_index_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: 2021-Present The Zarf Authors

// Package test provides e2e tests for Zarf.
package test

import (
"testing"

"github.com/stretchr/testify/require"
)

func TestCreateIndexShaErrors(t *testing.T) {
AustinAbro321 marked this conversation as resolved.
Show resolved Hide resolved
t.Log("E2E: CreateIndexShaErrors")

_, stderr, err := e2e.Zarf("package", "create", "src/test/packages/14-index-sha/image-index", "--confirm")
require.Error(t, err)
require.Contains(t, stderr, "ghcr.io/defenseunicorns/zarf/agent:v0.32.6@sha256:b3fabdc7d4ecd0f396016ef78da19002c39e3ace352ea0ae4baa2ce9d5958376")

_, stderr, err = e2e.Zarf("package", "create", "src/test/packages/14-index-sha/manifest-list", "--confirm")
require.Error(t, err)
require.Contains(t, stderr, "docker.io/defenseunicorns/zarf-game@sha256:f78e442f0f3eb3e9459b5ae6b1a8fda62f8dfe818112e7d130a4e8ae72b3cbff")

}
9 changes: 9 additions & 0 deletions src/test/packages/14-index-sha/image-index/zarf.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
kind: ZarfPackageConfig
metadata:
name: image-index

components:
- name: baseline
required: true
images:
- ghcr.io/defenseunicorns/zarf/agent:v0.32.6@sha256:05a82656df5466ce17c3e364c16792ae21ce68438bfe06eeab309d0520c16b48
9 changes: 9 additions & 0 deletions src/test/packages/14-index-sha/manifest-list/zarf.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
kind: ZarfPackageConfig
metadata:
name: manifest-list

components:
- name: baseline
required: true
images:
- defenseunicorns/zarf-game@sha256:0b694ca1c33afae97b7471488e07968599f1d2470c629f76af67145ca64428af
Loading