Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement PackageVariantSet v1alpha2 #3930

Merged
merged 26 commits into from May 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
ce198a7
Add the PackageVariantSet v1alpha2 API types
johnbelamaric Apr 25, 2023
2d3471f
Generate the updated CRD
johnbelamaric Apr 25, 2023
ab46a70
Start work on the conversion methods
johnbelamaric Apr 25, 2023
d60c41d
Fix gofmt issue
johnbelamaric Apr 25, 2023
0f51df9
Refactor and switch 'hub' type to v1alpha2
johnbelamaric Apr 26, 2023
c7d3bb4
gofmt fix
johnbelamaric Apr 26, 2023
56a5a58
Implement more conversion and tests
johnbelamaric Apr 27, 2023
cc21ced
Updates for v2 api
johnbelamaric Apr 28, 2023
b62d089
Fix repositorySelector, ensurePV test
johnbelamaric Apr 28, 2023
72fb285
Fix formatting
johnbelamaric Apr 28, 2023
491c0f1
Fix status
johnbelamaric Apr 28, 2023
ad2756e
Add TestRenderPackageVariantSpec
johnbelamaric Apr 28, 2023
f506b98
Clean up the API a bit
johnbelamaric Apr 29, 2023
b80b074
Update validations for change API
johnbelamaric Apr 29, 2023
455a02e
Implement all non-CEL template fields
johnbelamaric Apr 29, 2023
95e87e4
Split into a few smaller files
johnbelamaric Apr 29, 2023
540ea3c
Move render test to a separate file
johnbelamaric May 1, 2023
9044bbe
Move validate tests to their own file
johnbelamaric May 1, 2023
0ca5162
Update the design doc, add some API comments
johnbelamaric May 1, 2023
00c685e
Initial CEL implementation
johnbelamaric May 2, 2023
bdf5aa1
More tests and some fixes
johnbelamaric May 2, 2023
1e2c55e
go mod tidy
johnbelamaric May 2, 2023
292ffa3
Stop serving v1alpha1 altogether
johnbelamaric May 2, 2023
b6ea267
Address review comments
johnbelamaric May 4, 2023
2aeb6b6
Post rebase go mod tidy
johnbelamaric May 4, 2023
9dfc65b
Fix panic due to not returning on error
johnbelamaric May 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
123 changes: 86 additions & 37 deletions docs/design-docs/08-package-variant.md
Expand Up @@ -914,56 +914,99 @@ template API is shown below.

```go
type PackageVariantTemplate struct {
Downstream *pkgvarapi.Downstream `json:"downstream,omitempty"`
DownstreamExprs *DownstreamExprs `json:"downstreamExprs,omitempty"`
// Downstream allows overriding the default downstream package and repository name
// +optional
Downstream *DownstreamTemplate `json:"downstream,omitempty"`

// AdoptionPolicy allows overriding the PackageVariant adoption policy
// +optional
AdoptionPolicy *pkgvarapi.AdoptionPolicy `json:"adoptionPolicy,omitempty"`

// DeletionPolicy allows overriding the PackageVariant deletion policy
// +optional
DeletionPolicy *pkgvarapi.DeletionPolicy `json:"deletionPolicy,omitempty"`

Labels map[string]string `json:"labels,omitempty"`
LabelExprs []MapExpr `json:"labelExprs,omitemtpy"`
// Labels allows specifying the spec.Labels field of the generated PackageVariant
// +optional
Labels map[string]string `json:"labels,omitempty"`

// LabelsExprs allows specifying the spec.Labels field of the generated PackageVariant
// using CEL to dynamically create the keys and values. Entries in this field take precedent over
// those with the same keys that are present in Labels.
// +optional
LabelExprs []MapExpr `json:"labelExprs,omitemtpy"`

Annotations map[string]string `json:"annotations,omitempty"`
AnnotationExprs []MapExpr `json:"annotationExprs,omitempty"`
// Annotations allows specifying the spec.Annotations field of the generated PackageVariant
// +optional
Annotations map[string]string `json:"annotations,omitempty"`

PackageContext map[string]string `json:"packageContext,omitempty"`
PackageContextExprs *PackageContextExprs `json:"packageContextExprs,omitempty"`
// AnnotationsExprs allows specifying the spec.Annotations field of the generated PackageVariant
// using CEL to dynamically create the keys and values. Entries in this field take precedent over
// those with the same keys that are present in Annotations.
// +optional
AnnotationExprs []MapExpr `json:"annotationExprs,omitempty"`

// PackageContext allows specifying the spec.PackageContext field of the generated PackageVariant
// +optional
PackageContext *PackageContextTemplate `json:"packageContext,omitempty"`

// Pipeline allows specifying the spec.Pipeline field of the generated PackageVariant
// +optional
Pipeline *kptfilev1.Pipeline `json:"pipeline,omitempty"`

Injectors []pkgvarapi.InjectionSelector `json:"injectors,omitempty"`
InjectorExprs []InjectionSelectorExprs `json:"injectorExprs,omitempty"`
// Injectors allows specifying the spec.Injectors field of the generated PackageVariant
// +optional
Injectors *InfectionSelectorTemplate `json:"injectors,omitempty"`
}

type DownstreamExprs struct {
// DownstreamTemplate is used to calculate the downstream field of the resulting
// package variants. Only one of Repo and RepoExpr may be specified;
// similarly only one of Package and PackageExpr may be specified.
type DownstreamTemplate struct {
Repo *string `json:"repo,omitempty"`
Package *string `json:"package,omitempty"`
RepoExpr *string `json:"repoExpr,omitempty"`
PackageExpr *string `json:"packageExpr,omitempty"`
}

type PackageContextExprs struct {
DataExprs []MapExpr `json:"dataExprs,omitempty"`
RemoveKeyExprs []string `json:"removeKeyExprs,omitempty"`
// PackageContextTemplate is used to calculate the packageContext field of the
// resulting package variants. The plain fields and Exprs fields will be
// merged, with the Exprs fields taking precedence.
type PackageContextTemplate struct {
Data map[string]string `json:"data,omitempty"`
RemoveKeys []string `json:"removeKeys,omitempty"`
DataExprs []MapExpr `json:"dataExprs,omitempty"`
RemoveKeyExprs []string `json:"removeKeyExprs,omitempty"`
}

type InjectionSelectorExprs struct {
GroupExpr *string `json:"groupExpr,omitempty"`
VersionExpr *string `json:"versionExpr,omitempty"`
KindExpr *string `json:"kindExpr,omitempty"`
NameExpr string `json:"nameExpr"`
// InjectionSelectorTemplate is used to calculate the injectors field of the
// resulting package variants. Only one of the Name and NameExpr fields may be
// specified.
type InjectionSelectorTemplate struct {
Group *string `json:"group,omitempty"`
Version *string `json:"version,omitempty"`
Kind *string `json:"kind,omitempty"`
Name *string `json:"name,omitempty"`

NameExpr *string `json:"nameExpr,omitempty"`
}

// MapExpr is used for various fields to calculate map entries. Only one of
// Key and KeyExpr may be specified; similarly only on of Value and ValueExpr
// may be specified.
type MapExpr struct {
Key *string `json:"key,omitempty"`
Value *string `json:"value,omitempty"`
KeyExpr *string `json:"keyExpr,omitempty"`
ValueExpr *string `json:"valueExpr,omitempty"`
}
```

This is a pretty complicated structure. To make it more understandable, the
first thing to notice is that many fields have a plain version, and an `Expr`
version. Only one of these should be used for any given field. For example, you
can use either `downstream` or `downstreamExpr`, but not both. The plain
version is used when the value is static across all the PackageVariants; the
`Expr` version is used when the value needs to vary across PackageVariants.
version. The plain version is used when the value is static across all the
PackageVariants; the `Expr` version is used when the value needs to vary across
PackageVariants.

Let's consider a simple example. Suppose we have a package for provisioning
namespaces called "base-ns". We want to instantiate this several times in the
Expand Down Expand Up @@ -1108,7 +1151,7 @@ spec:
org: hr
template:
labelExprs:
keyExpr: "'org'"
key: org
valueExpr: "repository.labels['org']"
injectorExprs:
- nameExpr: "repository.labels['region'] + '-endpoints'"
Expand Down Expand Up @@ -1183,16 +1226,16 @@ them. That is, there are certain objects that are accessible within the CEL
expression. For CEL expressions used in the PackageVariantSet `template` field,
the following variables are available:

| CEL Variable | Variable Contents |
| -------------| ------------------------------------------------------------ |
| repo | The default repository name based on the targeting criteria. |
| package | The default package name based on the targeting criteria. |
| upstream | The upstream PackageRevision. |
| repository | The downstream Repository. |
| CEL Variable | Variable Contents |
| -------------- | ------------------------------------------------------------ |
| repoDefault | The default repository name based on the targeting criteria. |
| packageDefault | The default package name based on the targeting criteria. |
| upstream | The upstream PackageRevision. |
| repository | The downstream Repository. |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add target here too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's described in the paragraph below, because it's contented varies based on the type of target. But I will add it here in a follow up PR.


There is one expression that is an exception to the table above. Since the
`repository` value corresponds to the Repository of the downstream, we must
first evaluate the `downstreamExpr.repoExpr` expression to *find* that
first evaluate the `downstream.repoExpr` expression to *find* that
repository. Thus, for that expression only, `repository` is not a valid
variable.

Expand All @@ -1202,26 +1245,32 @@ target, as follows:

| Target Type | `target` Variable Contents |
| ------------------- | -------------------------- |
| Repo/Package List | A struct with two fields: `name` and `packageName`, the same as the `repo` and `package` values. |
| Repository Selector | The Repository selected by the selector. Although not recommended, this could be different than the `repository` value, which can be altered with `downstream.repo` or `downstreamExprs.repoExpr`. |
| Repo/Package List | A struct with two fields: `name` and `packageName`, the same as the `repoDefault` and `packageDefault` values. |
| Repository Selector | The Repository selected by the selector. Although not recommended, this could be different than the `repository` value, which can be altered with `downstream.repo` or `downstream.repoExpr`. |
| Object Selector | The Object selected by the selector. |

For the various resource variables - `upstream`, `repository`, and `target` -
arbitrary access to all fields of the object could lead to security concerns.
Therefore, only a subset of the data is available for use in CEL exressions.
Specifically, the following fields: `name`, `namespace`, `labels`, and
`annotations`.

Given the slight quirk with the `repoExpr`, it may be helpful to state the
processing flow for the template evaluation:

1. The upstream PackageRevision is loaded. It must be in the same namespace as
the PackageVariantSet[^multi-ns-reg].
1. The targets are determined.
1. For each target:
1. The CEL environment is prepared with `repo`, `package`, `upstream`, and
`target` variables.
1. The CEL environment is prepared with `repoDefault`, `packageDefault`,
`upstream`, and `target` variables.
1. The downstream repository is determined and loaded, as follows:
- If present, `downstreamExprs.repoExpr` is evaluated using the CEL
- If present, `downstream.repoExpr` is evaluated using the CEL
environment, and the result used as the downstream repository name.
- Otherwise, if `downstream.repo` is set, that is used as the downstream
repository name.
- If neither is present, the default repository name based on the target is
used (i.e., the same value as the `repo` variable).
used (i.e., the same value as the `repoDefault` variable).
- The resulting downstream repository name is used to load the corresponding
Repository object in the same namespace as the PackageVariantSet.
1. The downstream Repository is added to the CEL environment.
Expand Down