Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

625 user control over pod labels #1069

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ SPDX-License-Identifier: Apache-2.0
- [Defining Retention Parameters](#defining-retention-parameters)
- [Defining Volumes](#defining-volumes)
- [BuildRun deletion](#BuildRun-deletion)
- [Labels](#labels)

## Overview

Expand Down Expand Up @@ -640,3 +641,18 @@ metadata:
annotations:
build.shipwright.io/build-run-deletion: "true"
```

## Labels

Labels can be defined for a Build as for any other Kubernetes object. Labels are propagated to the TaskRun and from there, Tekton propagates them to the Pod. A common use case for this is being able to filter resources, e.g. when fetching logs.

Since you can define labels for all of BuildStrategy/ClusterBuildStrategy, Build and BuildRun resources, if you define the same label key for different resources in the same build "pipeline", only the value in the last definition stage will be used (e.g. if you define `someproj.io/label: value01` in a Build and `someproj.io/label: value02` in a BuildRun that references such Build, `value02` will be used).

The following labels are not propagated:

- `*kubernetes.io/*`
- `*k8s.io/*`
- `*shipwright.io/*`
- `*tekton.dev/*`

A Kubernetes administrator can further restrict the usage of labels by using policy engines like [Open Policy Agent](https://www.openpolicyagent.org/).
16 changes: 16 additions & 0 deletions docs/buildrun.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ SPDX-License-Identifier: Apache-2.0
- [Step Results in BuildRun Status](#step-results-in-buildrun-status)
- [Build Snapshot](#build-snapshot)
- [Relationship with Tekton Tasks](#relationship-with-tekton-tasks)
- [Labels](#labels)

## Overview

Expand Down Expand Up @@ -470,3 +471,18 @@ For every BuildRun controller reconciliation, the `buildSpec` in the status of t
The `BuildRun` resource abstracts the image construction by delegating this work to the Tekton Pipeline [TaskRun](https://github.com/tektoncd/pipeline/blob/main/docs/taskruns.md). Compared to a Tekton Pipeline [Task](https://github.com/tektoncd/pipeline/blob/main/docs/tasks.md), a `TaskRun` runs all `steps` until completion of the `Task` or until a failure occurs in the `Task`.

During the Reconcile, the `BuildRun` controller will generate a new `TaskRun`. The controller will embed in the `TaskRun` `Task` definition the requires `steps` to execute during the execution. These `steps` are defined in the strategy defined in the `Build` resource, either a `ClusterBuildStrategy` or a `BuildStrategy`.

## Labels

Labels can be defined for a BuildRun as for any other Kubernetes object. Labels are propagated to the TaskRun and from there, Tekton propagates them to the Pod. A common use case for this is being able to filter resources, e.g. when fetching logs.

Since you can define labels for all of BuildStrategy/ClusterBuildStrategy, Build and BuildRun resources, if you define the same label key for different resources in the same build "pipeline", only the value in the last definition stage will be used (e.g. if you define `someproj.io/label: value01` in a Build and `someproj.io/label: value02` in a BuildRun that references such Build, `value02` will be used).

The following labels are not propagated:

- `*kubernetes.io/*`
- `*k8s.io/*`
- `*shipwright.io/*`
- `*tekton.dev/*`

A Kubernetes administrator can further restrict the usage of labels by using policy engines like [Open Policy Agent](https://www.openpolicyagent.org/).
24 changes: 18 additions & 6 deletions docs/buildstrategies.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ SPDX-License-Identifier: Apache-2.0
- [Strategies with different resources](#strategies-with-different-resources)
- [How does Tekton Pipelines handle resources](#how-does-tekton-pipelines-handle-resources)
- [Examples of Tekton resources management](#examples-of-tekton-resources-management)
- [Annotations](#annotations)
- [Labels and Annotations](#labels-and-annotations)
- [Volumes and VolumeMounts](#volumes-and-volumemounts)

## Overview
Expand Down Expand Up @@ -862,12 +862,24 @@ In the above scenario, we can see how the maximum numbers for resource requests

When a `LimitRange` exists on the namespace, `Tekton Pipeline` controller will do the same approach as stated in the above two scenarios. The difference is that for the containers that have lower values, instead of zero, they will get the `minimum values of the LimitRange`.

## Annotations
## Labels and Annotations

Annotations can be defined for a BuildStrategy/ClusterBuildStrategy as for any other Kubernetes object. Annotations are propagated to the TaskRun and from there, Tekton propagates them to the Pod. Use cases for this are for example:
Labels and annotations can be defined for a BuildStrategy/ClusterBuildStrategy as for any other Kubernetes object. Labels and annotations are propagated to the TaskRun and from there, Tekton propagates them to the Pod. Use cases for this are for example:

- The Kubernetes [Network Traffic Shaping](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#support-traffic-shaping) feature looks for the `kubernetes.io/ingress-bandwidth` and `kubernetes.io/egress-bandwidth` annotations to limit the network bandwidth the `Pod` is allowed to use.
- The [AppArmor profile of a container](https://kubernetes.io/docs/tutorials/clusters/apparmor/) is defined using the `container.apparmor.security.beta.kubernetes.io/<container_name>` annotation.
- Labels
- Being able to filter resources, e.g. when fetching logs.
- Annotations
- The Kubernetes [Network Traffic Shaping](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#support-traffic-shaping) feature looks for the `kubernetes.io/ingress-bandwidth` and `kubernetes.io/egress-bandwidth` annotations to limit the network bandwidth the `Pod` is allowed to use.
- The [AppArmor profile of a container](https://kubernetes.io/docs/tutorials/clusters/apparmor/) is defined using the `container.apparmor.security.beta.kubernetes.io/<container_name>` annotation.

Since you can define labels for all of BuildStrategy/ClusterBuildStrategy, Build and BuildRun resources, if you define the same label key for different resources in the same build "pipeline", only the value in the last definition stage will be used (e.g. if you define `someproj.io/label: value01` in a Build and `someproj.io/label: value02` in a BuildRun that references such Build, `value02` will be used).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please mention here that annotations cannot be overriden.


The following labels are not propagated:

- `*kubernetes.io/*`
- `*k8s.io/*`
- `*shipwright.io/*`
- `*tekton.dev/*`

The following annotations are not propagated:

Expand All @@ -877,7 +889,7 @@ The following annotations are not propagated:
- `build.shipwright.io/*`
- `buildrun.shipwright.io/*`

A Kubernetes administrator can further restrict the usage of annotations by using policy engines like [Open Policy Agent](https://www.openpolicyagent.org/).
A Kubernetes administrator can further restrict the usage of labels and annotations by using policy engines like [Open Policy Agent](https://www.openpolicyagent.org/).

## Volumes and VolumeMounts

Expand Down
2 changes: 1 addition & 1 deletion hack/install-kind.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ echo "# Creating a new Kubernetes cluster..."
kind create cluster --name="${KIND_CLUSTER_NAME}" --image="kindest/node:${KIND_CLUSTER_VERSION}" --wait=120s --config="${DIR}/../test/kind/config.yaml"

echo "# Using KinD context..."
kubectl config use-context "kind-kind"
kubectl config use-context "kind-${KIND_CLUSTER_NAME}"

echo "# KinD nodes:"
kubectl get nodes
5 changes: 5 additions & 0 deletions pkg/apis/build/v1alpha1/build_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -273,3 +273,8 @@ type BuildRetention struct {
func init() {
SchemeBuilder.Register(&Build{}, &BuildList{})
}

// GetLabels returns the labels of the Build
func (b Build) GetLabels() map[string]string {
return b.Labels
}
Comment on lines +278 to +280
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is needed because Kubernetes already provides that function on ObjectMeta.

5 changes: 5 additions & 0 deletions pkg/apis/build/v1alpha1/buildrun_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,11 @@ func (br *BuildRun) IsCanceled() bool {
return br.Spec.State != nil && *br.Spec.State == BuildRunStateCancel
}

// GetLabels returns the labels of the BuildRun
func (br *BuildRun) GetLabels() map[string]string {
return br.Labels
}
Comment on lines +275 to +278
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is needed because Kubernetes already provides that function on ObjectMeta.


// Conditions defines a list of Condition
type Conditions []Condition

Expand Down
4 changes: 2 additions & 2 deletions pkg/apis/build/v1alpha1/buildstrategy.go
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,7 @@ type BuildStep struct {
}

// BuildStrategyStatus defines the observed state of BuildStrategy
type BuildStrategyStatus struct {
}
type BuildStrategyStatus struct{}

// BuildStrategyKind defines the type of BuildStrategy used by the build.
type BuildStrategyKind string
Expand Down Expand Up @@ -112,4 +111,5 @@ type BuilderStrategy interface {
GetBuildSteps() []BuildStep
GetParameters() []Parameter
GetVolumes() []BuildStrategyVolume
GetLabels() map[string]string
}
5 changes: 5 additions & 0 deletions pkg/apis/build/v1alpha1/buildstrategy_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,11 @@ func (s BuildStrategy) GetVolumes() []BuildStrategyVolume {
return s.Spec.Volumes
}

// GetLabels returns the labels the build strategy
func (s BuildStrategy) GetLabels() map[string]string {
return s.Labels
}
Comment on lines +87 to +90
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is needed because Kubernetes already provides that function on ObjectMeta.


func init() {
SchemeBuilder.Register(&BuildStrategy{}, &BuildStrategyList{})
}
5 changes: 5 additions & 0 deletions pkg/apis/build/v1alpha1/clusterbuildstrategy_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,11 @@ func (s ClusterBuildStrategy) GetVolumes() []BuildStrategyVolume {
return s.Spec.Volumes
}

// GetLabels returns the labels of the build strategy
func (s ClusterBuildStrategy) GetLabels() map[string]string {
return s.Labels
}
Comment on lines +88 to +91
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is needed because Kubernetes already provides that function on ObjectMeta.


func init() {
SchemeBuilder.Register(&ClusterBuildStrategy{}, &ClusterBuildStrategyList{})
}
33 changes: 29 additions & 4 deletions pkg/reconciler/buildrun/resources/taskrun.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ package resources
import (
"fmt"
"path"
"regexp"
"strconv"
"strings"

Expand Down Expand Up @@ -35,12 +36,13 @@ const (
inputParamContextDir = "CONTEXT_DIR"

imageMutateContainerName = "mutate-image"

reservedLabels = `([a-z0-9]+\.)*(kubernetes.io|k8s.io|tekton.dev|shipwright.io)`
)

// getStringTransformations gets us MANDATORY replacements using
// a poor man's templating mechanism - TODO: Use golang templating
func getStringTransformations(fullText string) string {

stringTransformations := map[string]string{
// this will be removed, build strategy author should use $(params.shp-output-image) directly
"$(build.output.image)": fmt.Sprintf("$(params.%s-%s)", prefixParamsResultsVolumes, paramOutputImage),
Expand Down Expand Up @@ -69,7 +71,6 @@ func GenerateTaskSpec(
parameterDefinitions []buildv1alpha1.Parameter,
buildStrategyVolumes []buildv1alpha1.BuildStrategyVolume,
) (*v1beta1.TaskSpec, error) {

generatedTaskSpec := v1beta1.TaskSpec{
Params: []v1beta1.ParamSpec{
{
Expand Down Expand Up @@ -255,7 +256,6 @@ func GenerateTaskRun(
serviceAccountName string,
strategy buildv1alpha1.BuilderStrategy,
) (*v1beta1.TaskRun, error) {

// retrieve expected imageURL form build or buildRun
var image string
if buildRun.Spec.Output != nil {
Expand Down Expand Up @@ -322,6 +322,32 @@ func GenerateTaskRun(
expectedTaskRun.Labels[label] = value
}

reserved, err := regexp.Compile(reservedLabels)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regular expression should be compiled only once.

if err != nil {
return nil, err
}

// assign labels from the build strategy, filter out those that should not be propagated
for key, value := range strategy.GetLabels() {
if !reserved.MatchString(value) {
expectedTaskRun.Labels[key] = value
}
}

// assign labels from the build, filter out those that should not be propagated
for key, value := range build.GetLabels() {
if !reserved.MatchString(value) {
expectedTaskRun.Labels[key] = value
}
}

// assign labels from the buildrun, filter out those that should not be propagated
for key, value := range buildRun.GetLabels() {
if !reserved.MatchString(value) {
expectedTaskRun.Labels[key] = value
}
}

expectedTaskRun.Spec.Timeout = effectiveTimeout(build, buildRun)

params := []v1beta1.Param{
Expand Down Expand Up @@ -409,7 +435,6 @@ func GenerateTaskRun(
func effectiveTimeout(build *buildv1alpha1.Build, buildRun *buildv1alpha1.BuildRun) *metav1.Duration {
if buildRun.Spec.Timeout != nil {
return buildRun.Spec.Timeout

} else if build.Spec.Timeout != nil {
return build.Spec.Timeout
}
Expand Down
101 changes: 101 additions & 0 deletions pkg/reconciler/buildrun/resources/taskrun_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -607,5 +607,106 @@ var _ = Describe("GenerateTaskrun", func() {
Expect(paramOutputImageFound).To(BeTrue())
})
})

Context("when the taskrun is generated and strategy has custom labels", func() {
BeforeEach(func() {
build, err = ctl.LoadBuildYAML([]byte(test.BuildahBuildWithOutput))
Expect(err).To(BeNil())

buildRun, err = ctl.LoadBuildRunFromBytes([]byte(test.BuildahBuildRunWithSAAndOutput))
Expect(err).To(BeNil())

buildStrategy, err = ctl.LoadBuildStrategyFromBytes([]byte(test.BuildStrategyWithCustomLabels))
Expect(err).To(BeNil())
})

JustBeforeEach(func() {
got, err = resources.GenerateTaskRun(config.NewDefaultConfig(), build, buildRun, serviceAccountName, buildStrategy)
Expect(err).To(BeNil())
})

It("should propagate custom labels to the TaskRun", func() {
Expect(got.Labels["domain/bar"]).To(Equal("baz"))
})

It("shouldn't override custom labels set by the strategy", func() {
Expect(got.Labels["domain/foo"]).To(Equal("strategy"))
})
})

Context("when the taskrun is generated and build has custom labels", func() {
BeforeEach(func() {
build, err = ctl.LoadBuildYAML([]byte(test.MinimalBuildahBuildWithCustomLabels))
Expect(err).To(BeNil())

buildRun, err = ctl.LoadBuildRunFromBytes([]byte(test.BuildahBuildRunWithSAAndOutput))
Expect(err).To(BeNil())

buildStrategy, err = ctl.LoadBuildStrategyFromBytes([]byte(test.BuildStrategyWithCustomLabels))
Expect(err).To(BeNil())
})

JustBeforeEach(func() {
got, err = resources.GenerateTaskRun(config.NewDefaultConfig(), build, buildRun, serviceAccountName, buildStrategy)
Expect(err).To(BeNil())
})

It("should propagate custom labels to the TaskRun", func() {
Expect(got.Labels["domain/bar"]).To(Equal("baz"))
})

It("should override custom labels defined by strategy", func() {
Expect(got.Labels["domain/foo"]).ToNot(Equal("strategy"))
Expect(got.Labels["domain/foo"]).To(Equal("build"))
})
})

Context("when the taskrun is generated and buildrun has custom labels", func() {
BeforeEach(func() {
build, err = ctl.LoadBuildYAML([]byte(test.MinimalBuildahBuildWithCustomLabels))
Expect(err).To(BeNil())

buildRun, err = ctl.LoadBuildRunFromBytes([]byte(test.BuildahBuildRunWithCustomLabels))
Expect(err).To(BeNil())

buildStrategy, err = ctl.LoadBuildStrategyFromBytes([]byte(test.BuildStrategyWithCustomLabels))
Expect(err).To(BeNil())
})

JustBeforeEach(func() {
got, err = resources.GenerateTaskRun(config.NewDefaultConfig(), build, buildRun, serviceAccountName, buildStrategy)
Expect(err).To(BeNil())
})

It("should propagate custom labels to the TaskRun", func() {
Expect(got.Labels["domain/foo"]).To(Equal("buildrun"))
Expect(got.Labels["domain/bar"]).To(Equal("baz"))
})

It("should override custom labels defined by strategy or build", func() {
Expect(got.Labels["domain/foo"]).ToNot(Equal("strategy"))
Expect(got.Labels["domain/foo"]).ToNot(Equal("build"))
Expect(got.Labels["domain/foo"]).To(Equal("buildrun"))
})

It("should avoid reserved label to be set by the strategy", func() {
_, isSet := got.Labels["kubernetes.io"]
Expect(isSet).To(BeFalse())
_, isSet = got.Labels["foo.kubernetes.io"]
Expect(isSet).To(BeFalse())
_, isSet = got.Labels["k8s.io"]
Expect(isSet).To(BeFalse())
_, isSet = got.Labels["foo.k8s.io"]
Expect(isSet).To(BeFalse())
_, isSet = got.Labels["tekton.dev"]
Expect(isSet).To(BeFalse())
_, isSet = got.Labels["foo.tekton.dev"]
Expect(isSet).To(BeFalse())
_, isSet = got.Labels["shipwright.io"]
Expect(isSet).To(BeFalse())
_, isSet = got.Labels["foo.shipwright.io"]
Expect(isSet).To(BeFalse())
})
})
})
})
25 changes: 25 additions & 0 deletions test/build_samples.go
Original file line number Diff line number Diff line change
Expand Up @@ -586,3 +586,28 @@ spec:
ttlAfterFailed: 1m
ttlAfterSucceeded: 1m
`

const MinimalBuildahBuildWithCustomLabels = `
apiVersion: shipwright.io/v1alpha1
kind: Build
metadata:
name: buildah
labels:
domain/foo: build
domain/bar: baz
kubernetes.io/foo: bar
foo.kubernetes.io/bar: baz
k8s.io/foo: bar
foo.k8s.io/bar: baz
tekton.dev/foo: bar
foo.tekton.dev/bar: baz
shipwright.io/foo: bar
foo.shipwright.io/bar: baz
spec:
source:
url: "https://github.com/shipwright-io/sample-go"
strategy:
name: buildah
kind: ClusterBuildStrategy
dockerfile: Dockerfile
`
Loading