Skip to content

Ensure the platform operator stack is deployed before running e2e tests #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 26, 2022
Merged

Ensure the platform operator stack is deployed before running e2e tests #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cd9ca47
Refactor Dockerfile to copy locally built executables
timflannagan May 26, 2022
9310fc7
ci: Install platform operators before running e2e tests
timflannagan May 26, 2022
eb6135b
ci: Explicitly set the kind cluster cluster_name field
timflannagan May 26, 2022
6e9be74
Avoid failing at manager startup when the hardcoded CatalogSource isn…
timflannagan May 26, 2022
19acd4d
Dockerfile: Use the debug-nonroot tag to fix securityContext runtime …
timflannagan May 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,11 @@ jobs:

- name: Create k8s Kind Cluster
uses: helm/kind-action@v1.2.0
with:
cluster_name: kind

- name: Install platform operators
run: make deploy

- name: Run e2e tests
run: make e2e
Expand Down
28 changes: 3 additions & 25 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,5 @@
# Build the manager binary
FROM golang:1.17 as builder

WORKDIR /workspace
# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
RUN go mod download

# Copy the go source
COPY main.go main.go
COPY api/ api/
COPY controllers/ controllers/

# Build
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go

# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
FROM gcr.io/distroless/static:nonroot
FROM gcr.io/distroless/static:debug-nonroot
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update from the debug tag that matched the rukpak container image to debug-nonroot as I ran into the following error locally:

container has runasnonroot and image will run as root

And that's because kubebuilder/operator-sdk automatically scaffold out that securityContext for our manager resources by default (and we didn't use that SDK tooling in rukpak).

WORKDIR /
COPY --from=builder /workspace/manager .
USER 65532:65532

ENTRYPOINT ["/manager"]
COPY manager manager
EXPOSE 8080
26 changes: 10 additions & 16 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
IMG ?= controller:latest
# ENVTEST_K8S_VERSION refers to the version of kubebuilder assets to be downloaded by envtest binary.
ENVTEST_K8S_VERSION = 1.22
BIN_DIR := bin
CONTAINER_RUNTIME ?= docker

# Get the currently used golang install path (in GOPATH/bin, unless GOBIN is set)
ifeq (,$(shell go env GOBIN))
Expand Down Expand Up @@ -69,20 +71,12 @@ e2e: generate ginkgo ## Run e2e tests
##@ Build

.PHONY: build
build: generate fmt vet ## Build manager binary.
go build -o bin/manager main.go
build: ## Build manager binary.
CGO_ENABLED=0 go build -o bin/manager main.go

.PHONY: run
run: generate fmt vet ## Run a controller from your host.
go run ./main.go

.PHONY: docker-build
docker-build: ## Build docker image with the manager.
docker build -t ${IMG} .

.PHONY: docker-push
docker-push: ## Push docker image with the manager.
docker push ${IMG}
.PHONY: build-container
build-container: build ## Builds provisioner container image locally
$(CONTAINER_RUNTIME) build -f Dockerfile -t $(IMG) $(BIN_DIR)

##@ Deployment

Expand All @@ -91,12 +85,12 @@ ifndef ignore-not-found
endif

.PHONY: demo
# NOTE: This will fail as the currently available version of RukPak (v0.3.0) does not have
# NOTE: This will fail as the currently available version of RukPak (v0.4.0) does not have
# the requisite code.
demo: deploy install-samples

.PHONY: kind-load
kind-load: docker-build
kind-load: build-container
kind load docker-image $(IMG)

.PHONY: install
Expand All @@ -108,7 +102,7 @@ uninstall: generate kustomize ## Uninstall CRDs from the K8s cluster specified i
$(KUSTOMIZE) build config/crd | kubectl delete --ignore-not-found=$(ignore-not-found) -f -

.PHONY: deploy
deploy: generate kustomize kind-load install rukpak ## Deploy controller to the K8s cluster specified in ~/.kube/config.
deploy: build-container kind-load install rukpak ## Deploy controller to the K8s cluster specified in ~/.kube/config.
cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
$(KUSTOMIZE) build config/default | kubectl apply -f -

Expand Down
37 changes: 28 additions & 9 deletions controllers/platformoperator_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,10 @@ import (
platformv1alpha1 "github.com/timflannagan/platform-operators/api/v1alpha1"
)

const channelName = "4.12"
const (
channelName = "4.12"
plainProvisionerID = "core.rukpak.io/plain"
)

// PlatformOperatorReconciler reconciles a PlatformOperator object
type PlatformOperatorReconciler struct {
Expand Down Expand Up @@ -102,8 +105,27 @@ func (r *PlatformOperatorReconciler) Reconcile(ctx context.Context, req ctrl.Req
return ctrl.Result{}, client.IgnoreNotFound(err)
}

css := &operatorsv1alpha1.CatalogSourceList{}
if err := r.List(ctx, css); err != nil {
log.Error(err, "failed to list the catalogsource resources in the cluster")
return ctrl.Result{}, err
}
if len(css.Items) == 0 {
log.Info("unable to query catalog content as no catalogsources are available")
return ctrl.Result{}, nil
}
// TODO(tflannag): properly handle multiple catalogsources in a cluster
cs := css.Items[0]
Comment on lines +117 to +118
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a super short term hack that's needed in order to work with the basic e2e tests that I have sitting locally, which deploy a magic catalog instance, so hardcoding the CatalogSource reference, and even worse hardcode that logic at the main.go level as it fails the container at startup, wouldn't work with that setup.


log.Info("creating registry client from catalogsource")
rc, err := registryClient.NewClient(cs.Spec.Address)
if err != nil {
log.Error(err, "failed to create registry client from catalogsource", "name", cs.GetName(), "namespace", cs.GetNamespace(), "address", cs.Spec.Address)
return ctrl.Result{}, err
}

log.Info("listing bundles from context")
it, err := r.RegistryClient.ListBundles(ctx)
it, err := rc.ListBundles(ctx)
if err != nil {
log.Error(err, "failed to list bundles in the platform operators catalog source")
return ctrl.Result{}, err
Expand Down Expand Up @@ -153,22 +175,19 @@ func (r *PlatformOperatorReconciler) ensureBundleInstance(ctx context.Context, p

_, err := controllerutil.CreateOrUpdate(ctx, r.Client, bi, func() error {
bi.SetOwnerReferences([]metav1.OwnerReference{*controllerRef})
bi.Spec = *buildBundleInstance(bi.GetName(), bundle.BundlePath)
bi.Spec = *buildBundleInstance(bundle.BundlePath)
return nil
})
return err
}

// createBundleInstance is responsible for taking a name and image to create an embedded BundleInstance
func buildBundleInstance(name, image string) *rukpakv1alpha1.BundleInstanceSpec {
func buildBundleInstance(image string) *rukpakv1alpha1.BundleInstanceSpec {
return &rukpakv1alpha1.BundleInstanceSpec{
ProvisionerClassName: "core.rukpak.io/plain",
ProvisionerClassName: plainProvisionerID,
Template: &rukpakv1alpha1.BundleTemplate{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{"app": name},
},
Spec: rukpakv1alpha1.BundleSpec{
ProvisionerClassName: "core.rukpak.io/plain",
ProvisionerClassName: plainProvisionerID,
Source: rukpakv1alpha1.BundleSource{
Type: rukpakv1alpha1.SourceTypeImage,
Image: &rukpakv1alpha1.ImageSource{
Expand Down
31 changes: 4 additions & 27 deletions main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,14 @@ limitations under the License.
package main

import (
"context"
"flag"
"os"
"time"

// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
// to ensure that exec-entrypoint and run can make use of them.
_ "k8s.io/client-go/plugin/pkg/client/auth"

operatorsv1alpha1 "github.com/operator-framework/api/pkg/operators/v1alpha1"
registryClient "github.com/operator-framework/operator-registry/pkg/client"
rukpakv1alpha1 "github.com/operator-framework/rukpak/api/v1alpha1"
"k8s.io/apimachinery/pkg/runtime"
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
Expand All @@ -41,16 +38,9 @@ import (
//+kubebuilder:scaffold:imports
)

const (
catalogSourceName = "platform-operators-catalog-source"
catalogSourceNamespace = "platform-operators-system"
catalogSourceReconnectTime = time.Second * 5
)

var (
scheme = runtime.NewScheme()
setupLog = ctrl.Log.WithName("setup")
catalogSourceService = catalogSourceName + "." + catalogSourceNamespace + ".svc:50051"
scheme = runtime.NewScheme()
setupLog = ctrl.Log.WithName("setup")
)

func init() {
Expand Down Expand Up @@ -91,22 +81,9 @@ func main() {
os.Exit(1)
}

// Create a registryClient by referencing the requisite catalogSource's service and then
// check that it is healthy
c, err := registryClient.NewClient(catalogSourceService)
if err != nil {
setupLog.Error(err, "failed to create registry client from "+catalogSourceName+" catalog source")
os.Exit(1)
}
if healthy, err := c.HealthCheck(context.Background(), catalogSourceReconnectTime); !healthy || err != nil {
setupLog.Error(err, "failed to connect to "+catalogSourceName+" catalog source via the registry client")
os.Exit(1)
}

if err = (&controllers.PlatformOperatorReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
RegistryClient: c,
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "PlatformOperator")
os.Exit(1)
Expand Down