Support K8s 1.22 in profile controller (kubeflow#6491)

* profile-controller: Delete old kubebuilder v1 project

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: Initialize Kubebuilder v3 project

go module name:
github.com/kubeflow/kubeflow/components/profile-controller

project domain:
kubeflow.org

Changes:

* Go Version got upgraded from v1.15 to v1.17
* With Kubebuilder V3 our project dependencies got
updated to a more up-to-date version.
* controller-runtime library got updated from v0.2.0 to v0.12.1.

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: create placeholder API types and controller logic files

We created the Profile API types
v1: kubebuilder create api --version v1 kind Profile
v1beta1: kubebuilder create api --version v1beta1 Profile

We copied over the API definitions from the old project
to the new one

(Note: Currently KFAM uses the v1beta1 version of Profile
to provide fine-grain user-namespace level access control)

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: copy controllers logic from old project to the new one

* Copied the files under controller/ dir from the old to our new project

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: Code modifications to fix profile-controller

With Kubebuilder V3 the dependencies of our project got updated
to a more up-to-date version. Newer versions of certain packages
resulted to broken code parts that we had to fix. More specifically:

* In this project we are using a namespace-labels-data ConfigMap
to set certain labels on every Profile namespace. Every change in
this ConfigMap produces an event and for every event we construct
a list of reconcile.Requests that contains the name and namespace
of every Profile in our system.

Thus we introduced a mapEventToRequest MapFunc (profile_controller.go)
that maps an event to reconcile requests for all Profiles. This change
was necessary because controller-runtime library got updated from v0.6.3
to v0.12.1

* Kubebuilder V3 introduced for the profile-controller:
1) a livenessprobe (/healthz) on port :8081
2) a readinessProbe (/readyz) on port :8081

We have a conflict here as KFAM Service uses the :8081 port as well.
Therefore, we switched the port to :9876 for the above Probes.

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: Fix manifests, Makefile, Dockerfile

Makefile:
* Add image (build + push) target
* Add test-overlays target
* Build image using public.ecr.aws registry instead of gcr.io.

Dockerfile:
* fix Dockerfile to build our controller correclty

Manifests
* Add base/ and overlays/ manifests
* Fix the manifests to be in sync with upstream

other changes:
Update README.md file

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: Add test files

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: Fix Makefile

* Set the correct IMG:TAG
* Remove test dependency

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

components/profile-controller/.dockerignore

@@ -0,0 +1,4 @@
+# More info: https://docs.docker.com/engine/reference/builder/#dockerignore-file
+# Ignore build and test binaries.
+bin/
+testbin/

components/profile-controller/.gitignore

@@ -6,6 +6,7 @@
 *.so
 *.dylib
 bin
+testbin/*
 
 # Test binary, build with `go test -c`
 *.test
@@ -22,5 +23,3 @@ bin
 *.swp
 *.swo
 *~
-
-gcb_build

components/profile-controller/Dockerfile

@@ -1,6 +1,5 @@
 # Build the manager binary
-ARG GOLANG_VERSION=1.15
-FROM golang:${GOLANG_VERSION} as builder
+FROM golang:1.17 as builder
 
 WORKDIR /workspace
 # Copy the Go Modules manifests
@@ -36,3 +35,4 @@ COPY --from=builder /go/pkg/mod/github.com/hashicorp third_party/library/
 EXPOSE 8080
 
 ENTRYPOINT ["/manager"]
+

components/profile-controller/Makefile

@@ -1,8 +1,8 @@
 # Image URL to use all building/pushing image targets
 IMG ?= profile-controller
 TAG ?= $(shell git describe --tags --always --dirty)
-GOLANG_VERSION ?= 1.15
-CONTROLLER_GEN_VERSION ?= 0.4.0
+# ENVTEST_K8S_VERSION refers to the version of kubebuilder assets to be downloaded by envtest binary.
+ENVTEST_K8S_VERSION = 1.23
 
 # Get the currently used golang install path (in GOPATH/bin, unless GOBIN is set)
 ifeq (,$(shell go env GOBIN))
@@ -11,65 +11,136 @@ else
 GOBIN=$(shell go env GOBIN)
 endif
 
-all: manager
+# Setting SHELL to bash allows bash commands to be executed by recipes.
+# This is a requirement for 'setup-envtest.sh' in the test target.
+# Options are set to exit when a recipe line exits non-zero or a piped command fails.
+SHELL = /usr/bin/env bash -o pipefail
+.SHELLFLAGS = -ec
 
-# Run tests
-test: generate fmt vet
-	go test ./... -coverprofile cover.out
+.PHONY: all
+all: build
 
-# Build manager binary
-manager: generate fmt vet
-	go build -o bin/manager main.go
-
-# Run against the configured Kubernetes cluster in ~/.kube/config
-run: generate fmt vet
-	go run ./main.go
-
-# Install CRDs into a cluster
-install:
-	kustomize build config/crd | kubectl apply -f -
+.PHONY: help
+help: ## Display this help.
+	@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n  make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf "  \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)
 
-# Deploy controller in the configured Kubernetes cluster in ~/.kube/config
-deploy: manifests
-	cd config/manager && kustomize edit set image gcr.io/kubeflow-images-public/profile-controller=${IMG}
-	kustomize build config/default | kubectl apply -f -
+##@ Development
 
-# Generate manifests e.g. CRD, RBAC etc.
-manifests: controller-gen
-	$(CONTROLLER_GEN) crd:trivialVersions=true paths="./..." output:crd:artifacts:config=config/crd/bases
+.PHONY: manifests
+manifests: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
+	$(CONTROLLER_GEN) crd paths="./..." output:crd:artifacts:config=config/crd/bases
 	# Uncomment when we remove the permissive ClusterRoleBinding to cluster-admin
 	# $(CONTROLLER_GEN) rbac:roleName=cluster-role-binding webhook paths="./..."
 
-# Run go fmt against code
-fmt:
+.PHONY: generate
+generate: controller-gen ## Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
+	$(CONTROLLER_GEN) object:headerFile="hack/boilerplate.go.txt" paths="./..."
+
+
+.PHONY: fmt
+fmt: ## Run go fmt against code.
 	go fmt ./...
 
-# Run go vet against code
-vet:
+.PHONY: vet
+vet: ## Run go vet against code.
 	go vet ./...
 
-# Generate code
-generate: controller-gen
-	$(CONTROLLER_GEN) object:headerFile=./hack/boilerplate.go.txt paths="./..."
+.PHONY: test
+test: manifests generate fmt vet envtest ## Run tests.
+	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) -p path)" go test ./... -coverprofile cover.out
+
+##@ Build
+
+.PHONY: build
+build: generate fmt vet ## Build manager binary.
+	go build -o bin/manager main.go
+
+.PHONY: run
+run: manifests generate fmt vet ## Run a controller from your host.
+	go run ./main.go -namespace-labels-path ./config/base/namespace-labels.yaml 
 
-# Build the docker image
-docker-build:
-	docker build --build-arg GOLANG_VERSION=${GOLANG_VERSION} -t ${IMG}:${TAG} .
-	@echo Built ${IMG}:${TAG}
+.PHONY: docker-build
+docker-build: ## Build docker image with the manager.
+	docker build -t ${IMG}:${TAG} .
+	@echo Built ${IMG}:${TAG}.
 
-# Push the docker image
-docker-push: 
+.PHONY: docker-push
+docker-push: ## Push docker image with the manager.
 	docker push ${IMG}:${TAG}
-	@echo Pushed ${IMG}:${TAG}
 
-image: docker-build docker-push
+.PHONY: image
+image: docker-build docker-push ## Build and push docker image with the manager.
 
-# find or download controller-gen
-# download controller-gen if necessary
-controller-gen:
-ifeq (, $(shell which controller-gen))
-	go get sigs.k8s.io/controller-tools/cmd/controller-gen@v${CONTROLLER_GEN_VERSION}
-CONTROLLER_GEN=$(GOBIN)/controller-gen
-else
-CONTROLLER_GEN=$(shell which controller-gen)
+.PHONY: build-gcp
+build-gcb:
+	gcloud --project=$(PROJECT) \
+		builds submit \
+		--machine-type=n1-highcpu-32 \
+		--substitutions=_TAG=$(TAG),_REGISTRY=$(PROJECT) \
+		--config=cloudbuild.yaml .
+
+##@ Deployment
+
+ifndef ignore-not-found
+  ignore-not-found = false
 endif
+
+.PHONY: install
+install: manifests  ## Install CRDs into the K8s cluster specified in ~/.kube/config.
+	$(KUSTOMIZE) build config/crd | kubectl apply -f -
+
+.PHONY: uninstall
+uninstall: manifests ## Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
+	$(KUSTOMIZE) build config/crd | kubectl delete --ignore-not-found=$(ignore-not-found) -f -
+
+.PHONY: deploy
+deploy: manifests ## Deploy controller to the K8s cluster specified in ~/.kube/config.
+	@echo IMG=${IMG}
+	@echo TAG=${TAG}
+	sed -i'' -e 's@newName: .*@newName: '"${IMG}"'@' ./config/base/kustomization.yaml
+	sed -i'' -e 's@newTag: .*@newTag: '"${TAG}"'@' ./config/base/kustomization.yaml
+	$(KUSTOMIZE) build config/base | kubectl apply -f -
+
+.PHONY: test-overlays
+test-overlays: manifests ## Deploy controller to the K8s cluster specified in ~/.kube/config. with overlay changes on top
+	@echo IMG=${IMG}
+	@echo TAG=${TAG}
+	sed -i'' -e 's@newName: .*@newName: '"${IMG}"'@' ./config/base/kustomization.yaml
+	sed -i'' -e 's@newTag: .*@newTag: '"${TAG}"'@' ./config/base/kustomization.yaml
+	$(KUSTOMIZE) build config/overlays/kubeflow | kubectl apply -f -
+
+.PHONY: undeploy
+undeploy: ## Undeploy controller from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
+	$(KUSTOMIZE) build config/base | kubectl delete --ignore-not-found=$(ignore-not-found) -f -
+
+##@ Build Dependencies
+
+## Location to install dependencies to
+LOCALBIN ?= $(shell pwd)/bin
+$(LOCALBIN):
+	mkdir -p $(LOCALBIN)
+
+## Tool Binaries
+KUSTOMIZE ?= $(LOCALBIN)/kustomize
+CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
+ENVTEST ?= $(LOCALBIN)/setup-envtest
+
+## Tool Versions
+KUSTOMIZE_VERSION ?= v3.2.0
+CONTROLLER_TOOLS_VERSION ?= v0.8.0
+
+KUSTOMIZE_INSTALL_SCRIPT ?= "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"
+.PHONY: kustomize
+kustomize: $(KUSTOMIZE) ## Download kustomize locally if necessary.
+$(KUSTOMIZE): $(LOCALBIN)
+	curl -s $(KUSTOMIZE_INSTALL_SCRIPT) | bash -s -- $(subst v,,$(KUSTOMIZE_VERSION)) $(LOCALBIN)
+
+.PHONY: controller-gen
+controller-gen: $(CONTROLLER_GEN) ## Download controller-gen locally if necessary.
+$(CONTROLLER_GEN): $(LOCALBIN)
+	GOBIN=$(LOCALBIN) go install sigs.k8s.io/controller-tools/cmd/controller-gen@$(CONTROLLER_TOOLS_VERSION)
+
+.PHONY: envtest
+envtest: $(ENVTEST) ## Download envtest-setup locally if necessary.
+$(ENVTEST): $(LOCALBIN)
+	GOBIN=$(LOCALBIN) go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest

components/profile-controller/PROJECT

@@ -1,7 +1,22 @@
-version: "2"
 domain: kubeflow.org
-repo: github.com/kubeflow/kubeflow
+layout:
+- go.kubebuilder.io/v3
+projectName: profile-controller
+repo: github.com/kubeflow/kubeflow/components/profile-controller
 resources:
-- group: profile
+- api:
+    crdVersion: v1
+    namespaced: true
+  controller: true
+  domain: kubeflow.org
+  kind: Profile
+  path: github.com/kubeflow/kubeflow/components/profile-controller/api/v1
   version: v1
+- api:
+    crdVersion: v1
+    namespaced: true
+  domain: kubeflow.org
   kind: Profile
+  path: github.com/kubeflow/kubeflow/components/profile-controller/api/v1beta1
+  version: v1beta1
+version: "3"

components/profile-controller/README.md

@@ -92,3 +92,49 @@ Plugin owners have full control over plugin spec struct and implementation.
   - Type: credential binding
   - IAM For Service Account plugin will grant k8s service account permission of IAM role,
   so pods in profile namespace can authenticate AWS services as IAM role.
+
+# Deployment
+
+Install the `profiles.kubeflow.org` CRD:
+```sh
+make install
+```
+
+Deploy the profile controller manager:
+```sh
+make deploy
+```
+
+Verify that the controller is running in the `profiles-system` namespace:
+```sh
+kubectl get pods -l kustomize.component=profiles -n profiles-system
+```
+
+### Clean-up
+
+Uninstall the profile controller manager:
+
+```
+make undeploy
+```
+
+Uninstall the `profiles.kubeflow.org` CRD:
+
+```
+make uninstall
+```
+
+### Running the controller locally
+
+In order for the custom Notebook Controller to be functional from your local machine, the admins must:
+
+1. Set the number of replicas to zero:
+```sh
+kubectl edit deployment profiles-controller-deployment -n=kubeflow
+```
+
+2. Start the manager locally:
+```sh
+make run
+```
+

components/profile-controller/api/v1/groupversion_info.go

@@ -1,4 +1,5 @@
 /*
+Copyright 2022.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -13,9 +14,9 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-// Package v1 contains API Schema definitions for the profile v1 API group
-// +kubebuilder:object:generate=true
-// +groupName=kubeflow.org
+// Package v1 contains API Schema definitions for the  v1 API group
+//+kubebuilder:object:generate=true
+//+groupName=kubeflow.org
 package v1
 
 import (

components/profile-controller/api/v1/profile_types.go

@@ -1,11 +1,8 @@
 /*
-
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
-
     http://www.apache.org/licenses/LICENSE-2.0
-
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

components/profile-controller/api/v1beta1/groupversion_info.go

@@ -1,4 +1,5 @@
 /*
+Copyright 2022.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -13,9 +14,9 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-// Package v1beta1 contains API Schema definitions for the profile v1beta1 API group
-// +kubebuilder:object:generate=true
-// +groupName=kubeflow.org
+// Package v1beta1 contains API Schema definitions for the  v1beta1 API group
+//+kubebuilder:object:generate=true
+//+groupName=kubeflow.org
 package v1beta1
 
 import (

components/profile-controller/api/v1beta1/profile_types.go

@@ -1,11 +1,8 @@
 /*
-
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
-
     http://www.apache.org/licenses/LICENSE-2.0
-
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

components/profile-controller/config/base/kustomization.yaml

@@ -11,8 +11,8 @@ patchesStrategicMerge:
 
 images:
 - name: public.ecr.aws/j1r0q0g6/notebooks/profile-controller
-  newName: public.ecr.aws/j1r0q0g6/notebooks/profile-controller
-  newTag: v1.5.0
+  newName: gcr.io/arrikto-playground/profile-controller
+  newTag: v1.4-rc.0-200-gb79289d1
 
 configMapGenerator:
 - name: namespace-labels-data