Improve make and development setup #41

MarcelMue · 2021-07-06T18:25:45Z

No description provided.

MarcelMue · 2021-07-06T18:57:40Z

.circleci/config.yml

      - architect/run-tests-with-abs:
          chart_dir: "./helm/policies-common"
          app-build-suite_version: "v0.2.3"
          app-build-suite_container_tag: "0.2.3"
          additional_app-build-suite_flags: "--external-cluster-version $KUBERNETES_VERSION"
+      - run:
+          name: Export kind logs


Export the logs so we have all the pod logs in circle.

MarcelMue · 2021-07-06T18:58:07Z

.circleci/config.yml

          command: |
-            ./kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/release-0.3/config/crd/bases/cluster.x-k8s.io_clusters.yaml > /dev/null
-            ./kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/release-0.3/config/crd/bases/cluster.x-k8s.io_machinedeployments.yaml > /dev/null
+            make setup


Use make for local dev and ci now.

MarcelMue · 2021-07-06T18:58:37Z

Makefile.custom.mk

@@ -1,13 +1,17 @@
 .PHONY: generate
 generate:
-	./template.sh
+	./hack/template.sh


Move scripts to the hack folder to clean up a little bit.

MarcelMue · 2021-07-06T18:59:14Z

hack/setup-local.sh

 kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/giantswarm/apiextensions/b9a6bed05850045530eaf8ba7fd01941de6c8525/helm/crds-common/templates/giantswarm.yaml
 kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/release-0.3/config/crd/bases/cluster.x-k8s.io_clusters.yaml
 kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/release-0.3/config/crd/bases/cluster.x-k8s.io_machinedeployments.yaml
 kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-aws/main/config/crd/bases/infrastructure.cluster.x-k8s.io_awsclusters.yaml
 kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-aws/main/config/crd/bases/infrastructure.cluster.x-k8s.io_awsclusterroleidentities.yaml
+kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/kyverno/kyverno/main/definitions/release/install.yaml


This needed to be reordered following upstream issue: kyverno/kyverno#2094

Does abs have logic to wait for kyverno to be up? If not, you could wait here. Something like https://github.com/giantswarm/coredns-warnlist-plugin/blob/f1d8ff108254e4f07968b718e933593de97ecd5d/.circleci/config.yml#L98

app-build-suite (soon to be replaced with app-test-suite for testing) usually waits until the app cr shows status deployed

https://github.com/giantswarm/app-test-suite/blob/master/app_test_suite/steps/base_test_runner.py#L315

I thought about waiting here but currently it will just make the test run longer - kyverno is up way before the tests start.

The problem is that kyverno is not the app-cr and therefor we wait for the setup of the kyverno-policies.

All in all not a big deal currently.

The problem with the bug I discovered was, that kyverno does currently not reload CRDs after it is started. This caused kyverno to fail if it was ready before all the CRDs were applied.

MarcelMue · 2021-07-06T18:59:33Z

helm/policies-aws/templates/AWSCAPI.yaml

@@ -34,9 +34,6 @@ spec:
      - key: "{{ `{{` }} releaseVersion {{ `}}` }}"
        operator: Equals
        value: "20.0.0"
-      - key: "{{ `{{` }} request.object.metadata.labels.\"release.giantswarm.io/version\" {{ `}}` }}"


Unrelated change which I messed up on main.

stone-z

One comment, but looking cleaner already

stone-z · 2021-07-06T19:34:13Z

.circleci/config.yml

          command: |
-            ./kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/release-0.3/config/crd/bases/cluster.x-k8s.io_clusters.yaml > /dev/null
-            ./kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/release-0.3/config/crd/bases/cluster.x-k8s.io_machinedeployments.yaml > /dev/null
+            make setup


stone-z · 2021-07-06T19:40:34Z

hack/setup-local.sh

 kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/giantswarm/apiextensions/b9a6bed05850045530eaf8ba7fd01941de6c8525/helm/crds-common/templates/giantswarm.yaml
 kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/release-0.3/config/crd/bases/cluster.x-k8s.io_clusters.yaml
 kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/release-0.3/config/crd/bases/cluster.x-k8s.io_machinedeployments.yaml
 kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-aws/main/config/crd/bases/infrastructure.cluster.x-k8s.io_awsclusters.yaml
 kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-aws/main/config/crd/bases/infrastructure.cluster.x-k8s.io_awsclusterroleidentities.yaml
+kubectl create --context kind-kyverno-cluster -f https://raw.githubusercontent.com/kyverno/kyverno/main/definitions/release/install.yaml


Does abs have logic to wait for kyverno to be up? If not, you could wait here. Something like https://github.com/giantswarm/coredns-warnlist-plugin/blob/f1d8ff108254e4f07968b718e933593de97ecd5d/.circleci/config.yml#L98

ubergesundheit · 2021-07-07T06:00:22Z

.circleci/config.yml

-      KIND_VERSION: v0.10.0
-      KUBERNETES_VERSION: v1.19.7
+      KIND_VERSION: v0.11.1
+      KUBERNETES_VERSION: v1.21.1


You'll want to update the kubernetes version also in the .abs/main.yaml

I am still looking for a way to have a single source of truth for this - currently thinking the make file / env and then replacing in main.yaml.

Nothing seems beautiful though 😅

ubergesundheit · 2021-07-07T06:01:47Z

hack/setup-local.sh

@@ -1,8 +1,9 @@
 kind create cluster --name kyverno-cluster


This is missing the --image flag from ci

This should also be fixed now.

MarcelMue added 4 commits July 6, 2021 19:15

Improve project structure and make file

aa148f2

Do some sudo

c943f5f

Update versions

126033c

Get logs

6b77982

MarcelMue self-assigned this Jul 6, 2021

MarcelMue added 3 commits July 6, 2021 20:32

Export and store kind logs

099d474

Always export logs

8928f79

Fix order of setup

91883aa

MarcelMue commented Jul 6, 2021

View reviewed changes

MarcelMue requested a review from a team July 6, 2021 18:59

ericgraf approved these changes Jul 6, 2021

View reviewed changes

stone-z approved these changes Jul 6, 2021

View reviewed changes

ubergesundheit reviewed Jul 7, 2021

View reviewed changes

Use correct kubernetes version consistently

0b1bcaa

MarcelMue merged commit 5a168f6 into main Jul 7, 2021

MarcelMue deleted the improve-make branch July 7, 2021 08:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve make and development setup #41

Improve make and development setup #41

MarcelMue commented Jul 6, 2021

MarcelMue Jul 6, 2021

MarcelMue Jul 6, 2021

stone-z Jul 6, 2021

MarcelMue Jul 6, 2021

MarcelMue Jul 6, 2021

stone-z Jul 6, 2021

ubergesundheit Jul 7, 2021

MarcelMue Jul 7, 2021

MarcelMue Jul 7, 2021

MarcelMue Jul 6, 2021

stone-z left a comment

stone-z Jul 6, 2021

stone-z Jul 6, 2021

ubergesundheit Jul 7, 2021

MarcelMue Jul 7, 2021

MarcelMue Jul 7, 2021

ubergesundheit Jul 7, 2021

MarcelMue Jul 7, 2021

Improve make and development setup #41

Improve make and development setup #41

Conversation

MarcelMue commented Jul 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stone-z left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment