test workflow fails after bootstrap - Failed initializing schema

[annabarnes1138]

I bootstrapped an existing k8s cluster with the staging config. After it committed the flux-system files the test workflow started failing. Here is the pertinent part of the log

INFO - Validating clusters
PASS - ./clusters/production/infrastructure.yaml contains a valid Kustomization (flux-system.infrastructure)
PASS - ./clusters/production/apps.yaml contains a valid Kustomization (flux-system.apps)
ERR  - ./clusters/staging/flux-system/gotk-components.yaml: Failed initializing schema https://kubernetesjsonschema.dev/master-standalone-strict/customresourcedefinition-apiextensions-v1.json: Could not read schema from HTTP, response status is 404 Not Found
Failed initializing schema file:///tmp/flux-crd-schemas/master-standalone-strict/customresourcedefinition-apiextensions-v1.json: open /tmp/flux-crd-schemas/master-standalone-strict/customresourcedefinition-apiextensions-v1.json: no such file or directory
Error: Process completed with exit code 1.

It seems to be failing at line 45:
kubeval ${file} --strict --additional-schema-locations=file:///tmp/flux-crd-schemas

Any suggestions?

[ogomezm]

I am having the same issue and tried to workaround it without success.

First thing i did was trying to ignore-missing-schemas in the kubeval command scripts/validate.sh
kubeval ${file} **--ignore-missing-schemas** --strict --additional-schema-locations=file:///tmp/flux-crd-schemas

After doing that I am able to progress yet having the next issue.

Run ./scripts/validate.sh
INFO - Downloading Flux OpenAPI schemas
INFO - Validating ./clusters/staging/flux-system/gotk-components.yaml
INFO - Validating ./clusters/staging/flux-system/kustomization.yaml
INFO - Validating ./clusters/staging/flux-system/gotk-sync.yaml
INFO - Validating ./clusters/staging/infrastructure.yaml
INFO - Validating ./clusters/staging/apps.yaml
INFO - Validating ./clusters/production/infrastructure.yaml
INFO - Validating ./clusters/production/apps.yaml
INFO - Validating ./.github/workflows/test.yaml
INFO - Validating ./.github/workflows/e2e.yaml
INFO - Validating ./infrastructure/kustomization.yaml
INFO - Validating ./infrastructure/sources/kustomization.yaml
INFO - Validating ./infrastructure/sources/podinfo.yaml
INFO - Validating ./apps/base/podinfo/release.yaml
INFO - Validating ./apps/base/podinfo/kustomization.yaml
INFO - Validating ./apps/base/podinfo/namespace.yaml
INFO - Validating ./apps/staging/kustomization.yaml
INFO - Validating ./apps/staging/podinfo-values.yaml
INFO - Validating ./apps/production/kustomization.yaml
INFO - Validating ./apps/production/podinfo-values.yaml
INFO - Validating clusters
WARN - Set to ignore missing schemas
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Namespace (flux-system)
WARN - ./clusters/staging/flux-system/gotk-components.yaml containing a CustomResourceDefinition (alerts.notification.toolkit.fluxcd.io) was not validated against a schema
WARN - ./clusters/staging/flux-system/gotk-components.yaml containing a CustomResourceDefinition (buckets.source.toolkit.fluxcd.io) was not validated against a schema
WARN - ./clusters/staging/flux-system/gotk-components.yaml containing a CustomResourceDefinition (gitrepositories.source.toolkit.fluxcd.io) was not validated against a schema
WARN - ./clusters/staging/flux-system/gotk-components.yaml containing a CustomResourceDefinition (helmcharts.source.toolkit.fluxcd.io) was not validated against a schema
WARN - ./clusters/staging/flux-system/gotk-components.yaml containing a CustomResourceDefinition (helmreleases.helm.toolkit.fluxcd.io) was not validated against a schema
WARN - ./clusters/staging/flux-system/gotk-components.yaml containing a CustomResourceDefinition (helmrepositories.source.toolkit.fluxcd.io) was not validated against a schema
WARN - ./clusters/staging/flux-system/gotk-components.yaml containing a CustomResourceDefinition (kustomizations.kustomize.toolkit.fluxcd.io) was not validated against a schema
WARN - ./clusters/staging/flux-system/gotk-components.yaml containing a CustomResourceDefinition (providers.notification.toolkit.fluxcd.io) was not validated against a schema
WARN - ./clusters/staging/flux-system/gotk-components.yaml containing a CustomResourceDefinition (receivers.notification.toolkit.fluxcd.io) was not validated against a schema
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ServiceAccount (flux-system.helm-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ServiceAccount (flux-system.kustomize-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ServiceAccount (flux-system.notification-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ServiceAccount (flux-system.source-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ClusterRole (crd-controller-flux-system)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ClusterRoleBinding (cluster-reconciler-flux-system)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ClusterRoleBinding (crd-controller-flux-system)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Service (flux-system.notification-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Service (flux-system.source-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Service (flux-system.webhook-receiver)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Deployment (flux-system.helm-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Deployment (flux-system.kustomize-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Deployment (flux-system.notification-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Deployment (flux-system.source-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid NetworkPolicy (flux-system.allow-scraping)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid NetworkPolicy (flux-system.allow-webhooks)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid NetworkPolicy (flux-system.deny-ingress)
WARN - Set to ignore missing schemas
ERR  - ./clusters/staging/flux-system/kustomization.yaml: Missing 'metadata' key
Error: Process completed with exit code 1.

As you can see it detects missing metadata in the kustomization.yaml...

I then added the a metadata field in the kustomization.yaml

metadata:
  name: flux

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: flux
resources:
- gotk-components.yaml
- gotk-sync.yaml

After doing this I've got and exit 1 and the same error from the begining ;).

PASS - ./clusters/production/apps.yaml contains a valid Kustomization (flux-system.apps)
INFO - Validating kustomize overlays
INFO - Validating kustomization ./clusters/staging/flux-system/
ERR  - stdin: Failed initializing schema https://kubernetesjsonschema.dev/master-standalone-strict/customresourcedefinition-apiextensions-v1.json: Could not read schema from HTTP, response status is 404 Not Found
Failed initializing schema file:///tmp/flux-crd-schemas/master-standalone-strict/customresourcedefinition-apiextensions-v1.json: open /tmp/flux-crd-schemas/master-standalone-strict/customresourcedefinition-apiextensions-v1.json: no such file or directory
Error: Process completed with exit code 1.

Then I tried downloading the missing schema

curl -sL -o /tmp/flux-crd-schemas/master-standalone-strict/customresourcedefinition-apiextensions-v1.json https://raw.githubusercontent.com/instrumenta/kubernetes-json-schema/master/master-local/customresourcedefinition-apiextensions-v1.json

I also had to add the following base dependency of the schema.

curl -sL -o /tmp/flux-crd-schemas/master-standalone-strict/_definitions.json https://raw.githubusercontent.com/instrumenta/kubernetes-json-schema/master/master-local/_definitions.json

Then I had an exit 1 with no more details.

 WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (helmreleases.helm.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (helmrepositories.source.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (kustomizations.kustomize.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (providers.notification.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (receivers.notification.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ServiceAccount (flux-system.helm-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ServiceAccount (flux-system.kustomize-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ServiceAccount (flux-system.notification-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ServiceAccount (flux-system.source-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ClusterRole (crd-controller-flux-system)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ClusterRoleBinding (cluster-reconciler-flux-system)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid ClusterRoleBinding (crd-controller-flux-system)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Service (flux-system.notification-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Service (flux-system.source-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Service (flux-system.webhook-receiver)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Deployment (flux-system.helm-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Deployment (flux-system.kustomize-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Deployment (flux-system.notification-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid Deployment (flux-system.source-controller)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid NetworkPolicy (flux-system.allow-scraping)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid NetworkPolicy (flux-system.allow-webhooks)
PASS - ./clusters/staging/flux-system/gotk-components.yaml contains a valid NetworkPolicy (flux-system.deny-ingress)
Error: Process completed with exit code 1.

After that i removed the --ignore-missing-schemas from the kubeval command in scripts/validate.sh as the dependencies should be already in place.

kubeval ${file} --strict --additional-schema-locations=file:///tmp/flux-crd-schemas

I got the same result. There weren't schema missing errors yet the exit code 1.
There were some warnings in the gotk-components.yaml file flux bootstrap pushes into clusters/staging/flux-system.

These are the warnings i got.

WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (alerts.notification.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (buckets.source.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (gitrepositories.source.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (helmcharts.source.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (helmreleases.helm.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (helmrepositories.source.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (kustomizations.kustomize.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (providers.notification.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null
WARN - ./clusters/staging/flux-system/gotk-components.yaml contains an invalid CustomResourceDefinition (receivers.notification.toolkit.fluxcd.io) - metadata.creationTimestamp: Invalid type. Expected: string, given: null