Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

make apply error #425

Closed
Tracked by #423
chusband opened this issue May 2, 2023 · 9 comments
Closed
Tracked by #423

make apply error #425

chusband opened this issue May 2, 2023 · 9 comments
Assignees
@gkcalat

Comments

@chusband
Copy link

chusband commented May 2, 2023

Version: 1.7.0
Following step: https://googlecloudplatform.github.io/kubeflow-gke-docs/docs/deploy/deploy-cli/#deploy-kubeflow

Error:

% make apply                                                                                  
cluster_name_regex=^[a-z][-a-z0-9]{0,22}[a-z0-9]$
The kubeflow cluster name "kubeflow" is valid.
...
Component path: common/cnrm
Apply component resources: common/cnrm
Found Makefile, call 'make apply' of this component Makefile. 
echo ./build
./build
/Library/Developer/CommandLineTools/usr/bin/make apply-cnrm build_dir=./build
rm -rf ./build && mkdir -p ./build
kustomize build -o ./build ./
kubectl --context=mgmt-kubeflow apply -f ./build
computeaddress.compute.cnrm.cloud.google.com/kubeflow-ip unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-bigquery unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-cloudbuild unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-cloudsql unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-dataflow unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-dataproc unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-istio-wi unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-kubeflow-wi unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-logging unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-manages-user unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-metricwriter unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-ml unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-monitoringviewer unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-network unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-servicemanagement unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-source unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-storage unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-viewer unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-workload-identity-user unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-bigquery unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-cloudbuild unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-cloudsql unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-dataflow unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-dataproc unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-logging unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-metricwriter unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-ml unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-monitoringviewer unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-source unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-storage unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-viewer unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-workload-identity-user-ml-pipeline-ui unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-workload-identity-user-ml-pipeline-visualizationserver unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-workload-identity-user-pipeline-runner unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-logging unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-policy-cloudtrace unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-policy-meshtelemetry unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-policy-monitoring-viewer unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-policy-monitoring unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-policy-storage unchanged
iamserviceaccount.iam.cnrm.cloud.google.com/kubeflow-admin unchanged
iamserviceaccount.iam.cnrm.cloud.google.com/kubeflow-user unchanged
iamserviceaccount.iam.cnrm.cloud.google.com/kubeflow-vm unchanged
service.serviceusage.cnrm.cloud.google.com/anthos.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/cloudbuild.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/cloudresourcemanager.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/compute.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/container.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/gkeconnect.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/gkehub.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/iamcredentials.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/iap.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/logging.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/meshca.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/meshconfig.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/meshtelemetry.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/monitoring.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/servicemanagement.googleapis.com unchanged
Error from server (BadRequest): error when creating "build/container.cnrm.cloud.google.com_v1beta1_containercluster_kubeflow.yaml": ContainerCluster in version "v1beta1" cannot be handled as a ContainerCluster: strict decoding error: unknown field "metadata.clusterName"
make[2]: *** [apply-cnrm] Error 1
make[1]: *** [apply] Error 2
make: *** [apply] Error 1

As I understand it v1beta1 ContainerCluster spec includes metadata.name, not metadata.clusterName.
@gkcalat
Copy link
Contributor

gkcalat commented May 8, 2023

Hi @chusband,

Could you please share the entire log and all steps to reproduce starting from cloning the repo?

@gkcalat gkcalat self-assigned this May 8, 2023
@gkcalat gkcalat mentioned this issue May 8, 2023
Closed
8 tasks
@den1den
Copy link

den1den commented May 10, 2023

I managed to get around this error by removing the lines with clusterName in the following files:

  • kubeflow-distribution/kubeflow/common/cluster/upstream/cluster.yaml
  • kubeflow-distribution/kubeflow/common/cluster/upstream/nodepool.yaml

I'm still new to Kubeflow / GKE so I'm not sure what the implications are but it's worth a shot

@chusband
Copy link
Author

I followed all the steps in https://googlecloudplatform.github.io/kubeflow-gke-docs/docs/deploy/project-setup/
up to https://googlecloudplatform.github.io/kubeflow-gke-docs/docs/deploy/project-setup/
and make apply

% make apply
cluster_name_regex=^[a-z][-a-z0-9]{0,22}[a-z0-9]$
The kubeflow cluster name "kubeflow" is valid.
PROJECT=<redacted> NAME=kubeflow ./hack/check_domain_length.sh
Build directory: ./build
Component path: common/managed-storage
Apply component resources: common/managed-storage
Found Makefile, call 'make apply' of this component Makefile. 
rm -rf ./build
mkdir -p ./build
kustomize build -o ./build/ .
kubectl --context=mgmt-kubeflow apply -f ./build
sqlinstance.sql.cnrm.cloud.google.com/kubeflow-kfp unchanged
storagebucket.storage.cnrm.cloud.google.com/<redacted>-kfp unchanged
# Wait for all Google Cloud resources to get created and become ready.
# If this takes long, you can view status by:
cd common/managed-storage && make status
# For resources with READY=False, debug by:
kubectl --context=mgmt-kubeflow -n <redacted> describe <KIND>/<NAME>

kubectl --context=mgmt-kubeflow wait --for=condition=Ready --timeout=100s -f ./build \
		|| kubectl --context=mgmt-kubeflow get -f ./build
sqlinstance.sql.cnrm.cloud.google.com/kubeflow-kfp condition met
storagebucket.storage.cnrm.cloud.google.com/<redacted>-kfp condition met
kubectl --context=mgmt-kubeflow wait --for=condition=Ready --timeout=500s -f ./build
sqlinstance.sql.cnrm.cloud.google.com/kubeflow-kfp condition met
storagebucket.storage.cnrm.cloud.google.com/<redacted>-kfp condition met
Build directory: ./build
Component path: common/cnrm
Apply component resources: common/cnrm
Found Makefile, call 'make apply' of this component Makefile. 
echo ./build
./build
/Library/Developer/CommandLineTools/usr/bin/make apply-cnrm build_dir=./build
rm -rf ./build && mkdir -p ./build
kustomize build -o ./build ./
kubectl --context=mgmt-kubeflow apply -f ./build
computeaddress.compute.cnrm.cloud.google.com/kubeflow-ip unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-bigquery unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-cloudbuild unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-cloudsql unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-dataflow unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-dataproc unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-istio-wi unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-kubeflow-wi unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-logging unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-manages-user unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-metricwriter unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-ml unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-monitoringviewer unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-network unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-servicemanagement unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-source unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-storage unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-viewer unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-admin-workload-identity-user unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-bigquery unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-cloudbuild unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-cloudsql unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-dataflow unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-dataproc unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-logging unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-metricwriter unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-ml unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-monitoringviewer unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-source unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-storage unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-viewer unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-workload-identity-user-ml-pipeline-ui unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-workload-identity-user-ml-pipeline-visualizationserver unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-user-workload-identity-user-pipeline-runner unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-logging unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-policy-cloudtrace unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-policy-meshtelemetry unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-policy-monitoring-viewer unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-policy-monitoring unchanged
iampolicymember.iam.cnrm.cloud.google.com/kubeflow-vm-policy-storage unchanged
iamserviceaccount.iam.cnrm.cloud.google.com/kubeflow-admin unchanged
iamserviceaccount.iam.cnrm.cloud.google.com/kubeflow-user unchanged
iamserviceaccount.iam.cnrm.cloud.google.com/kubeflow-vm unchanged
service.serviceusage.cnrm.cloud.google.com/anthos.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/cloudbuild.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/cloudresourcemanager.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/compute.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/container.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/gkeconnect.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/gkehub.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/iamcredentials.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/iap.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/logging.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/meshca.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/meshconfig.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/meshtelemetry.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/monitoring.googleapis.com unchanged
service.serviceusage.cnrm.cloud.google.com/servicemanagement.googleapis.com unchanged
Error from server (BadRequest): error when creating "build/container.cnrm.cloud.google.com_v1beta1_containercluster_kubeflow.yaml": ContainerCluster in version "v1beta1" cannot be handled as a ContainerCluster: strict decoding error: unknown field "metadata.clusterName"
make[2]: *** [apply-cnrm] Error 1
make[1]: *** [apply] Error 2
make: *** [apply] Error 1

@zijianjoy
Copy link
Collaborator

We might need to consider using annotation to specify a target project: https://cloud.google.com/config-connector/docs/reference/resource-docs/container/containercluster#annotations

@dev-onejun
Copy link

Hi, I'm a student who is a newbie for Kubeflow.

I managed to get around this error by removing the lines with clusterName in the following files:

  • kubeflow-distribution/kubeflow/common/cluster/upstream/cluster.yaml
  • kubeflow-distribution/kubeflow/common/cluster/upstream/nodepool.yaml

I'm still new to Kubeflow / GKE so I'm not sure what the implications are but it's worth a shot

It works for me too, but
is it due to the fact that the api server changes required form?

And now, to solve the problem, discussing a usage of annotation is required. Do I understand correctly?

@zijianjoy
Copy link
Collaborator

zijianjoy commented May 18, 2023 •
edited

  1. Kubeflow cluster is supposed to be independent from Management cluster in terms of GCP projects. (Management cluster can manage Kubeflow cluster in another GCP project). I think we can represent GCP project id which Kubeflow cluster lives, by using namespace field in cluster.yaml file. Example:

    kubeflow-distribution/kubeflow/common/cnrm/kustomization.yaml

    Line 8 in e24e730

    namespace: PROJECT # kpt-set: ${gcloud.core.project}
    .
  2. I think location field can represent the GKE cluster location within the given GCP project, example:

    kubeflow-distribution/kubeflow/common/cluster/upstream/cluster.yaml

    Line 56 in e24e730

    location: LOCATION # kpt-set: ${location}
    .
  3. The name field should represent Kubeflow cluster name:

    kubeflow-distribution/kubeflow/common/cluster/upstream/cluster.yaml

    Line 23 in e24e730

    name: KUBEFLOW-NAME # kpt-set: ${name}
    .

If combining all points above, I tend to think we can safely remove clusterName: "PROJECT/LOCATION/KUBEFLOW-NAME", given the information provided by the community in this thread. As long as we add proper namespace configuration mentioned by bullet point number 1.

@baelamri-ankorstore
Copy link

baelamri-ankorstore commented May 22, 2023 •
edited

what is the preferred workaround today to run kubeflow ?

  • change the yaml file
  • dowgrade kustomize
  • dowgrade kuberflow
  • add --validate=false to kubectl apply ...
  • ...

@Linchin
Copy link
Contributor

Linchin commented Jun 27, 2023

Verified that removing the two lines does not affect the ability to create kubeflow cluster in a different location or project.

@Linchin
Copy link
Contributor

Linchin commented Jun 28, 2023

closed by #426 and #427.

@Linchin Linchin closed this as completed Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gkcalat gkcalat

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@den1den @chusband @Linchin @gkcalat @zijianjoy @dev-onejun @baelamri-ankorstore