Error when installing m3db operator in Docker for Desktop

[pat-wong-affirm]

Thanks for opening an issue for the M3DB Operator! We'd love to help you, but we need the following information included
with any issue:

• What version of the operator are you running? Please include the docker tag. If using master, please include the git
  SHA logged when the operator first starts.

manifest = https://raw.githubusercontent.com/m3db/m3db-operator/master/bundle.yaml

docker image tag = quay.io/m3db/m3db-operator:latest

also tried with docker image tag being v0.13.0, same result

• What version of Kubernetes are you running? Please include the output of kubectl version.

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:33:37Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:42:41Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}

MacOS - Docker For Desktop - built-in kubernetes

• What are you trying to do?

Install the m3db operator

• What did you expect to happen?

Operator pod would be in Running state

• What happened?

{"level":"info","ts":1642550015.150743,"msg":"using InCluster k8s config"}
{"level":"info","ts":1642550015.1522524,"msg":"starting Operator controller","controller":"m3db-cluster-controller"}
E0118 23:53:35.163925       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to list *v1alpha1.M3DBCluster: the server could not find the requested resource (get m3dbclusters.operator.m3db.io)
{"level":"fatal","ts":1642550015.1678174,"msg":"error running controller","error":"could not create or update CRD: error creating CRD 'm3dbclusters.operator.m3db.io': the server could not find the requested resource","errorVerbose":"the server could not find the requested resource\nerror creating CRD 'm3dbclusters.operator.m3db.io'\ncould not create or update CRD"}

[schallert]

Hey @pat-wong-affirm, thanks for reaching out. It looks like we haven't cut a release since merging #302, which makes the operator compatible with Kubernetes 1.21+.

Can you try using the master tag of the operator and letting me know if that fixes it? If so, we can likely solve this by cutting a new release. Thanks!

[pat-wong-affirm]

unfortunately, got the same error when using image tag quay.io/m3db/m3db-operator:master

it's possible this error is specific to Docker for Desktop, as another team member was able to get the operator working in an EKS cluster

[kendrickclark]

I am having this issue as well

[larsf]

I am also seeing this and I am also trying to deploy it on Docker for Desktop

[larsf]

could this be because the api server in the desktop environment is not running as localhost? main.go sets masterURL to be 127.0.0:8001, but in the desktop environment you have:

kubectl cluster-info
Kubernetes control plane is running at https://kubernetes.docker.internal:6443
CoreDNS is running at https://kubernetes.docker.internal:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

I tried adding
args: ["-debug", "-masterhost", "https://kubernetes.docker.internal:6443"]

to bundle.yaml
but it didn't help ...

[larsf]

I was able to get the operator to start by passing in:
["-debug=true", "-manage-crd=false", "-masterhost", "https://kubernetes.docker.internal:6443"]

(I believe only -manage-crd=false is really needed to get it to start)

I do see the same errors from the controller as documented earlier periodically, but it does stay running.

Trying to stand up a m3db cluster though, fails with this error:
no matches for kind "M3DBCluster" in version "operator.m3db.io/v1alpha

[larsf]

the m3db-operator seems to use interfaces that are no longer supported (k8s v1.22+):
46: crdClient := k.kubeExt.ApiextensionsV1beta1().CustomResourceDefin

see: storageos/cluster-operator#348

Server Version: version.Info{Major:"1", Minor:"22"

[kendrickclark]

Would a fix be to change that to ApiextensionsV1()

[larsf]

Hi, Partly, I have a diff to 4 files that changes v1beta1 to v1, but I haven't had a chance to build and test it yet

…

On Tue, Mar 29, 2022 at 9:19 PM kendrickclark ***@***.***> wrote: Would a fix be to change that to ApiextensionsV1() — Reply to this email directly, view it on GitHub <#310 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALESIJB7UFEHYSILVW3DLVCO2ZRANCNFSM5MIPM3HA> . You are receiving this because you commented.Message ID: ***@***.***>

[larsf]

Tried to do a build with my changes, but found that the .ci directory is empty, so the build fails because it cannot find go-build-ldflags.sh and common.mk

Any reason they are not in the source tree?

Answered my own question - found the scripts in the m3db/ci-scripts repo
Got further, now I see:
~/src/m3operator$ make
--- clean-all
build .: cannot find module for path .
make: *** [Makefile:194: clean-all] Error 1

[kendrickclark]

@larsf Any luck there?

[kendrickclark]

Hey @pat-wong-affirm, thanks for reaching out. It looks like we haven't cut a release since merging #302, which makes the operator compatible with Kubernetes 1.21+.

Can you try using the master tag of the operator and letting me know if that fixes it? If so, we can likely solve this by cutting a new release. Thanks!

schallert , Unfortunately this does not resolve the issue.

[niravpatel2008]

Kubernetes version 1.22

i am also facing similar issue on eks while installing operator using following command.

kubectl apply -f https://raw.githubusercontent.com/m3db/m3db-operator/master/bundle.yaml

error:

{"level":"info","ts":1657946530.7369597,"msg":"using InCluster k8s config"}
{"level":"info","ts":1657946530.7382486,"msg":"starting Operator controller","controller":"m3db-cluster-controller"}
{"level":"fatal","ts":1657946530.7782364,"msg":"error running controller","error":"could not create or update CRD: could not fetch CRD 'm3dbclusters.operator.m3db.io': customresourcedefinitions.apiextensions.k8s.io \"m3dbclusters.operator.m3db.io\" is forbidden: User \"system:serviceaccount:default:m3db-operator\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope","errorVerbose":"customresourcedefinitions.apiextensions.k8s.io \"m3dbclusters.operator.m3db.io\" is forbidden: User \"system:serviceaccount:default:m3db-operator\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope\ncould not fetch CRD 'm3dbclusters.operator.m3db.io'\ncould not create or update CRD"}

[niravpatel2008]

Kubernetes version 1.22

i am also facing similar issue on eks while installing operator using following command.

kubectl apply -f https://raw.githubusercontent.com/m3db/m3db-operator/master/bundle.yaml

error:

{"level":"info","ts":1657946530.7369597,"msg":"using InCluster k8s config"}
{"level":"info","ts":1657946530.7382486,"msg":"starting Operator controller","controller":"m3db-cluster-controller"}
{"level":"fatal","ts":1657946530.7782364,"msg":"error running controller","error":"could not create or update CRD: could not fetch CRD 'm3dbclusters.operator.m3db.io': customresourcedefinitions.apiextensions.k8s.io \"m3dbclusters.operator.m3db.io\" is forbidden: User \"system:serviceaccount:default:m3db-operator\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope","errorVerbose":"customresourcedefinitions.apiextensions.k8s.io \"m3dbclusters.operator.m3db.io\" is forbidden: User \"system:serviceaccount:default:m3db-operator\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope\ncould not fetch CRD 'm3dbclusters.operator.m3db.io'\ncould not create or update CRD"}

using operator image tag 0484dcc instead of 1.13.0 works for me. but it seems it has some security issues. wish it will gets resolved with 1.14.0