Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add conversion v1 v2 #423

Closed
wants to merge 33 commits into from
Closed

Conversation

dvaldivia
Copy link
Collaborator

Bumps the recent changes to Tenant to v2 api version, and introduces a conversion webhook so that the k8s API can convert between the versions and keep compatibility.

Updates all code to work with v2 by default.

@dvaldivia dvaldivia added the wip label Jan 13, 2021
@dvaldivia dvaldivia self-assigned this Jan 13, 2021
@dvaldivia dvaldivia force-pushed the add-conversion-v1-v2 branch 2 times, most recently from ba415d4 to d019b25 Compare January 19, 2021 19:01
@dvaldivia dvaldivia removed the wip label Jan 20, 2021
@dvaldivia dvaldivia force-pushed the add-conversion-v1-v2 branch 2 times, most recently from 364c0eb to 8119bc8 Compare January 20, 2021 01:01
Copy link
Member

@harshavardhana harshavardhana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suspect since this a large change, this would need a good amount of testing.
Not sure what is the tentative date we should think for a new release?

@dvaldivia
Copy link
Collaborator Author

@harshavardhana agree, testing is needed, I've been doing it myself, and the one that is the most important is testing that a user of v3.0.29 can install v4.0.0 and do it smoothly, even if we changed the CRD, I'd like to merge this before any other changes to the CRD

Copy link
Contributor

@nitisht nitisht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plugin doesn't start with this image

kubectl minio init -i nitishtiwari/operator:v1v2
2021/01/21 18:10:27 common.go:97: file does not exist

Copy link
Contributor

@nitisht nitisht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I followed these steps:

  • Deploy v3.0.29 Operator
  • Create a tenant
  • Patch the operator pod with new image based on this PR with this command
kubectl patch pod minio-operator-6f5b8cdcff-vkjj7 --type='json' -p='[{"op": "replace", "path": "/spec/containers/0/image", "value":"nitishtiwari/operator:v1v2"}]'
  • Create a fresh tenant --> Fails
  • Upgrade the existing tenant to a new image --> Fails

@dvaldivia
Copy link
Collaborator Author

@nitisht I just fixed it in my latest commit, can you try your steps again?

@dvaldivia
Copy link
Collaborator Author

dvaldivia commented Jan 21, 2021

To test:

1.- Install an older Operator

kubectl apply -k github.com/minio/operator/\?ref\=v3.0.29
cd /tmp/
git clone git@github.com:minio/operator.git
cd examples
k apply -f tenant.yaml

Now wait for the tenant to be ready
2.- Upgrade to the operator in this branch, change operator-kustomize/deployment.yaml to match your image then, on this branch of code

kustomize build . | kubectl apply -f -

3.- Create a tenant that uses zones with apiVersion "minio.min.io/v1"
4.- Create a tenant that uses pools with apiVersion "minio.min.io/v2"

The previous tenant and the new tenants, all should work cc @harshavardhana @nitisht @Alevsk

@nitisht
Copy link
Contributor

nitisht commented Jan 22, 2021

To test:

1.- Install an older Operator

kubectl apply -k github.com/minio/operator/\?ref\=v3.0.29
cd /tmp/
git clone git@github.com:minio/operator.git
cd examples
k apply -f tenant.yaml

Now wait for the tenant to be ready
2.- Upgrade to the operator in this branch, change operator-kustomize/deployment.yaml to match your image then, on this branch of code

kustomize build . | kubectl apply -f -

3.- Create a tenant that uses zones with apiVersion "minio.min.io/v1"
4.- Create a tenant that uses pools with apiVersion "minio.min.io/v2"

The previous tenant and the new tenants, all should work cc @harshavardhana @nitisht @Alevsk

I am getting this error for apiVersion minio.min.io/v2

$ kubectl apply -f examples/tenant.yaml 
secret/minio-creds-secret unchanged
secret/console-secret unchanged
error: unable to recognize "examples/tenant.yaml": no matches for kind "Tenant" in version "minio.min.io/v2"

@nitisht
Copy link
Contributor

nitisht commented Jan 22, 2021

With minio.min.io/v1 the yaml gets applied but it doesn't create the tenant. Operator pod logs

I0122 12:30:55.807257       1 main.go:72] Starting MinIO Operator
E0122 12:30:55.817553       1 main.go:131] Error getting CRD for adding caBundle: customresourcedefinitions.apiextensions.k8s.io "tenants.minio.min.io" is forbidden: User "system:serviceaccount:default:minio-operator" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
I0122 12:30:55.818016       1 main-controller.go:248] Setting up event handlers
I0122 12:30:55.818131       1 main-controller.go:609] Starting Tenant controller
I0122 12:30:55.818141       1 main-controller.go:612] Waiting for informer caches to sync
E0122 12:30:55.820470       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
I0122 12:30:55.836719       1 main-controller.go:573] operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)
W0122 12:30:55.838243       1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
W0122 12:30:55.849968       1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
W0122 12:30:55.853729       1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
I0122 12:30:55.854146       1 csr.go:217] Start polling for certificate of csr/operator-default-csr, every 5s, timeout after 20m0s
E0122 12:30:57.108789       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:30:59.813385       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
W0122 12:31:01.042122       1 warnings.go:70] certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
I0122 12:31:01.045723       1 csr.go:242] Certificate successfully fetched, creating secret with Private key and Certificate
I0122 12:31:01.085914       1 main-controller.go:576] Waiting for the operator certificates to be issued%!(EXTRA string=waiting for Operator cert)
E0122 12:31:03.235073       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:31:10.641544       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
I0122 12:31:11.091966       1 main-controller.go:600] Starting api server
E0122 12:31:24.687351       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:31:57.996002       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:32:43.459231       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:33:37.875561       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:34:14.307149       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:34:55.733639       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:35:35.280374       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:36:19.353717       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:36:57.853783       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:37:36.653009       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:38:27.027815       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:39:07.859048       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:39:54.983820       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:40:33.781310       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:41:12.698099       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:41:48.898868       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:42:44.863096       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:43:35.770682       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:44:21.487749       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:45:09.709036       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:46:08.969742       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:46:41.359258       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:47:29.207115       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:48:00.984402       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:48:51.751763       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:49:26.467641       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:50:04.831647       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:50:50.756127       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:51:29.296484       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:52:07.763857       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:53:01.423711       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:53:42.280278       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:54:38.705247       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:55:17.623132       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:55:50.548200       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:56:49.862378       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:57:22.099511       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:58:01.453020       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:58:59.445828       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 12:59:51.705714       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 13:00:45.740578       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 13:01:37.651698       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 13:02:13.145944       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)
E0122 13:02:56.000927       1 reflector.go:138] k8s.io/client-go@v0.20.1/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: the server could not find the requested resource (get tenants.minio.min.io)

@Alevsk
Copy link
Contributor

Alevsk commented Jan 23, 2021

To test:

1.- Install an older Operator

kubectl apply -k github.com/minio/operator/\?ref\=v3.0.29
cd /tmp/
git clone git@github.com:minio/operator.git
cd examples
k apply -f tenant.yaml

Now wait for the tenant to be ready
2.- Upgrade to the operator in this branch, change operator-kustomize/deployment.yaml to match your image then, on this branch of code

kustomize build . | kubectl apply -f -

3.- Create a tenant that uses zones with apiVersion "minio.min.io/v1"
4.- Create a tenant that uses pools with apiVersion "minio.min.io/v2"

The previous tenant and the new tenants, all should work cc @harshavardhana @nitisht @Alevsk

Tested, worked for me using latest code in thus branch

pkg/apis/minio.min.io/v1/constants.go Outdated Show resolved Hide resolved
@nitisht
Copy link
Contributor

nitisht commented Jan 25, 2021

Tested, worked for me using latest code in thus branch

Yes, working for me as well.

nitisht
nitisht previously approved these changes Jan 25, 2021
Copy link
Contributor

@nitisht nitisht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, basic functionality tested, seems to work fine.

@ravindk89
Copy link
Contributor

What is the process (if any) for updating a tenant on the v1 API to v2?

@dvaldivia
Copy link
Collaborator Author

@ravindk89 the idea is that it will magically work, if you have the tenant v1 from v3.0.29 you should be able to start using the new operator and they will automatically convert to v1 and v2 as needed

@ravindk89
Copy link
Contributor

@dvaldivia good to know. Is this something the user needs to check the status on? I'd like to provide simple commands for checking the API status of a tenant after upgrading the operator.

@dvaldivia
Copy link
Collaborator Author

This is hitting us really hard kubernetes-sigs/controller-tools#448

dvaldivia and others added 5 commits January 27, 2021 14:54
Co-authored-by: Lenin Alevski <alevsk.8772@gmail.com>
Co-authored-by: Lenin Alevski <alevsk.8772@gmail.com>
Co-authored-by: Lenin Alevski <alevsk.8772@gmail.com>
Co-authored-by: Lenin Alevski <alevsk.8772@gmail.com>
Co-authored-by: Lenin Alevski <alevsk.8772@gmail.com>
Alevsk
Alevsk previously approved these changes Jan 27, 2021
harshavardhana
harshavardhana previously approved these changes Jan 28, 2021
@cesnietor
Copy link
Contributor

I created a tenant using operator v3.0.28:
Then updated operator changing image and using kustomize build . | k apply -f -.
Getting:

k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watchlist *v2.Tenant: conversion webhook for minio.min.io/v1, Kind=Tenant failed: Post https://operator.minio-operator.svc:4222/webhook/v1/crd-conversion?timeout=30s: service "operator" not found              

@cesnietor
Copy link
Contributor

cesnietor commented Jan 28, 2021

creating a second tenant with latest console with deps fixed causes:

021-01-28 01:41:53.723398 I | 2021/01/28 01:41:53 admin_tenants.go:880: Create Tenant.minio.min.io "cestenant2" is invalid: spec.zones: Required value                                          │
│ 2021-01-28 01:41:53.723566 I | 2021/01/28 01:41:53 error.go:44: original error: Tenant.minio.min.io "cestenant2" is invalid: spec.zones: Required value                                          │
│ 2021-01-28 01:41:53.723646 I | 2021/01/28 01:41:53 admin_tenants.go:566: deleting secrets created for failed tenant: cestenant2 if any    

Seems crd is still specting zones. That is a response from kubernetes. @dvaldivia ^

@harshavardhana harshavardhana dismissed stale reviews from Alevsk and themself via 0ce833e January 28, 2021 02:21
required:
- spec
type: object
served: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also add the v1 deprecation warning like this https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#version-deprecation

# This indicates the v1alpha1 version of the custom resource is deprecated.
    # API requests to this version receive a warning header in the server response.
    deprecated: true
    # This overrides the default warning returned to API clients making v1alpha1 API requests.
    deprecationWarning: "example.com/v1alpha1 CronTab is deprecated; see http://example.com/v1alpha1-v1 for instructions to migrate to example.com/v1 CronTab"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, I'll look into adding the deprecated fields, I can see it's not supported by controller-gen so I'll need to patch it

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, also sent a PR to controller-gen kubernetes-sigs/controller-tools#541


// Console Related Constants

// DefaultConsoleImage specifies the latest Console Docker hub image
const DefaultConsoleImage = "minio/console:v0.4.6"
const DefaultConsoleImage = "minio/console:v0.3.14"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not fixed yet

@nitisht
Copy link
Contributor

nitisht commented Jan 28, 2021

5.- notice the statefulset for the existing tenant gets updated which causes a rolling restart

Old tenant (v1) that gets migrated to v2 never gets to running state - since the operator is expecting a new statefulset name but the statefulset name in the tenant remains the same. Pods no longer have resolvable names.

API: SYSTEM()
Time: 13:14:12 UTC 01/28/2021
Error: lookup minio-ss-0-0.minio-hl.default.svc.cluster.local on 10.96.0.10:53: no such host (*net.DNSError)
       host=minio-ss-0-0.minio-hl.default.svc.cluster.local, elapsedTime=7 minutes elapsed
       5: cmd/endpoint.go:447:cmd.Endpoints.UpdateIsLocal()
       4: cmd/endpoint.go:631:cmd.CreateEndpoints()
       3: cmd/endpoint-ellipses.go:372:cmd.createServerEndpoints()
       2: cmd/server-main.go:139:cmd.serverHandleCmdArgs()
       1: cmd/server-main.go:378:cmd.serverMain()

API: SYSTEM()
Time: 13:14:12 UTC 01/28/2021
Error: lookup minio-ss-0-1.minio-hl.default.svc.cluster.local on 10.96.0.10:53: no such host (*net.DNSError)
       host=minio-ss-0-1.minio-hl.default.svc.cluster.local, elapsedTime=7 minutes elapsed
       5: cmd/endpoint.go:447:cmd.Endpoints.UpdateIsLocal()
       4: cmd/endpoint.go:631:cmd.CreateEndpoints()
       3: cmd/endpoint-ellipses.go:372:cmd.createServerEndpoints()
       2: cmd/server-main.go:139:cmd.serverHandleCmdArgs()
       1: cmd/server-main.go:378:cmd.serverMain()

API: SYSTEM()
Time: 13:14:12 UTC 01/28/2021
Error: lookup minio-ss-0-2.minio-hl.default.svc.cluster.local on 10.96.0.10:53: no such host (*net.DNSError)
       host=minio-ss-0-2.minio-hl.default.svc.cluster.local, elapsedTime=7 minutes elapsed
       5: cmd/endpoint.go:447:cmd.Endpoints.UpdateIsLocal()
       4: cmd/endpoint.go:631:cmd.CreateEndpoints()
       3: cmd/endpoint-ellipses.go:372:cmd.createServerEndpoints()
       2: cmd/server-main.go:139:cmd.serverHandleCmdArgs()
       1: cmd/server-main.go:378:cmd.serverMain()

@dvaldivia
Copy link
Collaborator Author

@nitisht just fixed the issue you ran into

@cesnietor
Copy link
Contributor

cesnietor commented Jan 28, 2021

error with tenant created using console v0.4.3:

error syncing 'default/cestenantv1': Tenant.minio.min.io "cestenantv1" is invalid: spec.pools.volumeClaimTemplate.metadata.creationTimestamp: Invalid value: "null": spec.pools.volumeClaimTemplate.metadata.creationTimestamp in body must be of type string: "null"   

yaml:

apiVersion: minio.min.io/v1
kind: Tenant
metadata:
  creationTimestamp: "2021-01-28T20:59:44Z"
  generation: 1
  managedFields:
  - apiVersion: minio.min.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:scheduler:
        .: {}
        f:name: {}
      f:spec:
        .: {}
        f:console:
          .: {}
          f:consoleSecret:
            .: {}
            f:name: {}
          f:image: {}
          f:replicas: {}
          f:resources:
            .: {}
            f:requests:
              .: {}
              f:memory: {}
        f:credsSecret:
          .: {}
          f:name: {}
        f:image: {}
        f:imagePullSecret: {}
        f:mountPath: {}
        f:zones: {}
    manager: console
    operation: Update
    time: "2021-01-28T20:59:44Z"
  - apiVersion: minio.min.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:console:
          f:imagePullPolicy: {}
        f:imagePullPolicy: {}
        f:podManagementPolicy: {}
      f:status:
        .: {}
        f:availableReplicas: {}
        f:currentState: {}
    manager: minio-operator
    operation: Update
    time: "2021-01-28T21:01:10Z"
  name: cestenantv1
  namespace: default
  resourceVersion: "3073"
  selfLink: /apis/minio.min.io/v1/namespaces/default/tenants/cestenantv1
  uid: ba02d1e4-2ef6-40af-8158-fcf12df45414
scheduler:
  name: ""
spec:
  console:
    consoleSecret:
      name: cestenantv1-console-secret
    image: minio/console:dev
    replicas: 1
    resources:
      requests:
        memory: 64Mi
  credsSecret:
    name: cestenantv1-secret
  image: minio/minio:RELEASE.2020-10-18T21-54-12Z
  imagePullSecret: {}
  mountPath: /export
  zones:
  - affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: v1.min.io/tenant
              operator: In
              values:
              - cestenantv1
            - key: v1.min.io/zone
              operator: In
              values:
              - zone-0
          topologyKey: kubernetes.io/hostname
    name: zone-0
    resources:
      requests:
        memory: "0"
    servers: 4
    volumeClaimTemplate:
      metadata:
        creationTimestamp: null
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: "2684354560"
        storageClassName: standard
      status: {}
    volumesPerServer: 1
status:
  availableReplicas: 4
  currentState: Initialized

Also during this it killed one pod and started failing searching for a secret eventhough tls is disabled for that tenant.

Events:
  Type     Reason       Age                   From                         Message
  ----     ------       ----                  ----                         -------
  Normal   Scheduled    7m41s                 default-scheduler            Successfully assigned default/cestenantv1-zone-0-3 to kind-control-plane
  Warning  FailedMount  5m38s                 kubelet, kind-control-plane  Unable to attach or mount volumes: unmounted volumes=[cestenantv1-tls], unattached volumes=[cestenantv1-tls default-token-j97rt data0]: timed out waiting for the condition
  Warning  FailedMount  3m20s                 kubelet, kind-control-plane  Unable to attach or mount volumes: unmounted volumes=[cestenantv1-tls], unattached volumes=[default-token-j97rt data0 cestenantv1-tls]: timed out waiting for the condition
  Warning  FailedMount  89s (x11 over 7m41s)  kubelet, kind-control-plane  MountVolume.SetUp failed for volume "cestenantv1-tls" : secret "cestenantv1-tls" not found
  Warning  FailedMount  63s                   kubelet, kind-control-plane  Unable to attach or mount volumes: unmounted volumes=[cestenantv1-tls], unattached volumes=[data0 cestenantv1-tls default-token-j97rt]: timed out waiting for the condition

@harshavardhana
Copy link
Member

error with tenant created using console v0.4.3:

why are we still using console v0.4.3 ? @cesnietor

@harshavardhana
Copy link
Member

Squashed and merged to master 7e532e6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants