[WIP] Docker registry re-deployment tips by miminar · Pull Request #8666 · openshift/openshift-docs

miminar · 2018-04-10T15:12:38Z

miminar · 2018-04-10T15:20:40Z

@dmage, @legionus, @bparees WDYT? Too verbose? Or shall I mention anything else?

miminar · 2018-04-10T15:25:45Z

Rendered: https://miminar.fedorapeople.org/_preview/openshift-enterprise/registry-redeploy/install_config/registry/deploy_registry_existing_clusters.html#redeploy-registry

miminar · 2018-04-10T15:29:40Z

@tnozicka Any change regarding this for 3.10/3.11?

I think roullout is supported since like Kubernetes 1.7

don't say unfortunately, we apologize for nothing! :) (but seriously, it strikes an awkward tone. just state the facts).

I think roullout is supported since like Kubernetes 1.7

Well, this is what I get with oc v3.9.16-1+23bf34b-dirty:

$ oc rollout latest ds/docker-registry error: docker-registry is not a deployment config

don't say unfortunately

My disappontment leaked into the text. I'll try to be less emotional ;-).

You said oc rollout, not oc rollout latest. DS (with rolling Update strategy set) rollout automatically on config change. You can do e.g. oc rollout undo daemonset <daemonset-name> --to-revision=<revision> to rollback. If you need to force redeployment without a change in PodSpec you need to e.g. annotate the pod template with current time until we implement rolling-restart.

@tnozicka thanks, that looks promising, I think I'll use this.

miminar · 2018-04-10T15:29:55Z

@tnozicka similar question.

not sure, but feel free to create an issue if that's not supported. oc create service and using the labels from DS should work though

miminar · 2018-04-10T15:31:01Z

@tnozicka Is there a better command that will not cause noticeable down-time?

https://v1-9.docs.kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/

set the strategy to rollingUpdate and insert a label with current time into the pod template in the DS to force it to redeploy (rolling-restart command is planned to avoid this)

oc rollout? (won't cause downtime if the strategy is rolling)

(not applicable to daemonsets of course)

oc patch ds <ds_name> -p='{"spec":{"template":{"metadata":{"annotations":{"timestamp":"'$(date +%s%N)'"}}}}}'

You want my contribution to become the least readable piece ever. I'm missing a note type [NerdsOnly] :-)

But I like the hack nevertheless.

dmage · 2018-04-10T16:28:05Z

AFAIK image streams are created with an IP address and it's asynchronously replaced by the FQDN, so FQDN may appear with a little delay.

I think that happens only when you have master config misconfigured, at least from the issues we have been seeing with double deployments.

The hostname specified in the configuration file takes precedence. If not present, OPENSHIFT_DEFAULT_REGISTRY env var is used. When unset, the master evaluates service's "${DOCKER_REGISTRY_SERVICE_HOST}:${DOCKER_REGISTRY_SERVICE_PORT}".

With oc cluster up, all is unset until the service exists (~~or maybe is up - need to verify~~ so just the service's existence matters). Nevetheless, good point - I need to explain the empty output as well.

TL;DR; if there's no output, no explicit address is configured and the service does not yet exist.

bparees · 2018-04-10T20:08:40Z

s/is/are/

(docs team may have a better suggestion on cleaning up the informal language)

bparees · 2018-04-10T20:09:38Z

unclear what this means. Do you mean mean the service ip will not be changed as a result of redeploying, or is this a command to the user ("don't change the service ip when you redeploy")?

The latter. It's very easy to change the service IP and it should be avoided. I'll try to be more clear.

bparees · 2018-04-10T20:10:03Z

this is confusing after having just read that the IP won't be changed. why do i need to know this if the IP isn't going to change? When do i need to know it?

If you choose to change the service IP. I'll rephrase.

bparees · 2018-04-10T20:11:54Z

again unclear if this means "you must keep the service ip the same" or "the system will keep the service ip the same"

bparees · 2018-04-10T20:13:47Z

s/though/when the DNS name is used/

bparees · 2018-04-10T20:58:10Z

if what isn't the case? if they don't match? again what if they were using a hostname?

and what if a custom IP is being used? (custom ip for what? the service ip?)

bparees · 2018-04-10T20:59:26Z

don't you need to delete the existing "docker-registry" service first?

bparees · 2018-04-10T21:00:06Z

oc rollout? (won't cause downtime if the strategy is rolling)

bparees · 2018-04-10T21:00:25Z

(not applicable to daemonsets of course)

bparees · 2018-04-10T21:03:33Z

why is this the error when the docker-registry service does not exist? That env var doesn't have anything to do w/ the service.

Yeah, this is a bit misleading. The REGISTRY_OPENSHIFT_SERVER_ADDR is the preferred env var for configuring this. If it's unset as well as some other deprecated vars with the same meaning, the registry looks for the variables inherited from the associated service ("DOCKER_REGISTRY_SERVICE_HOST").

I'm putting the explanation in there.

miminar · 2018-04-13T08:09:26Z

Apparently, there's a new deploy_registry.yml playbook for the re-deployment. I'll give it a shot and include it here.

Also the point of mainaining the registry's service IP is already in the docs but it's a bit outdated. I'm now merging the two.

Resolves: openshift/origin#10585 Signed-off-by: Michal Minář <miminar@redhat.com>

miminar · 2018-04-16T15:43:39Z

+reload their internal registry address.
+
+----
+$ oc env -n default ds/docker-registry "TIMESTAMP=$(date +%s%N)"


@tnozicka is this acceptable?

Env works as well but pollutes pod's env a bit (program can see it), annotation is clearer. I'll leave the choice with you :)

I get it's shorter but people will likely just copy paste it.

miminar · 2018-04-16T15:49:10Z

Rewritten. Merged a similar section from the extended configuration. Rendered here.

I still need to include instructions for using the stand-alone registry playbook.

miminar · 2018-04-16T15:52:11Z

+      {"op":"add","path":"/spec/sessionAffinity","value":"ClientIP"},
+      {"op":"add","path":"/spec/selector","value":{"docker-registry":"default"}}
+    ]' --local=true -o json | \
+    oc replace -n default -f - --force <2>


Couldn't find out anything simpler. Originally I wanted to use oc adm registry and filter out everything but the service template. But I couldn't pull it off without using external tools like jq.

miminar · 2018-04-16T15:55:50Z

 include the hostnames by which you expect the registry to be referenced.
 See xref:securing_and_exposing_registry.adoc#securing-the-registry[securing
 the registry] for instructions on adding hostnames to the server certificate.
+- 


TODO: restore the original

openshift-bot · 2018-07-06T22:34:12Z

@miminar: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vikram-redhat · 2018-08-01T01:49:24Z

@miminar - is this PR still valid?

miminar · 2018-08-01T07:04:32Z

@vikram-redhat I'm afraid this needs a complete rewrite. Also with the upcoming registry operator this will become obsolete. Unfortunately, I don't have time to continue work on this now.

vikram-redhat · 2018-08-01T07:05:34Z

@miminar - thanks for letting us know and closing this PR.

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 10, 2018

miminar force-pushed the registry-redeploy branch 2 times, most recently from 6279b96 to eca7637 Compare April 10, 2018 15:18

miminar mentioned this pull request Apr 10, 2018

Missing documentation on how to redeploy broken registry. openshift/origin#10585

Open

miminar commented Apr 10, 2018

View reviewed changes

dmage reviewed Apr 10, 2018

View reviewed changes

bparees requested changes Apr 10, 2018

View reviewed changes

miminar changed the title ~~Docker registry re-deployment tips~~ [WIP] Docker registry re-deployment tips Apr 11, 2018

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 11, 2018

miminar mentioned this pull request Apr 13, 2018

RFE a dedicated playbook for (re-)deploying integrated docker-registry openshift/openshift-ansible#7917

Closed

Docker registry re-deployment guidelines

ed3ad18

Resolves: openshift/origin#10585 Signed-off-by: Michal Minář <miminar@redhat.com>

miminar force-pushed the registry-redeploy branch from eca7637 to ed3ad18 Compare April 16, 2018 15:42

miminar commented Apr 16, 2018

View reviewed changes

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 6, 2018

miminar closed this Aug 1, 2018

Conversation

miminar commented Apr 10, 2018

Uh oh!

miminar commented Apr 10, 2018

Uh oh!

miminar commented Apr 10, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miminar Apr 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miminar commented Apr 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miminar Apr 11, 2018 •

edited

Loading

miminar commented Apr 13, 2018 •

edited

Loading