Use templates only to deploy Che to OpenShift #9190

ghost · 2018-03-22T15:45:32Z

The purpose of this PR is to:

Eventually get rid of deployment scripts to avoid deployment complexity
Use OpenShift way of creating objects when user input is required

What is done

OpenShift templates and other objects are introduced
All env vars are parameterized and a user can provide own values
Readme answers some questions and covers deployment of all Che flavors in all possible configurations.

Why not a single template?

There's no universal template that will fit all use cases. It will be easier to use upstream che-server template for those who depend on it.

Is it tested?

Yes. MiniShift and OSO. Both single and multi user. http and https

Will existing scripts go away?

Eventually, yes. But not immediately. As soon as those who depend on current deployment scripts are ready to migrate, scripts will be deleted.

Does Che deployment have all envs as parameters?

No, only those that make sense at a first glance. Further PRs are welcome.

Will these changes break CI tests?

Even though there are changes in eclipse/che-keycloak Dockerfile, a different entrypoint script is used in Keycloak deployment. So, both deployment script and templates works well, even though they use the same images.

What about Keycloak configuration pod?

Master and Che realm, as well as admin user for Keycloak and Che are created in keycloak image entrypoint.

Will docs be provided?

There's README.md to start with, but eventually docs will be updated to reference template based deployment method.

How to test?

Follow instructions in README. Use eivantsov/keycloak as Keycloak image for Keycloak deployment:

oc process -f multi/keycloak-template.yaml -p ROUTING_SUFFIX=$(minishift ip).nip.io -p IMAGE_KEYCLOAK=eivantsov/keycloak | oc apply -f -

sleshchenko · 2018-03-22T16:00:36Z

deploy/openshift/templates/README.md

+
+All of the below commands create a PVC for Che server and Che deployment uses this PVC.
+
+It is possible to avoid using PVC for Che server - just skip creating pvc `oc apply -f pvc/che-server-pvc.yaml` and `oc set volume dc/che --add -m /data --name=che-data-volume --claim-name=che-data-volume` commands.


Actually, PVC is used by Single-User Che to store H2 Database file. Please, add a note that without PVC Single-User Che will lose all data after Pod restarting.

Good point will do. For single user PVC is created by default - this is what I suggest in an example command.

But once again - good point.

sleshchenko · 2018-03-22T16:05:47Z

deploy/openshift/templates/che-server-template.yaml

+            value: "${PROTOCOL}://che-${NAMESPACE}.${ROUTING_SUFFIX}/agent-binaries/linux_amd64/bootstrapper/bootstrapper"
+          - name: CHE_INFRA_KUBERNETES_MACHINE__START__TIMEOUT__MIN
+            value: '5'
+          - name: CHE_INFRA_KUBERNETES_MASTER__URL


Why doesn't this parameter have value? It makes sense to configure it for multi-cluster mode (When Che Server is started on one OS cluster while Workspace are created on another one)

sleshchenko · 2018-03-22T16:07:17Z

deploy/openshift/templates/che-server-template.yaml

+          - name: CHE_INFRA_KUBERNETES_PVC_STRATEGY
+            value: "${CHE_INFRA_KUBERNETES_PVC_STRATEGY}"
+          - name: CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS
+            value: 'true'


Please add a variable here. It makes sense to configure it. On newest OpenShift instances it may be disabled, and workspace start may speed up.

sleshchenko · 2018-03-22T16:08:30Z

deploy/openshift/templates/che-server-template.yaml

+            value: '5'
+          - name: CHE_INFRA_KUBERNETES_MASTER__URL
+          - name: CHE_INFRA_KUBERNETES_OAUTH__TOKEN
+            value: "${TOKEN}"


Why OPENSHIFT_USERNAME, OPENSHIFT_PASSWORD but TOKEN? Maybe it would be better to name them in the same way.

sleshchenko · 2018-03-22T16:09:13Z

deploy/openshift/templates/che-server-template.yaml

+          - name: CHE_INFRA_KUBERNETES_MASTER__URL
+          - name: CHE_INFRA_KUBERNETES_OAUTH__TOKEN
+            value: "${TOKEN}"
+          - name: CHE_INFRA_KUBERNETES_PASSWORD


It makes sense to place CHE_INFRA_KUBERNETES_PASSWORD and CHE_INFRA_KUBERNETES_USERNAME together.

sleshchenko · 2018-03-22T16:10:52Z

deploy/openshift/templates/che-server-template.yaml

+            value: "${CHE_KEYCLOAK_AUTH__SERVER__URL}"
+          - name: CHE_JDBC_URL
+            value: "${CHE_JDBC_URL}"
+          - name: CHE_OAUTH_GITHUB_CLIENTID


How can it be configured if there is no specified value?

sleshchenko · 2018-03-22T16:11:34Z

deploy/openshift/templates/che-server-template.yaml

+    test: false
+    triggers:
+    - type: ConfigChange
+  status:


Are you sure that it makes sense to specify status here?

sleshchenko · 2018-03-22T16:13:46Z

deploy/openshift/templates/multi/postgres-template.yaml

+- apiVersion: v1
+  kind: DeploymentConfig
+  metadata:
+    creationTimestamp: null


Why do we need it?

Where is it removed? I still see few creationTimestamp fields in this template. Do I have an issue with browser cache?

riuvshin · 2018-03-22T16:17:23Z

dockerfiles/keycloak/Dockerfile

@@ -11,3 +11,4 @@ ADD . /scripts/
 USER root
 RUN chgrp -R 0 /scripts && \
    chmod -R g+rwX /scripts
+


is that mandatory change? :)

benoitf · 2018-03-22T16:29:34Z

deploy/openshift/templates/README.md

+
+## Templates and Parameters
+
+Templates are provided with a set of predefined params which you can override with with `-p key=value`. If you miss envs and parameters, you can add them to your template.


maybe we could add which command will list parameters for a given template ?

$ oc process --parameters -f <filename>

Added a note in readme

benoitf · 2018-03-22T16:30:55Z

deploy/openshift/templates/README.md

+
+## Creating Workspace Objects
+
+By default, che SA is used to create workspace objects in the same namespace with Che server. This is configurable - you may provide login and password or token, and set


FYI sometimes Che is written che sometimes Che

is there any word expected after and set ?

benoitf · 2018-03-22T16:32:58Z

deploy/openshift/templates/README.md

+3. Once done, go to `https://keycloak.${NAMESPACE}.${ROUTING_SUFFIX}`, log in to admin console.
+Default credentials are `admin:admin`.
+Go to Clients, `che-public` client and edit **Valid Redirect URIs** and **Web Origins** URLs so that they use **https** protocol.
+You do not need to do that if you initially deploy Che with 	https support.


like https here have spaces around

benoitf · 2018-03-22T16:34:07Z

deploy/openshift/templates/README.md

+oc apply -f pvc/che-server-pvc.yaml; \
+oc process -f che-server-template.yaml -p ROUTING_SUFFIX=$(minishift ip).nip.io \
+		-p PROTOCOL=https \
+		-p WS_PROTOCOL=wss \


could WS_PROTOCOL/TLS be automatically computed based on the fact that it's https ?

I wish a template can handle some ifs. I failed to find a good universal solution. Any help is appreciated.

l0rd · 2018-03-22T17:38:37Z

deploy/openshift/templates/README.md

+```
+oc new-project che
+
+oc process -f che-server-template.yaml -p ROUTING_SUFFIX=$(minishift ip).nip.io | oc apply -f -; \


Instead oc process -f <template> | oc apply -f - you could use a single command oc new-app -f <template> .... .

Besides the fact that it's a single command oc new-app has an option to add applications env vars even if these were not defined in the template: -e ENV_VAR_KEY=ENV_VAR_VALUE (c.f. oc new-app reference). That may be useful for our use cases.

Good point. Examples updated

garagatyi · 2018-03-23T08:28:50Z

deploy/openshift/templates/README.md

+
+Templates are provided with a set of predefined params which you can override with with `-p key=value`. If you miss envs and parameters, you can add them to your template.
+
+IMPORTANT! **ROUTING_SUFFIX** value in the below commands `$(minishift ip).nip.io` works for MiniShift only. You MUST provide **own routing suffix** when deploying to OCP cluster.


It would be useful to have a command to get this value if there is a command for that.

I wish I knew such a command :)

could we create a dummy service then we got the default routing suffix and then we can display/reuse it ?

Not sure I understand. Dummy service for what?

@eivantsov If I create a dummy application on openshift like nginx app and I create a service nginx and expose port 80, openshift will provide me an URL. From this URL I can guess the "routing suffix" so user has a way to find it (through a command that can try to create app, create service and route and then display the url and delete project)

It's just to have a way to provide to users "hey we can give you some hints around this routing suffix"

A lame workaround may be updating Che dc after routes for Che and Keycloak are created. This will add complexity to commands, but this way, no input is required from a user

in my approach, it was a way to see what the route could be.
in your approach, we create using defaults, grab values and update again.

in both case it would help "newcomers" so I'm fine.

Added a sub-section on How to get routing suffix.

garagatyi · 2018-03-23T09:27:47Z

deploy/openshift/templates/README.md

+```
+oc new-project che
+
+oc process -f che-server-template.yaml -p ROUTING_SUFFIX=$(minishift ip).nip.io | oc apply -f -; \


Interestingly oc process supports golang templates with -t option and looks like they support if clauses https://golang.org/pkg/text/template/
With this approach, we would have something similar to helm I suppose.

garagatyi · 2018-03-23T09:31:47Z

deploy/openshift/templates/README.md

+
+oc process -f che-server-template.yaml -p ROUTING_SUFFIX=$(minishift ip).nip.io | oc apply -f -; \
+oc apply -f pvc/che-server-pvc.yaml; \
+oc set volume dc/che --add -m /data --name=che-data-volume --claim-name=che-data-volume


I would suggest using set before applying template

We don't have dc/che at that point. I need to create Che deploymentConfig and then set volume for it.

The spec says oc set volumes RESOURCE/NAME --add|--remove|--list [options] so I need an object to modify, while process just produces output.

garagatyi · 2018-03-23T10:13:48Z

deploy/openshift/templates/README.md

+
+By default, che SA is used to create workspace objects in the same namespace with Che server. This is configurable - you may provide login and password or token, and set
+
+## Upgrade from http to https


I would put this section somewhere in the end - more advanced stuff

garagatyi · 2018-03-23T10:14:41Z

deploy/openshift/templates/README.md

+
+## Deploy multi user Che with bundled Keycloak and Postgres (https)
+
+```


Looks like step with creation of the project is missing here

Looks like it is not.

I can see it

garagatyi · 2018-03-23T13:02:20Z

deploy/openshift/templates/che-server-template.yaml

+            timeoutSeconds: 60
+          name: che
+          ports:
+          - containerPort: 8080


Looks like ports are declared twice

garagatyi · 2018-03-23T14:43:34Z

deploy/openshift/templates/multi/keycloak-template.yaml

+          persistentVolumeClaim:
+            claimName: keycloak-log
+    test: false
+  status: {}


Is it needed?

garagatyi · 2018-03-23T14:43:40Z

deploy/openshift/templates/multi/keycloak-template.yaml

+    resources:
+      requests:
+        storage: 1Gi
+  status: {}


Is it needed?

garagatyi · 2018-03-23T14:43:46Z

deploy/openshift/templates/multi/keycloak-template.yaml

+    resources:
+      requests:
+        storage: 1Gi
+  status: {}


Is it needed?

garagatyi · 2018-03-23T14:43:51Z

deploy/openshift/templates/multi/postgres-template.yaml

+          persistentVolumeClaim:
+            claimName: postgres-data
+    test: false
+  status: {}


Is it needed?

garagatyi · 2018-03-23T14:43:58Z

deploy/openshift/templates/multi/postgres-template.yaml

+      targetPort: 5432
+    selector:
+      io.kompose.service: postgres
+  status:


Is it needed?

garagatyi · 2018-03-23T14:44:03Z

deploy/openshift/templates/multi/postgres-template.yaml

+    resources:
+      requests:
+        storage: 1Gi
+  status: {}


Is it needed?

gorkem · 2018-04-12T02:13:09Z

Do we want to retire the minishift add-on eventually as well? @l0rd @eivantsov

ghost · 2018-04-12T05:12:54Z

@gorkem good question. That's two sets of yamls to maintain. Though from user's perspective, applying an addon is somewhat easier.

I tried to craft a simple CLI that deploys to MiniShift and OCP https://github.com/eivantsov/che-cli, lame code but works well, and all a user needs to do is:

oc login
oc new-project che
./eclipse-che deploy

to deploy to MiniShift. Works with OSO Pro too. I know Florent has some rust cli too.

Maybe we should look in this direction and start with an OpenShift CLI that will make it easy to deploy to any OpenShift cluster?

l0rd · 2018-04-12T09:58:11Z

@gorkem that's a good point. A copy of these yaml files should be in rh-che repo too. So there will be 3 copies in 3 different repositories. Even if we retire the minishift addon we will still have the problem with rh-che. What about moving the yaml in a distinct github repository and referring to this repository? Then yaml could be versioned and published on maven central as part of the build. In this case the deployment scripts (and minishift addons) should download the yaml from the maven central as part of the deployment.

ghost · 2018-04-12T10:23:08Z

@l0rd can't rh use templates for upsteam repo?

l0rd · 2018-04-12T10:34:34Z

@l0rd can't rh use templates for upsteam repo?

How? I mean we could add instructions for rh-che developers to clone upstream repo and use these yaml but that's not ideal. And the second problem is that the openshift.io CI/CD is not so flexible. Currently that's how rh-che deployment config for osio is specified.

ghost · 2018-04-12T12:21:45Z

@l0rd yes, agree.. Even now, for multi user setup a user will have to git clone Che to get all yamls. Having them versioned and packaged as tar.gz is a very good idea.

benoitf · 2018-04-12T12:23:17Z

@l0rd could we just add a pom.xml on these upstream templates so that they could be published on m2 repository ?

ghost · 2018-04-12T12:25:31Z

@benoitf do you mean maven central or Che nexus?

benoitf · 2018-04-12T12:30:01Z

@eivantsov if #6444 is done it should be maven central as well :-)

garagatyi · 2018-04-12T14:02:49Z

@benoitf 👍 Yeah, we don't really need an additional repo to publish the artifact

l0rd · 2018-04-17T09:28:38Z

@eivantsov this PR looks like Penelope's shroud :-) What is missing to merge it? I am asking because I have opened a similar PR and I am waiting impatiently for this one to be merged.

ghost · 2018-04-17T11:29:47Z

@l0rd thanks for your patience and a wonderful metaphor. I have been waiting for all interested parties to have their say on templates themselves, cross platform cli that deploys to all infras and related stuff. Besides, I have introduced some changes to ocp.sh that QE uses to run daily tests, and while testing discovered a bug where PVCs were not deleted upon workspace deletion if workspace objects are created in a shared dedicated namespace (#9431) (default behavior in new templates).

Now that that QA tests show no regression , with a light heart, I merge this PR. Thanks for all suggestions and helpful remarks - they did help make templates and deployment docs better.

P.S. Old scripts and deployment yamls will stay here for a while and will be deleted in a separate PR.

Related docs PR: eclipse-che/che-docs#391

Use templates only to deploy Che to OpenShift

03da30c

ghost requested review from benoitf, garagatyi, l0rd, riuvshin and sleshchenko as code owners March 22, 2018 15:45

Avoid breaking existing scripts

36432cc

benoitf added the status/code-review This issue has a pull request posted for it and is awaiting code review completion by the community. label Mar 22, 2018

Avoid breaking existing scripts

25f5418

sleshchenko reviewed Mar 22, 2018

View reviewed changes

riuvshin reviewed Mar 22, 2018

View reviewed changes

benoitf reviewed Mar 22, 2018

View reviewed changes

l0rd reviewed Mar 22, 2018

View reviewed changes

garagatyi reviewed Mar 23, 2018

View reviewed changes

Fixes

9dc824f

garagatyi reviewed Mar 23, 2018

View reviewed changes

deploy/openshift/templates/multi/keycloak-template.yaml Outdated

resources:

requests:

storage: 1Gi

status: {}

Copy link

garagatyi Mar 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it needed?

garagatyi reviewed Mar 23, 2018

View reviewed changes

deploy/openshift/templates/multi/keycloak-template.yaml Outdated

resources:

requests:

storage: 1Gi

status: {}

Copy link

garagatyi Mar 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it needed?

garagatyi reviewed Mar 23, 2018

View reviewed changes

deploy/openshift/templates/multi/postgres-template.yaml Outdated

resources:

requests:

storage: 1Gi

status: {}

Copy link

garagatyi Mar 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it needed?

Eugene Ivantsov added 4 commits April 4, 2018 14:23

Add missing keycloak param

8e91a7f

Changes to ocp.sh to use new templates. Fix Keycloak template

05b532c

Merge branch 'master' into templates

ceb5a41

Env support

2444a4b

Update server yaml. Update ocp.sh

ebfb913

Configure CHE_IMAGE and CHE_TAG

829f1f8

ghost mentioned this pull request Apr 13, 2018

K8s multi. Admin guide. New OpenShift deployment docs eclipse-che/che-docs#391

Merged

Eugene Ivantsov added 6 commits April 13, 2018 18:36

Add recycler

05bf51c

Revern recycler sa

be179b2

Using credentials and creating ws in separate namespaces

309a6c0

Use custom Keycloak image

cba747d

Use custom Keycloak image

2e88cd6

Remove use of credentials

c980af9

ghost merged commit fa046bd into master Apr 17, 2018

ghost deleted the templates branch April 17, 2018 11:30

benoitf removed the status/code-review This issue has a pull request posted for it and is awaiting code review completion by the community. label Apr 17, 2018

benoitf added this to the 6.4.0 milestone Apr 17, 2018

ghost mentioned this pull request Apr 22, 2018

Update deploy_che script #9513

Merged

This pull request was closed.


		All of the below commands create a PVC for Che server and Che deployment uses this PVC.

		It is possible to avoid using PVC for Che server - just skip creating pvc `oc apply -f pvc/che-server-pvc.yaml` and `oc set volume dc/che --add -m /data --name=che-data-volume --claim-name=che-data-volume` commands.


		## Templates and Parameters

		Templates are provided with a set of predefined params which you can override with with `-p key=value`. If you miss envs and parameters, you can add them to your template.


		## Creating Workspace Objects

		By default, che SA is used to create workspace objects in the same namespace with Che server. This is configurable - you may provide login and password or token, and set


		Templates are provided with a set of predefined params which you can override with with `-p key=value`. If you miss envs and parameters, you can add them to your template.

		IMPORTANT! ROUTING_SUFFIX value in the below commands `$(minishift ip).nip.io` works for MiniShift only. You MUST provide own routing suffix when deploying to OCP cluster.


		By default, che SA is used to create workspace objects in the same namespace with Che server. This is configurable - you may provide login and password or token, and set

		## Upgrade from http to https


		## Deploy multi user Che with bundled Keycloak and Postgres (https)

		```

Use templates only to deploy Che to OpenShift #9190

Use templates only to deploy Che to OpenShift #9190

Conversation

ghost commented Mar 22, 2018

The purpose of this PR is to:

What is done

Why not a single template?

Is it tested?

Will existing scripts go away?

Does Che deployment have all envs as parameters?

Will these changes break CI tests?

What about Keycloak configuration pod?

Will docs be provided?

How to test?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benoitf Mar 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost Mar 23, 2018 • edited by ghost Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost Mar 23, 2018 • edited by ghost Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gorkem commented Apr 12, 2018

ghost commented Apr 12, 2018

l0rd commented Apr 12, 2018

ghost commented Apr 12, 2018

l0rd commented Apr 12, 2018 • edited Loading

ghost commented Apr 12, 2018

benoitf Mar 22, 2018 •

edited

Loading

ghost Mar 23, 2018 •

edited by ghost

Loading

ghost Mar 23, 2018 •

edited by ghost

Loading

l0rd commented Apr 12, 2018 •

edited

Loading