Build an offline version of the Devfile registry image #112

amisevsk · 2019-10-07T20:08:09Z

Important Note: this PR is set to target the override-images branch until #110 is merged, as it also requires modification of the container entrypoint. Once PR #110 is merged, I will update target branch to master.

What does this PR do?

Enables building an offline version of the devfile registry.

As in the similar feature for the plugin registry, this is done by specifying --target offline-registry during the docker build (with the current online registry being built by --target registry).

The offline registry downloads all project zip files for projects hosted on github and stores them in the /resources dir. Devfiles are updated to contain a placeholder, which is filled from an env var (CHE_DEVFILE_HTTPS_ENDPOINT) that must be set at runtime.

What issues does this PR fix or reference?

eclipse-che/che#14733

Additional Info

Tested on OpenShift 3.11, manually setting the required environment variable. It works generally as desired, except I encountered issue eclipse-che/che#14800.

Caveat

For this solution to be viable, all the sample projects to be bundled need to have their license/copyright info correctly configured.

See https://issues.jboss.org/browse/CRW-317?focusedCommentId=13798222&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13798222 for a list of the problems with the sample projects' licenses/copyright info.

amisevsk · 2019-10-07T20:38:35Z

Update: the last commit adds an attempt to determine the registry's service IP and port from k8s env vars when the required environment variable is not set. This should work in many cases but is rather fragile. I haven't tested it for workspaces in another namespace, etc.

amisevsk · 2019-10-10T22:13:06Z

Squashed and rebased onto master now that #110 is merged; PR is ready for review

cc: @l0rd @nickboldt

davidfestal · 2019-10-11T08:35:33Z

build/dockerfiles/entrypoint.sh

+# In addition, this script will perform the necessary set up for the offline
+# devfile registry, replacing placeholders in all devfiles based off environment
+# variable
+#     CHE_DEVFILE_HTTPS_ENDPOINT


So I assume that in airgap mode, from the operator, we would also have to add this variable in the new devfile registry config map, based on the external URL of the devfile registry ?

@l0rd do you confirm ?

cc @tomgeorge

Yes, that's accurate; we need the environment variable to be able to rewrite devfiles to point at exposed endpoints. Without the environment variable we do try to use k8s-provisioned env vars to determine service IP and route, but I haven't tested this much and it may be unreliable.

@davidfestal yes that's the weakness of this solution. There is no reliable way to figure the value automatically.

In particular, without the env var set, we try to go off of

URL="http://${CHE_DEVFILE_REGISTRY_SERVICE_HOST}:${CHE_DEVFILE_REGISTRY_SERVICE_PORT}"

which depends on the devfile registry server being called che-devfile-registry, and may not work in some scenarios (I don't know the internals of networking on OpenShift).

davidfestal · 2019-10-11T08:40:37Z

@l0rd @nickboldt So I assume that if we would like to use the devfile registry with embedded samples inside for the airgap mode, we should first have built that with the offine-registry mode.
So we should have 2 different registry images and switch from one to the other in the operator if we use airgap ?
Not sure I understand how this will be used in the case of the operator airgap mode.

nickboldt · 2019-10-11T12:07:43Z

I suspect for CRW 2 we will only build & publish the offline/airgap option. Yes, it's a bigger image to download than the online flavour, but it's also more guaranteed to work as all the plugins/projects will be self-contained in the devfile and plugin reg containers.

l0rd · 2019-10-11T13:00:30Z

I agree with @nickboldt. It will be the default downstream (and we should consider to make it the default upstream as well c.f. this comment)

l0rd · 2019-10-11T13:07:16Z

@amisevsk I think we should update the container build section in the README file as well and the "Customizing the devfile and plugin-in registries" section in the doc. But considered that this PR may block other PRs we can merge it and address the doc in a separate one.

davidfestal · 2019-10-11T15:46:10Z

we should consider to make it the default upstream as well

+1

amisevsk · 2019-10-11T17:12:52Z

@l0rd I added some docs to the README.md and opened issue eclipse-che/che#14866 to track the larger documentation task

nickboldt

I like this solution.

However...

For this solution to be viable, all the sample projects to be bundled need to have their license/copyright info correctly configured.

See https://issues.jboss.org/browse/CRW-317?focusedCommentId=13798222&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13798222 for a list of the problems with the sample projects' licenses/copyright info.

amisevsk · 2019-10-11T18:53:55Z

Rebased on master to fix merge conflict.

amisevsk · 2019-10-11T18:54:43Z

For this solution to be viable, all the sample projects to be bundled need to have their license/copyright info correctly configured.

Che issue for this: eclipse-che/che#14790

Enable users to build an offline version of the devfile registry, which contains .zip files for all projects. The offline registry should be launched with environment variable CHE_DEVFILE_HTTPS_ENDPOINT set to the public URL of the devfile registry for it to properly serve project zip files. If the environment variable is not set at startup, the registry will try to resolve its cluster IP and port from Kubernetes-provisioned environment variables. The regular (non-offline) build of the registry now requires docker builds to target 'registry', e.g. docker build [...] --target registry offline builds can instead target 'offline-registry' Signed-off-by: Angel Misevski <amisevsk@redhat.com>

Signed-off-by: Angel Misevski <amisevsk@redhat.com>

build/dockerfiles/rhel.Dockerfile

Signed-off-by: Angel Misevski <amisevsk@redhat.com>

* Enable building offline version of registry Enable users to build an offline version of the devfile registry, which contains .zip files for all projects. The offline registry should be launched with environment variable CHE_DEVFILE_HTTPS_ENDPOINT set to the public URL of the devfile registry for it to properly serve project zip files. If the environment variable is not set at startup, the registry will try to resolve its cluster IP and port from Kubernetes-provisioned environment variables. The regular (non-offline) build of the registry now requires docker builds to target 'registry', e.g. docker build [...] --target registry offline builds can instead target 'offline-registry' Signed-off-by: Angel Misevski <amisevsk@redhat.com>

amisevsk added the enhancement New feature or request label Oct 7, 2019

amisevsk requested a review from davidfestal October 7, 2019 20:08

amisevsk requested review from l0rd, nickboldt, rhopp, skabashnyuk and vparfonov as code owners October 7, 2019 20:08

amisevsk mentioned this pull request Oct 7, 2019

Build an offline version of the Devfile registry image eclipse-che/che#14733

Closed

amisevsk force-pushed the cache-projects branch from 6c1d9c3 to 3a61f90 Compare October 7, 2019 20:36

amisevsk force-pushed the cache-projects branch from 3a61f90 to 684d8c8 Compare October 10, 2019 22:12

amisevsk changed the base branch from override-images to master October 10, 2019 22:12

davidfestal reviewed Oct 11, 2019

View reviewed changes

nickboldt approved these changes Oct 11, 2019

View reviewed changes

l0rd approved these changes Oct 11, 2019

View reviewed changes

amisevsk force-pushed the cache-projects branch from c56b9cd to f8a47b0 Compare October 11, 2019 18:53

amisevsk force-pushed the cache-projects branch from f8a47b0 to a3fc742 Compare October 11, 2019 18:59

remove dockerfile.rhel

38035c2

Signed-off-by: Angel Misevski <amisevsk@redhat.com>

amisevsk force-pushed the cache-projects branch from 4bc0e9e to 38035c2 Compare October 11, 2019 19:21

nickboldt reviewed Oct 11, 2019

View reviewed changes

build/dockerfiles/rhel.Dockerfile Outdated Show resolved Hide resolved

fixup: move brew metadata line

754a807

Signed-off-by: Angel Misevski <amisevsk@redhat.com>

amisevsk merged commit d3ecb92 into eclipse-che:master Oct 11, 2019

nickboldt mentioned this pull request Oct 11, 2019

chmod +x #117

Merged

davidfestal mentioned this pull request Oct 14, 2019

Allow using the Devfile registry in offline mode in the Che Operator eclipse-che/che#14880

Closed

monaka mentioned this pull request Apr 29, 2020

Invalid PR. Please ignore this. #234

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build an offline version of the Devfile registry image #112

Build an offline version of the Devfile registry image #112

amisevsk commented Oct 7, 2019 •

edited by nickboldt

amisevsk commented Oct 7, 2019

amisevsk commented Oct 10, 2019

davidfestal Oct 11, 2019

amisevsk Oct 11, 2019

l0rd Oct 11, 2019

amisevsk Oct 11, 2019

davidfestal commented Oct 11, 2019

nickboldt commented Oct 11, 2019

l0rd commented Oct 11, 2019

l0rd commented Oct 11, 2019

davidfestal commented Oct 11, 2019

amisevsk commented Oct 11, 2019

nickboldt left a comment •

edited

amisevsk commented Oct 11, 2019

amisevsk commented Oct 11, 2019

Build an offline version of the Devfile registry image #112

Build an offline version of the Devfile registry image #112

Conversation

amisevsk commented Oct 7, 2019 • edited by nickboldt

What does this PR do?

What issues does this PR fix or reference?

Additional Info

Caveat

amisevsk commented Oct 7, 2019

amisevsk commented Oct 10, 2019

davidfestal Oct 11, 2019

Choose a reason for hiding this comment

amisevsk Oct 11, 2019

Choose a reason for hiding this comment

l0rd Oct 11, 2019

Choose a reason for hiding this comment

amisevsk Oct 11, 2019

Choose a reason for hiding this comment

davidfestal commented Oct 11, 2019

nickboldt commented Oct 11, 2019

l0rd commented Oct 11, 2019

l0rd commented Oct 11, 2019

davidfestal commented Oct 11, 2019

amisevsk commented Oct 11, 2019

nickboldt left a comment • edited

Choose a reason for hiding this comment

amisevsk commented Oct 11, 2019

amisevsk commented Oct 11, 2019

amisevsk commented Oct 7, 2019 •

edited by nickboldt

nickboldt left a comment •

edited