This guide will help you deploying and customizing your own Renku instance in a cluster.
To deploy Renku in a cluster, you need to have the following prerequisites:
- a Kubernetes cluster
- Helm
- :ref:`a DNS domain <dns>` that resolves to the IP of your cluster load-balancer
- :ref:`TLS certificates (or a cert-manager LetsEncrypt) <certificates>`
- :ref:`NGINX (or other) ingress <nginx>` capable of interpreting standard Kubernetes Ingress objects
- A GitLab instance
.. toctree:: :hidden: prerequisites/dns prerequisites/certificates prerequisites/nginx
You may choose to either deploy GitLab as a part of the Renku deployment or link to an existing GitLab instance. If you are deploying GitLab as a part of Renku, you need to configure data storage and be prepared to manage the CI runners etc. We encourage production deployments to not use the GitLab chart bundled with Renku, but instead either acquire GitLab as a service or deploy it using the official GitLab cloud-native kubernetes chart.
Some parts of these requirements are beyond the scope of this documentation - for example, load-balancer setup is highly cluster-dependent. If you have a configuration that works on a public cloud provider, please consider contributing to these docs! 🙏
If you have your cluster configured with a DNS, a TLS certificate, and an Nginx ingress running, you should be able to do
$ curl https://<hostname> default backend - 404
This response basically means that:
- a DNS server can resolve your hostname to the load-balancer IP
- TLS termination was successful
- the request made it to the ingress (but there are no applications serving the request, which is fine)
The following component versions have been tested in production.
Renku version | Kubernetes | Helm |
---|---|---|
0.6.x | 1.14, 1.15 | 2 |
0.7.x | 1.16, 1.17 | 3 |
0.8.x | 1.19, 1.20 | 3 |
0.12.x | 1.20, 1.21 | 3 |
.. toctree:: :maxdepth: 1 GitLab <configurations/external-gitlab> Keycloak <configurations/external-keycloak>
All of Renku information is stored in three volumes needed by Jena, Postgresql and GitLab (if not external). You can leave k8s to dynamically provision these volumes but we advise to create them beforehand and make regular backups. You can use the following manifest files as examples.
Note
These examples are specific to our use of OpenStack and the details of the storage classes and the volume provider will be different depending on the specifics of your cluster.
This step is very important as it will define the values that helm will use to install your Renku instance.
The full values file in all its glory can be found as a part of the Renku helm chart. We tried to document in-line all the important pieces that someone might want to configure, but there are a lot of options and many you probably won't care about immediately.
To generate a basic values file, you can use a script we have prepared. You need to have Python3 and wget
installed.
$ mkdir renku-values $ cd renku-values $ wget https://raw.githubusercontent.com/SwissDataScienceCenter/renku/master/scripts/generate-values/generate-values.sh $ sh generate-values.sh -o renku-values.yaml
By default, this will create a python virtual environment and run the script; if
you prefer to avoid any installation, you can pass the --docker
flag to the
generate-values.sh
script and it will execute in a docker container.
Note
By default the generate-values script assumes that an external GitLab will be used, i.e. it will
create values that will not deploy GitLab as part of a Renku deployment. If you wish to deploy
GitLab with Renku, add a --gitlab
flag when calling the generate-values.sh
script.
After preparing your values and before deploying Renku please make sure to check the following in the values file:
- the URLs and DNS names are correct
- all the necessary secrets are configured
- the ingress configuration is correct
- add/verify, if necessary, the persistent volume claims
Note
If you created persistent volumes in the step above, make sure to add their references to the appropriate components under storage.existingClaim.
You might want to modify some default values, here are some examples:
- Logged-out users can be given the permission to launch interactive sessions from public notebooks. This feature is turned off by default. For details on how to enable this feature, see the dedicated section below.
- A custom privacy notice and cookie banner can be configured (as of Renku 0.6.8). See the dedicated section below.
- It is possible to dedicate some nodes for running interactive user sessions.
.. toctree:: :maxdepth: 1 Enabling notebooks for anonymous users <anonymous-sessions> Configure privacy notice and cookie banner <privacycookie> Configuring the RenkuLab Homepage <homepage>
Once all the pieces are in place, you can deploy Renku with the following commands:
$ helm repo add renku https://swissdatasciencecenter.github.io/helm-charts/
$ helm upgrade --install renku renku/renku \
--namespace renku \
-f renku-values.yaml \
--timeout 1800s
During deployment you can check the Renku pods being started. If deploying for the first time, GitLab and Keycloak take around 15 minutes to get started. It is normal to see other pods of components that depend on these to be in a CrashLoopBackOff state.
Once all the pods are correctly running or completed, a GitLab Runner can be configured.
.. toctree:: :hidden: prerequisites/gitlabrunner
Check list:
- Helm command completed successfully.
- RenkuLab pods should be all running and jobs completed.
- RenkuLab pods logs should not show errors.
- Perform a quick check:
- go to your RenkuLab instance domain (this will verify the ingress is working properly)
- login with a valid user, you can register a new one (this will verify various authentication steps)
- create a project (this will verify further authentication, templates and GitLab runner)
- add a dataset (this will verify the core backend, LFS objects, and Knowledge Graph components work)
- launch an environment (this will verify the notebooks component and GitLab image registry)
- You should be now able to follow "First steps"
If some Renku pods are not starting or present some errors please check the troubleshooting page.
.. toctree:: :maxdepth: 1 Troubleshooting a RenkuLab deployment <troubleshooting>
If the Helm deployment ran successfully according to the check list above, you might want to run Renku's acceptance tests. You simply need to:
- create a user for testing (with no admin rights) and optionally have an S3 bucket credentials for saving screenshots in case a test fails.
- fill in the credentials (notably email and password) in the test section in the values file and update the deployment with
helm
.$ helm upgrade --install renku renku/renku \ --namespace renku \ --version <renku-version> \ -f renku-values.yaml \ --timeout 1800s
- run the tests. This will create a test pod that needs around 4GB of RAM available. Note that running the complete acceptance test suit takes around 40 minutes to complete.
$ helm test renku \ --namespace renku \ --log-file renku-tests.log \ --timeout 80m
After RenkuLab has been deployed you can make some post deployment configurations.
If your RenkuLab deployment includes GitLab you need to follow some additional steps to configure an admin user on GitLab.
- Turn off automatic redirection to GitLab by setting the value
gitlab.oauth.autoSignIn=false
and redeploy withhelm
. - Log in as the root user using the password from
gitlab.password
. - Modify any users you want to modify (e.g. to make them admin).
- Turn the
autoSignIn
setting back on and redeploy.
If the RenkuLab deployment is relying on an existing GitLab instance, this instance could be configured as an identity provider for RenkuLab's keycloak. This way all of the users from the existing GitLab instance can log into RenkuLab with their existing login information.
- Go to https://<renku-domain>/auth (or to your stand-alone Keycloak dashboard), login to
Admin console
usingadmin
as username and the decoded password stored inkeycloak-password-secret
. - Add an
Identity Provider
of typeOpenID Connect v1.0
. - Set
Alias
to <renku-domain>,Authorization URL
to https://<gitlab-domain>/oauth/authorize,Token URL
to https://<gitlab-domain>/oauth/token andClient ID
andClient Secret
to the values used in step 1. - Click
Save
. The new Identity Provider should appear and any user from the stand-alone GitLab instance should be able to login.
To upgrade RenkuLab to a newer version please check the Upgrading from 0.x.x section of the release notes in case the renku-values.yaml file needs to be modified. Following, you can upgrade Renku via Helm.
$ helm repo update
$ helm upgrade --install renku renku/renku \
--namespace renku \
--version <renku-version> \
-f renku-values.yaml \
--timeout 1800s