enable passing server options for notebook server launch #68

rokroskar · 2018-10-16T07:06:34Z

This PR allows the notebooks service setup to include default server options -- and to override them upon server launch. The options are set in values.yaml and written into a file in the container via a configMap.

The options can be queried by a UI client using GET <service_prefix>/<namespace>/<project>/<commit-sha>/server_options -- the returned JSON can be used to render a template.

The client should return only the option and the value, e.g. if the JSON server from server_options is

{
  "serverOptions":
  { 
    "resources":
      { 
        "cpu_limit":
         {
          "default": 1,
          "type": int,
          "range": [1,4]
          }
      }
  }
}

The client should use

{
  "serverOptions":
    {
      "resources":
      {
         "cpu_limit": "1"
      }
    }
}

as the json data in the POST request to spawn the server.

closes #66
closes #65
addresses SwissDataScienceCenter/renku#443

leafty · 2018-10-16T07:28:18Z

helm-chart/renku-notebooks/Chart.yaml

@@ -2,4 +2,4 @@ apiVersion: v1
 appVersion: '1.0'
 description: A Helm chart for the Renku Notebooks service
 name: renku-notebooks
-version: 0.2.0
+version: 0.0-unclean-191b684


Yes I know, I'll clean all this up when I'm done running it on my minikube :)

leafty

looks good

leafty · 2018-10-16T07:33:23Z

helm-chart/renku-notebooks/templates/deployment.yaml

          ports:
            - name: http
              containerPort: 8000
              protocol: TCP
+          volumeMounts:
+            - name: server-options
+              mountPath: /etc/renku-notebooks/server_options.json


/etc/renku-notebooks
(https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#add-configmap-data-to-a-volume)

leafty · 2018-10-16T07:34:51Z

jupyterhub/jupyterhub-k8s.Dockerfile

@@ -1,4 +1,4 @@
-FROM jupyterhub/k8s-hub:0.7.0
+FROM jupyterhub/k8s-hub:ae44b17


jupyterhub/spawners.py

src/notebooks_service.py

leafty · 2018-10-16T07:39:20Z

helm-chart/renku-notebooks/values.yaml

 ## Configuration for the jupyterhub service
 jupyterhub:
  rbac:
    enabled: true
  hub:
    image:
      name: renku/jupyterhub-k8s
-      tag: '0.2.0'
+      tag: '0.0-unclean-191b684'


leafty · 2018-10-16T07:39:27Z

helm-chart/renku-notebooks/values.yaml

@@ -140,14 +165,14 @@ jupyterhub:
  singleuser:
    image:
      name: renku/singleuser
-      tag: '0.2.0'
+      tag: '0.0-unclean-191b684'


leafty · 2018-10-16T07:39:35Z

helm-chart/renku-notebooks/values.yaml

    storage:
      type: none
    defaultUrl: /lab
  singleuser-r:
    image:
      name: renku/singleuser-r
-      tag: 'latest'
+      tag: '0.0-unclean-191b684'


helm-chart/renku-notebooks/templates/deployment.yaml

ableuler

Looking good so far. Can you please make sure that the default is always part of the available options for the enum case? This is currently not the case for the memory requirements.

leafty · 2018-10-17T13:32:47Z

helm-chart/renku-notebooks/values.yaml

+    mem_limit:
+      displayName: Memory
+      type: enum
+      default: 1G


leafty

NB: From the code of kubespawner, 0 can be used for no limit

leafty · 2018-10-17T13:57:00Z

helm-chart/renku-notebooks/values.yaml

+      displayName: Number of CPUs
+      type: float
+      default: 1.0
+      range: [0.1, 4.0]


only use discrete options

why? CPUs requests are fractional

a. are you going to use it?
b. with options: [0, 1, 2, 4, 8] you can have 0 = no limits
c. you can include 0.5 anyway

this is a limit, not a request -- so I think actually integer values are probably fine, so I propose we do range: [1,X] (I just pushed this change in de26aaa)

0 is valid :p

yes but we don't want people to specify 0. In any case, from what I've read the cpu limits are not strictly enforced, unlike the memory limits.

helm-chart/renku-notebooks/values.yaml

rokroskar · 2018-10-17T14:00:59Z

btw, I don't pass 0 as a gpu number -- if it is 0 then we don't ask for a gpu at all.

leafty · 2018-10-17T14:59:57Z

jupyterhub/spawners.py

+
+            gpu = server_options['resources'].get('gpu', {})
+            if gpu:
+                self.extra_resource_guarantees = {


extra_resource_limits

compatibility with SwissDataScienceCenter/renku-notebooks#68

this has been fixed

compatibility with SwissDataScienceCenter/renku-notebooks#68

rokroskar requested review from ableuler and leafty and removed request for ableuler October 16, 2018 07:06

rokroskar changed the title ~~enable passing server options for notebook server launch~~ [WIP] enable passing server options for notebook server launch Oct 16, 2018

rokroskar mentioned this pull request Oct 16, 2018

support interactive environment setup SwissDataScienceCenter/renku#443

Closed

5 tasks

leafty reviewed Oct 16, 2018

View reviewed changes

leafty requested changes Oct 16, 2018

View reviewed changes

leafty reviewed Oct 16, 2018

View reviewed changes

leafty requested changes Oct 16, 2018

View reviewed changes

helm-chart/renku-notebooks/templates/deployment.yaml Show resolved Hide resolved

rokroskar force-pushed the fix-66 branch from f6e08a4 to 096363c Compare October 16, 2018 15:22

ableuler previously requested changes Oct 17, 2018

View reviewed changes

leafty requested changes Oct 17, 2018

View reviewed changes

helm-chart/renku-notebooks/values.yaml Outdated

mem_limit:

displayName: Memory

type: enum

default: 1G

Copy link

Member

leafty Oct 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default 2G

rokroskar force-pushed the fix-66 branch from 4e31137 to b117abe Compare October 17, 2018 13:46

leafty requested changes Oct 17, 2018

View reviewed changes

rokroskar force-pushed the fix-66 branch from b117abe to 8efd453 Compare October 17, 2018 14:11

rokroskar added 3 commits October 17, 2018 16:12

update Pipfiles

a758360

add server_options endpoint

71097b7

jh spawner: cleanup init code

4aa564a

rokroskar force-pushed the fix-66 branch from 8efd453 to b58c5df Compare October 17, 2018 14:15

leafty previously approved these changes Oct 17, 2018

View reviewed changes

rokroskar dismissed leafty’s stale review via 698f9ce October 17, 2018 14:50

rokroskar force-pushed the fix-66 branch from 58160c6 to 698f9ce Compare October 17, 2018 14:50

leafty requested changes Oct 17, 2018

View reviewed changes

rokroskar added 3 commits October 17, 2018 17:07

pass server options to spawner and process them

3907ee7

temporarily disable authentication on server_options

f0f1984

add minimal resource requests

d7fe14d

rokroskar and others added 2 commits October 17, 2018 17:07

make cpu option an integer

ddc9f9a

include configmap hash

ba8db87

rokroskar force-pushed the fix-66 branch from 698f9ce to ba8db87 Compare October 17, 2018 15:07

rokroskar changed the title ~~[WIP] enable passing server options for notebook server launch~~ enable passing server options for notebook server launch Oct 17, 2018

change limits to requests

b0c93bb

rokroskar added a commit to rokroskar/renku that referenced this pull request Oct 17, 2018

charts: adapt for notebook server options

9516aab

compatibility with SwissDataScienceCenter/renku-notebooks#68

rokroskar mentioned this pull request Oct 17, 2018

charts: adapt for notebook server options SwissDataScienceCenter/renku#448

Closed

leafty approved these changes Oct 18, 2018

View reviewed changes

rokroskar merged commit b0c93bb into master Oct 18, 2018

rokroskar deleted the fix-66 branch October 18, 2018 07:37

This was referenced Oct 18, 2018

validate server_options #71

Closed

charts: adapt for notebook server options SwissDataScienceCenter/renku#451

Merged

rokroskar added a commit to SwissDataScienceCenter/renku that referenced this pull request Oct 18, 2018

charts: adapt for notebook server options (#451)

57c0569

compatibility with SwissDataScienceCenter/renku-notebooks#68

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable passing server options for notebook server launch #68

enable passing server options for notebook server launch #68

rokroskar commented Oct 16, 2018 •

edited

leafty Oct 16, 2018

rokroskar Oct 16, 2018

leafty left a comment

leafty Oct 16, 2018

leafty Oct 16, 2018

leafty Oct 16, 2018

leafty Oct 16, 2018

leafty Oct 16, 2018

ableuler left a comment •

edited

leafty Oct 17, 2018

leafty left a comment •

edited

leafty Oct 17, 2018

rokroskar Oct 17, 2018

leafty Oct 17, 2018

rokroskar Oct 17, 2018 •

edited

leafty Oct 17, 2018

rokroskar Oct 17, 2018

rokroskar commented Oct 17, 2018

leafty Oct 17, 2018

		@@ -1,4 +1,4 @@
		FROM jupyterhub/k8s-hub:0.7.0
		FROM jupyterhub/k8s-hub:ae44b17

enable passing server options for notebook server launch #68

enable passing server options for notebook server launch #68

Conversation

rokroskar commented Oct 16, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leafty left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ableuler left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leafty left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rokroskar Oct 17, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rokroskar commented Oct 17, 2018

Choose a reason for hiding this comment

rokroskar commented Oct 16, 2018 •

edited

ableuler left a comment •

edited

leafty left a comment •

edited

rokroskar Oct 17, 2018 •

edited