Skip to content

4. HowTo's

Jump to bottom
Wessel van der Linden edited this page Dec 25, 2024 · 3 revisions

Access Services

Access Kubernetes Services

The dev container runs in the kubernetes host network. This means that it can access any services hosted in kubernetes using fqdn's registered in kubernetes dns. Every service can be accessed with this naming convention:

(servicename).(namespace).svc.cluster.local

The servicename is the name of Service, examples are Lighter or JupyterHub. The namespace is the chosen namespace to deploy the kubernetes resource. Example is the spark namespace. A good example would be: jupyterhub.spark.svc.cluster.local. When executing the following command:

curl http://jupyterhub.spark.svc.cluster.local:8888/jupyterhub

will return the jupyter service.

Use Jupyter Notebooks on JupyterHub via Desktop

JupyterHub is exposed using a kubernetes ingress so you can access it directly using a weblink. JupyterHub can be accessed here: http://demo.localdev.me/jupyterlab

Monitor Lighter via Desktop

Lighter is exposed using a kubernetes ingress so you can access it directly using a weblink. Lighter UI can be accessed here: http://demo.localdev.me/ligher

Working Examples

Python/Pyspark Sessions

A example of a local pyspark session can be found here: example

Pro's:

  • Runs out of the box
  • Real local development with a Spark Cluster running on demand
  • can be integrated with Python Libraries and Test Frameworks

Jupyter Python3 Kernel with Spark

A example of a jupyter python3 with Pyspark can be found here: example

Requirements:

  • Lighter running in Kubernetes (part of setup steps)
  • JupyterLab running in Kubernetes (part of setup steps)
  • Sparkmagic

Pro's

  • Interactive programming for prototyping and data exploration
  • Faster then Pyspark Kernel

Jupyter Pyspark Kernel

A example of a jupyter pyspark with Pyspark can be found here: example

Requirements:

  • Lighter running in Kubernetes (part of setup steps)
  • JupyterLab running in Kubernetes (part of setup steps)
  • Sparkmagic

Pro's

  • Interactive programming for prototyping and data exploration
  • Easy of use

Con's

  • Slower kernel
  • Error Prone

Clone this wiki locally