Lucyfer is a specialization of the official Jupyter Datascience Notebook Docker image that brings additional tools into the Jupyter environment while maintaining independence from the local system's development tools. This empowers the user to use all Jupyter features without having to install any of the Jupyter packages locally.
tl;dr: Lucyfer is a zero-conf, zero-install Python IDE for buiding high performance computing prototypes. It enables data scientists and software engineers to work off the same code base and use the same tools.
- Git support
- SPARQL kernel for interactive dataset exploration
- Kotlin kernel for state-of-the-art interactive JVM REPL and exploration
jqpudb, linters, devpi,pytest,plotly, and data handling Python toolssshtree(file system visualization)- Vim with NERDTree and other dev tools
Lucyfer also fixes the Conda and jovyan ($NB_USER) permissions in the
Jupyter image and enables pip package installation.
The lucyfer image is a specialization of jupyter/datascience-notebook and
runs on x86 processors. It supports Python, Julia, Kotlin, R, and SPARQL
kernels.
The lucyfer-m image is a specialization of jupyter/scipy-notebook and runs
on ARM64 processors. It only supports Python, Kotlin, and SPARQL kernels.
- Coexists with local file system development tools like IDEs or Python virtual environments
- Run experiments using local code against remote data
- Develop production-ready modules and packages from within the Jupyter/Lucyfer environment as part of the DevOps process
- Develop and test on macOS and Linux from the same machine without having to deploy a full Linux VM (x86, M-processors)
- Access AWS data lakes without making local copies of remote data
- Docker desktop
wgetfor pulling the files off GitHub
Open a terminal and create a working directory anywhere in your file system, or pick any existing directory.
mkdir -p ~/development/mywork &&
cd ~/development/myworkGet the starter files from the Lucyfer project at GitHub and execute the runner:
wget https://raw.githubusercontent.com/pr3d4t0r/lucyfer/master/lucyfer-compose.yaml && \
wget https://raw.githubusercontent.com/pr3d4t0r/lucyfer/master/README-1ST.ipynb && \
wget https://raw.githubusercontent.com/pr3d4t0r/lucyfer/master/README-Kotlin.ipynb && \
wget https://raw.githubusercontent.com/pr3d4t0r/lucyfer/master/README-SPARQL.ipynb && \
wget https://raw.githubusercontent.com/pr3d4t0r/lucyfer/master/lucy && \
chmod +x ./lucy && \
./lucy start
curl -o lucyfer-compose.yaml https://raw.githubusercontent.com/pr3d4t0r/lucyfer/master/lucyfer-compose.yaml && \
curl -o README-1ST.ipynb https://raw.githubusercontent.com/pr3d4t0r/lucyfer/master/README-1ST.ipynb && \
curl -o README-Kotlin.ipynb https://raw.githubusercontent.com/pr3d4t0r/lucyfer/master/README-Kotlin.ipynb && \
curl -o README-SPARQL.ipynb https://raw.githubusercontent.com/pr3d4t0r/lucyfer/master/README-SPARQL.ipynb && \
curl -o lucy https://raw.githubusercontent.com/pr3d4t0r/lucyfer/master/lucy && \
chmod +x ./lucy && \
./lucy start
Docker displays Lucyfer and Kallisto among all the running containers:
Name Command State Ports
------------------------------------------------------------------------------------------
kallisto /bin/sh -c java -Xmx2g -Dc ... Up 0.0.0.0:8809->9999/tcp
lucyfer tini -g -- start-notebook.sh Up (health: starting) 0.0.0.0:8805->8888/tcp
Lucyfer authentication token = 1d156b04af64bded5ed6ca274e90a696cfd6f4be06bfbfc7
lucy startStarts all Lucyfer services/container: lucyfer IDE and kallisto graph DB.
lucy start containerStarts only the specific container. The container name must match the
definitions in the lucyfer-compose.yaml file. Service names correspond to the
container name, with a _service suffix, like:
lucyferis the container name forlucyfer_servicekallistois the container name forkallisto_service
The stop command may also take the container argument. All other commants
operate on the total service containers running when they are executed.
Open this URL in your browser: http://localhost:8805
Enter the Lucyfer authentication token in the field at the bottom of the page, followed by your password so that you can return the Lucyfer lab any time.
Open the README-1ST.ipynb notebook. It has live instructions on how to check
if the database is running and on how to conclude the Lucyfer setup.
There are use cases where the lucyfer-compose.yaml file doesn't fit the
science or engineering team's setup, and they wish to run Lucyfer from a custom
Docker Compose file, for example mornyngstar-compose.yaml. The lucy script
initializes the LUCYFER_COMPOSE internal environment variable with the default
Compose file. To override it, simply define the variable prior to launching the
container to use a different file:
LUCYFER_COMPOSE="mornyngstar-compose.yaml" ./lucy startDifferences between mornyngstar- and lucyfer- files:
- Use of port 8807 instead of 8805
- Docker assigns the container name instead of using the default
lucyfer
Lucyfer and Kallisto expose ports 8805 and 8809, respectively, to the host. Setting a different port is easy: set the new port number using these environment variables:
- LUCYFER_PORT
- KALLISTO_PORT
Example:
KALLISTO_PORT=5555 ./lucy startThis results in:
Name Command State Ports
------------------------------------------------------------------------------------------
kallisto /bin/sh -c java -Xmx2g -Dc ... Up 0.0.0.0:5555->9999/tcp
lucyfer tini -g -- start-notebook.sh Up (health: starting) 0.0.0.0:8805->8888/tcp
Use the password to log on to Lucyfer; no token available
Lucyfer is an ongoing effort, and the team builds new images once a week on average. To update your Lucyfer IDE setup all you have to do is run:
./lucy updateEffects:
- Updates the `README*.ipynb* files
- Updates the
lucyscript - Pulls
pr3d4t0r/lucyfer:latestfrom Docker Hub
The file updates are fetched from the Lucyfer project master branch on GitHub.
The authentication token for accessing the Lucyfer environment can be displayed
usint the token command:
./lucy tokenEffects:
- Displays the current authentication token if a password is not set
- On macOS, the authentication token is copied to the clipboard for easy pasting with Cmd-V
- Does nothing if the Lucyfer password is set
Lucyfer can be configured to open a session in a browser window as soon as the service starts. This convenience feature is enabled by defining:
LUCYFER_OPEN_ON_START="yes"
.
.
./lucy start lucyferThis environment variable is optional, and may only take the value of yes.
The implementation opens http://localhost:$LUCYFER_PORT and works under macOS
and Linux.
From 2010: Odyssey Two by Arthur C. Clarke:
ALL THESE WORLDS ARE YOURS -- EXCEPT EUROPA. ATTEMPT NO LANDING THERE.
HAL-9000 transmitted that warning to Earth after fusing with the Monolith and David Bowman, the Star Child, before Jupiter reached critical mass and exploded to become a star.
People on Earth named the new star Lucifer, the Morning Star.
Jupiter became a star thanks to the internal transformation that the cosmic general purpose tool, the Monolith, enabled.
Linked photo © 1984 Metro-Goldwyin-Mayer
The Dockerfille and Compose files associated with this project enabled an
internal transformation of Jupyter to turn it into an even better tool.
Last, why is the alternate Compose file named mornyngstar? Because that's
another name for Lucyfer.
Lucyfer is released under the BSD-3 license, and is based on the Jupyter Project, also released under the BSD-3 license.
Copyright © the Lucyfer Project Contributors
| Name | GitHub user name |
|---|---|
| Balmos, Scott | sbalmos |
| Ciurana, Eugene | pr3d4t0r |
| Fulgo, Matt | mfulgo |
| Heudecker, Nick | nheudecker |
| Lombardi, Andrew | kinabalu |
| Rook, Vivian | vivian-rook |
| Ottinger, Joe | jottinger |
| Ross, Phillip | phillipross |