Basics of Virtual Environments

Why virtual environments?

Virtual Environments are fun! They are great! They are my best friends.

Why? They ...

• ... prevent dependencies of one project from impacting another project;
• ... let people know what code bases and versions they need to run your stuff without hassle;
• ... make it easier to conveniently package up all of your dependencies for when other people need to run your code. :)

You’ll want at least one distinct virtual environment for each project. You might have one for each major result milestone.

The rigorous use of virtual environments dramatically improves the reproducibility of results and the speed of obtaining them.

Poetry

Our preferred method!

Quick How-To

• From within the project directory, starts the shell (like source venv/bin/activate if you used a venv)

poetry shell

• Add dependency from pypi listing

poetry add packagename

• To add something via the git repo

poetry add git+https://github.com/whatever.git

• If the project is private

poetry config http-basic.DeepDance askdjf [your un] jksajdf [your private key]

• Publish to pypi, if your publishing is set up.

poetry publish -0.1.0

References

• Official Docs with install instructions: https://python-poetry.org/
• Which Python Package Manager Should You Use? ←a good quick review, also talks a bit about pyenv
• Poetry: "dependency management and packaging made easy" ← this guy gives a fun intro talk to poetry
• "The Packaging Lifecycle with Poetry" - Clinton Roy (Kiwi Pycon X) ← more in depth, this guy is also pretty funny

How does poetry work?

• Libraries are packaged as wheels, installed via pip
• Applications are packaged via containers
• Has a bunch of different ways to specify packages: In a path, public pypi, private, pypi, github repo, url, name + constraint (pip)

Potential Downsides

• No environment logging - You need to remember what repos are set up with poetry and where they are.
• A little too easy to communicate with private repos, doesn’t scrub log in. Sessions end really fast, so it’s not an issue in most cases, but it is something to be aware of.

Conda

• conda is a package management system developed by a company called Anaconda, which is often in use on clusters but can also be used on a personal machine
• you can install either the whole enchilada, called anaconda, or the, uh, miniature taco, called miniconda (which includes pip for sure, plus also I think numpy, scipy, and a few other essentials); they are compared here, though at a debatably useful level of detail

Commands

• create a virtual environment

conda create -n {list of any packages you want to include from the get go}

• Add a package to your virtual env

conda install

• Start a virtual env

conda activate

• Leave a virtual env

conda deactivate

Upsides

• Very easy to start and operate
• Ships with Anaconda, the most common python distribution

Downsides

• it’s managed by a corporation (albeit one that operates on a freemium model, so you won’t have to pay to use it on your laptop)
• this is partially mitigated by using the conda-forge channel for installation like so conda install -c conda-forge
• conda-forge is open source (not administered by Anaconda)
• even miniconda is fairly “heavy” and the dependency solver takes O(10s) to run, so it’s not lightning speed

Mamba

• Have you ever thought about installing conda and felt “wow, this just seems too heavy for me, the dependency solver sounds like it takes foreeeeever to run, and I really value open-source projects!”? Well mamba might be the solution for you!
• mamba is a drop-in replacement for conda: you can start with mamba install <package>, for example, then create a venv with mamba create -n <new name> jax numpy matplotlib, and stay up-to-date with mamba update --all
• there is an even lighter version called micromamba. It’s self-contained. The dependency solver is lightning-fast. It’s, uh, “currently experimental”, so maybe don’t rely on it?
• Mamba and micromamba docs are found here