Lug is an open source package that allows you to move the execution of specific Python functions to different environments on each call. This means that instead of deploying a function permanently to a specific environment (e.g. your local computer or a cloud-based server), you can choose where you want to run the function at the time of execution. This flexibility allows you to optimize your resource consumption and cost for the needs of the specific function.
Lug is particularly useful for computational science, where researchers and scientists often have programs that are challenging to install and run. Lug automatically detects and packages pip-installed dependencies and local modules, and you can even attach a sidecar Docker image for the command-line programs that need their own image – making it simple to debug and scale your code.
Check out the full docs at lug.dev
- 📦 Automatic dependency detection and propagation
- 🐳 Attach function-scoped sidecar Docker images
- ☁️ Execute on your computer, the cloud, or on your own servers
- macOS or Linux
- Docker Engine, if you're using Lug Docker Sidecar functions
Install Docker Engine and the CLI, if you don't have it.
pip install lug
There are two ways to use Lug:
- Creating a Lug Hybrid function that executes on your computer or in the cloud.
- Creating a Lug Docker Sidecar function, which is the same as a Lug Hybrid function but with a function-scoped Docker image.
On this readme, we'll just create a short Lug Hybrid function, but there's more detail in the docs.
Let's create a simple Python function that tells us how many CPUs our system has:
import lug
import multiprocessing
@lug.hybrid(cloud=False)
def num_cpus():
return multiprocessing.cpu_count()
print(num_cpus())
This function will execute locally and print the number of CPUs on the system. To run on the cloud, first generate a
Toolchest API key and set key
to your Toolchest API key and cloud=True
:
import lug
import multiprocessing
@lug.hybrid(cloud=True, key="YOUR_KEY_HERE")
def num_cpus():
return multiprocessing.cpu_count()
print(num_cpus())
This function will execute on the cloud and print the number of CPUs on the cloud-based server.
For more detail, check out the docs at lug.dev
- Run a Python function in a local container
- Maintain Python major.version in function and in container
- Serialize and deserialize Python function and Python dependencies
-
os.system()
,subprocess.run()
, andsubprocess.Popen()
redirect to a user-specified container - Local files passed to as input go to
./input/
in remote Docker container - Remote files written to
./output/
in the container are written to local output Path - Runs locally
- Run in the cloud with Toolchest
- Stream live
stdout
during remote cloud execution -
pip
-based environment propagation -
conda
-based environment propagation (help needed)
Lug is licensed under the Apache License 2.0