README.md

Self shipping project with skein

1. Prepare a virtual environment and install the sample project using skein

$ cd examples/skein-project
$ python3 -m venv venv
$ . venv/bin/activate
$ pip install --upgrade pip setuptools
$ pip install -e .
python

2. Upload current virtual environment to the distributed storage (HDFS in this case)

import cluster_pack
package_path, _ = cluster_pack.upload_env()

3. Call skein config helper to get the config that easily accesses those uploaded packages on the cluster, skein_project.worker is the module we want to call remotely. It has been shipped by cluster-pack

from cluster_pack.skein import skein_config_builder
skein_config = skein_config_builder.build(
    module_name="skein_project.worker",
    package_path=package_path
)

4. Submit a simple skein application

import skein
with skein.Client() as client:
    service = skein.Service(
        resources=skein.model.Resources("1 GiB", 1),
        files=skein_config.files,
        script=skein_config.script
    )
    spec = skein.ApplicationSpec(services={"service": service})
    app_id = client.submit(spec)