Skip to content
This repository has been archived by the owner on Oct 13, 2023. It is now read-only.
/ hydra Public archive

A cloud-agnostic ML Platform that will enable Data Scientists to run multiple experiments, perform hyper parameter optimization, evaluate results and serve models (batch/realtime) while still maintaining a uniform development UX across cloud environments

License

Notifications You must be signed in to change notification settings

georgian-io-archive/hydra

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

hydra

A cloud-agnostic Machine Learning Platform that will enable Data Scientists to run multiple experiments, perform hyper parameter optimization, evaluate results and serve models (batch/realtime) while still maintaining a uniform development UX across cloud environments

Installation

To install Hydra using PyPI, run the following command

$ pip install hydra-ml

To install Hydra using GitHub source, first clone Hydra using git :

$ git clone https://github.com/georgianpartners/hydra

Then in the hydra repository that you cloned, run

$ python setup.py install

Check the current hydra version by running

$ hydra --version

Documentation

Prerequisites

  1. Github Token generation
    $ export GITHUB_TOKEN=<Fill your github token here>
    
  2. Setting up your Cloud's CLI tool locally

Getting started


hydra

Entrypoint for Hydra CLI

hydra [flags]

Examples
$ hydra --version
$ hydra --help
Options
  --version  Show hydra version
  --help     Show usage guide

hydra run

Submit a job to the selected cloud platform. You need to run this from inside a git hosted repository that contains your model code and a conda yaml file environment.yml . The command takes a number of options to tailor your job. These options can also be provided via a yaml file

hydra run [flags]

Examples
$ hydra run -m catboost_model.py --cloud gcp --cpu_count 8 --memory_size 20
$ hydra run -m catboost_model.py --cloud gcp --cpu_count 8 --memory_size 20 --options '{"iterations": 100, "depth": 20}'
$ hydra run -y catboost_model_configs.yaml

catboost_model_configs.yaml looks like this :

run:
  model_path: 'catboost_model.py'
  cloud: "gcp"
  cpu_count: 8
  memory_size: 16
  gpu_count: 1
  gpu_type: 'NVIDIA_TESLA_P4'
  region: 'us-west2'
  image_tag: 'batch'
  options:
    - project_name: "hydra-gcp-test-291317-aiplatform"
      bucket_name: "hydra-gcp-test-291317-aiplatform"
      blob_path: "hmnist/hmnist_64_64_L.csv"
      batch_size: 1
      epoch: 5
    - project_name: "hydra-gcp-test-291317-aiplatform"
      bucket_name: "hydra-gcp-test-291317-aiplatform"
      blob_path: "hmnist/hmnist_64_64_L.csv"
      batch_size: [1, 2, 3]
      epoch: [1, 2, 3]
Options
  -y, --yaml_path TEXT            Path to YAML file that contains preset options
  -m, --model_path TEXT           Path to file containing model code
  --cloud [fast_local|local|aws|gcp|azure]
  --github_token TEXT
  --cpu_count INTEGER RANGE       Number of CPU cores required
  --memory_size INTEGER RANGE     GB of RAM required
  --gpu_count INTEGER RANGE       Number of accelerator GPUs
  --gpu_type TEXT                 Accelerator GPU type
  --region TEXT                   Region of cloud server location
  -t, --image_tag TEXT            Docker image tag name
  -u, --image_url TEXT            Url to the docker image on cloud
  -o, --options TEXT              Environmental variables for the script

Options inherited from parent commands
  --help   Show usage guide for command

Infrastructure as Code


To get an overview of the infrastructure as code, please review the associated Wiki page.

About

A cloud-agnostic ML Platform that will enable Data Scientists to run multiple experiments, perform hyper parameter optimization, evaluate results and serve models (batch/realtime) while still maintaining a uniform development UX across cloud environments

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published