Skip to content
This repository has been archived by the owner on Apr 24, 2023. It is now read-only.

Latest commit

 

History

History
253 lines (187 loc) · 9.54 KB

README.md

File metadata and controls

253 lines (187 loc) · 9.54 KB
Raw

Cook CLI

Installation

To install the Cook CLI, clone this repo and from this folder run:

python3 setup.py install

This will install the cs command on your system.

Configuration

The Cook CLI client is designed with "end user ergonomics" in mind. Custom defaults may be provided for all commands. Multiple clusters are supported via configuration.

In order to use the Cook CLI, you’ll need a configuration file. cs uses a default .cs.json file located in either ${DIR}/.cs.json, ${DIR}/../.cs.json, or ${DIR}/../config/.cs.json, where ${DIR} is the location of the cs executable. This allows you to roll out configuration changes (such as adding a new cluster) without updating the configuration files of all end users. If a default .cs.json file is not found, an empty configuration is used.

Users can also add and overwrite any properties in the default .cs.json file using a local .cs.json file. cs first looks for this .cs.json file in your current directory, and then for this .cs.json file in your home directory. The path to this file may also be provided manually via the command line with the --config option.

There is a sample .cs.json file included in this directory, which looks something like this:

{
  "defaults":{
    "submit": {
      "mem": 128,
      "cpus": 1
    }
  },
  "clusters": [
    {
      "name": "dev0",
      "url": "http://127.0.0.1:12321/"
    },
    {
      "name": "dev1",
      "url": "http://127.0.0.1:12322/"
    }
  ]
}

The defaults map contains default values that are passed each time a command is run. For instance, the submit map above is used to set the defaults for mem and cpus when calling the submit command. These can always be overridden directly from the command line. For most commands, all available clusters are tried in order. Each entry in the clusters array conforms to a cluster specification ("spec"). A cluster spec requires a name and a url pointing to a Cook Scheduler.

Commands

The fastest way to learn more about cs is with the -h (or --help) option.

All global options (--cluster, --config, etc.) can be provided when using subcommands.

submit

You can submit one or more jobs with submit. A single command can be provided as a positional argument, or multiple commands can be provided, via stdin. Jobs can also be passed as "raw" JSON data by setting the --raw flag. The submit command returns UUIDs, one per line, for each created job.

wait

You can wait for jobs, job instances, and job groups to complete with wait. UUIDs are passed as positional arguments.

show

You can query jobs, instances, and groups with show. UUIDs are passed as positional arguments. This command shows information about the queried jobs, instances, and groups. If you use --json, the command returns a JSON representation instead of tables.

jobs

The jobs command allows you to list all jobs:

  • in a particular state (e.g. running / waiting),
  • for a particular user,
  • submitted within the last X hours,
  • with a particular name pattern (e.g. "spark*")

As with show, jobs will list jobs across all configured clusters.

ssh

The ssh command accepts either a job or instance uuid and executes ssh to the corresponding Mesos agent. Assuming the ssh connection is successful, it will also cd to the appropriate sandbox directory. When given a job instance uuid, it will choose the most recently started instance's agent to ssh to.

ls

The ls command accepts a job or instance uuid, along with an optional path, and lists sandbox directory contents. The path is relative to the sandbox directory on the Mesos agent where the instance runs. When given a job instance uuid, it will choose the most recently started instance's agent to list files.

tail

The tail command accepts a job or instance uuid and a path, and outputs the last part of the file with that path. The path is relative to the sandbox directory on the Mesos agent where the instance runs. When given a job instance uuid, it will choose the most recently started instance's agent to tail files. tail also supports "following" a file, meaning that it will output appended data as the file grows.

kill

You can kill jobs, instances, and groups with kill. UUIDs are passed as positional arguments.

config

You can query and set CLI configuration options with this command. The name is actually the sections and the key, separated by dots.

cat

The cat command accepts a job or instance uuid and a path, and outputs the contents of the file with that path. As with tail, the path is relative to the sandbox directory on the Mesos agent where the instance runs. When given a job instance uuid, it will choose the most recently started instance's agent to tail files.

usage

The usage command allows you to list cluster share and usage for a particular user. In addition to totals, it displays a breakdown of usage by application and job group. As with show, usage will list usage across all configured clusters.

Plugins

You can also specify custom functionality for some subcommands with your own plugin functions. You will need to create a new Python script to use as your custom entrypoint, invoking the provided cook.__main__.main method with your customized plugins dictionary:

custom_plugins = {
    "plugin-a-name": plugin_a_function,
    "plugin-b-name": plugin_b_function,
    "plugin-c-name": plugin_c_function,
}
cook.__main__.main(plugins=custom_plugins)

See test_ls_with_plugin in the CLI integration tests for a concrete example.

Plugins are available for the following subcommands:

ls

The ls command has the plugin name retrieve-job-instance-files. The plugin function must take in an instance, sandbox directory, and path, and must return an array of dicts containing the file metadata fields path, mode, nlink, size, uid, gid, and mtime.

cat

The cat command has the plugin name download-job-instance-file. The plugin function must take in an instance, sandbox directory, and path, and must return an iterator that takes in a chunk size, which generates the file contents.

tail

The tail command has the plugin name read-job-instance-file. The plugin function must take in an instance, sandbox directory, path, offset, and length, and must return a dict containing two fields: data and offset.

  • When the offset param is None, the returned data is empty and offset is the file size.
  • When the offset param is not None, the returned data is the file contents starting at offset offset with length length, and the returned offset is the offset param.

See the integration tests for simple examples of the subcommand plugins.

Plugin based subcommands

In addition to plugins for the core subcommands, we can add new subcommands through the plugin module.

Simply extend SubCommandPlugin and implement register, run and name.

See the integration tests for a concrete example.

Examples

Simple job creation:

$ cs submit echo 1
Attempting to submit on dev0 cluster...
Job submission succeeded on dev0. Your job UUID is:
9ee74508-8b07-4b9d-88b4-9af33f8f77a0

Create multiple jobs, each with their own command:

$ printf 'echo 1\necho 2' | cs submit
Enter the commands, one per line (press Ctrl+D on a blank line to submit)
Attempting to submit on dev0 cluster...
Job submission succeeded on dev0. Your job UUIDs are:
6c01233c-3cf6-4994-a48a-9678b20adc35
eb9ba2ef-90dc-4f6d-a5f9-bb1b6cb4ebd9

Create a job from a raw JSON spec:

$ cs submit --raw
Enter the raw job(s) JSON (press Ctrl+D on a blank line to submit)
{"command": "echo 1"}
Attempting to submit on dev0 cluster...
Job submission succeeded on dev0. Your job UUID is:
cec35b58-21e4-4f67-b510-3562d660dd9e

Wait for a job to complete:

$ cs wait 543ed1b9-db42-445f-a731-d6761b66e4a8

Submit a job with a custom name, and query it:

$ cs submit --name ls-job ls
Attempting to submit on dev0 cluster...
Job submission succeeded on dev0. Your job UUID is:
cecf5fe6-8e92-403f-a3ff-e658fb7aeb8d

$ cs show cecf5fe6-8e92-403f-a3ff-e658fb7aeb8d

=== Job: cecf5fe6-8e92-403f-a3ff-e658fb7aeb8d (ls-job) ===

Cluster   dev0                 Attempts    1 / 1
Memory    128 MB               Job Status  Success
CPUs      1.0                  Submitted   just now
User      root                 
Priority  50                   

Command:
ls

Job Instance                          Run Time                      Host        Instance Status
3c625387-b9a6-40db-a971-f37a7eb728e6  0 seconds (started just now)  172.17.0.7  Success

List all jobs submitted by root, in any state, within the last (one) hour:

$ cs jobs --user root --state all --lookback 1
Cluster    UUID                                  Name                                  Memory      CPUs    Priority  Attempts    Submitted       Command    Job Status
dev0       df1ecf44-e42e-4aa3-8cc3-40a289b98666  ff9e3613-97c5-4a33-b4bd-5194fae9c29e  128 MB         1          50  1 / 1       44 minutes ago  exit 1     Failed
dev0       ec62e5c3-a2e0-454b-ada6-e31ac35ff79b  ff9e3613-97c5-4a33-b4bd-5194fae9c29e  128 MB         1          50  1 / 1       44 minutes ago  ls         Success
dev0       9bd67f93-824a-4f82-a7b1-599b3cc5a8c3  ff9e3613-97c5-4a33-b4bd-5194fae9c29e  128 MB         1          50  1 / 1       44 minutes ago  sleep 60   Success
dev0       07420e7b-915a-468f-b53b-be67debcc915  ff9e3613-97c5-4a33-b4bd-5194fae9c29e  128 MB         1          50  0 / 1       44 minutes ago  ls         Waiting