API.md

Using shakedown helper methods in your DC/OS tests

Table of contents

• Usage
• Methods
  • General
    • authenticate()
    • dcos_url()
    • dcos_service_url()
    • dcos_state()
    • dcos_version()
    • dcos_acs_token()
    • master_ip()
  • Packaging
    • install_package()
    • install_package_and_wait()
    • uninstall_package()
    • uninstall_package_and_wait()
    • package_installed()
    • get_package_repos()
    • add_package_repo()
    • remove_package_repo()
  • Command execution
    • run_command()
    • run_command_on_master()
    • run_command_on_agent()
    • run_dcos_command()
  • File operations
    • copy_file()
    • copy_file_to_master()
    • copy_file_to_agent()
    • copy_file_from_master()
    • copy_file_from_agent()
  • Services
    • get_service()
    • get_service_framework_id()
    • get_service_task()
    • get_service_tasks()
    • get_service_ips()
    • get_marathon_task()
    • get_marathon_tasks()
    • service_healthy()
    • wait_for_service_endpoint()
    • wait_for_service_endpoint_removal()
  • Spinner
    • wait_for()
    • time_wait()
    • elapse_time()
  • Tasks
    • get_task()
    • get_tasks()
    • get_active_tasks()
    • task_completed()
    • wait_for_task()
    • wait_for_task_property()
    • wait_for_task_property_value()
    • wait_for_dns()
  • ZooKeeper
    • get_zk_node_data()
    • delete_zk_node()
  • Marathon
    • deployment_wait()
    • delete_all_apps()
    • delete_all_apps_wait()
  • Masters
    • partition_master()
    • reconnect_master()
    • disconnected_master()
  • Agents
    • get_agents()
    • get_private_agents()
    • get_public_agents()
    • partition_agent()
    • reconnect_agent()
    • restart_agent()
    • stop_agent()
    • start_agent()
    • delete_agent_log()
    • kill_process_on_host()
    • disconnected_agent()
  • Network
    • restore_iptables()
    • save_iptables()
    • flush_all_rules()
    • allow_all_traffic()
    • iptable_rules()

Usage

from shakedown import *

Methods

authenticate()

Authenticate against an EE DC/OS cluster using a username and password.

parameters

parameter	description	type	default
username	the username used for DC/OS authentication	str
password	the password used for DC/OS authentication	str

example usage

# Authenticate against DC/OS, receive an ACS token
token = authenticate('root', 's3cret')

dcos_url()

The URL to the DC/OS cluster under test.

parameters

None.

example usage

# Print the DC/OS dashboard URL.
dcos_url = dcos_url()
print("Dashboard located at: " + dcos_url)

dcos_service_url()

The URI to a named service.

parameters

parameter	description	type	default
service	the name of the service	str

example usage

# Print the location of the Jenkins service's dashboard
jenkins_url = dcos_service_url('jenkins')
print("Jenkins dashboard located at: " + jenkins_url)

dcos_state()

A JSON hash containing DC/OS state information.

parameters

None.

example usage

# Print state information of DC/OS slaves.
state_json = json.loads(dcos_json_state())
print(state_json['slaves'])

dcos_version()

The DC/OS version number.

parameters

None.

example usage

# Print the DC/OS version.
dcos_version = dcos_version()
print("Cluster is running DC/OS version " + dcos_version)

dcos_acs_token()

The DC/OS ACS token (if authenticated).

parameters

None.

example usage

# Print the DC/OS ACS token.
token = dcos_acs_token()
print("Using token " + token)

master_ip()

The current Mesos master's IP address.

parameters

None.

example usage

# What's our Mesos master's IP?
master_ip = master_ip()
print("Current Mesos master: " + master_ip)

install_package()

Install a package.

parameters

parameter	description	type	default
package_name	the name of the package to install	str
package_version	the version of the package to install	str	latest
service_name	custom service name	str	None
options_file	a file containing options in JSON format	str	None
options_json	a dict containing options in JSON format	dict	None
wait_for_completion	wait for service to become healthy before completing?	bool	False
timeout_sec	how long in seconds to wait before timing out	int	600

example usage

# Install the 'jenkins' package; don't wait the service to register
install_package('jenkins')

install_package_and_wait()

Install a package, and wait for the service to register.

This method uses the same parameters as install_package()

uninstall_package()

Uninstall a package.

parameters

parameter	description	type	default
package_name	the name of the package to install	str
service_name	custom service name	str	None
all_instances	uninstall all instances?	bool	False
wait_for_completion	wait for service to become healthy before completing?	bool	False
timeout_sec	how long in seconds to wait before timing out	int	600

example usage

# Uninstall the 'jenkins' package; don't wait for the service to unregister
uninstall_package('jenkins')

uninstall_package_and_wait()

Uninstall a package, and wait for the service to unregister.

This method uses the same parameters as uninstall_package()

package_installed()

Check whether a specified package is currently installed.

parameters

parameter	description	type	default
package_name	the name of the package to install	str
service_name	custom service name	str	None

example usage

# Is the 'jenkins' package installed?
if package_installed('jenkins'):
    print('Jenkins is installed!')

add_package_repo()

Add a repository to the list of package sources.

parameters

parameter	description	type	default
repo_name	the name of the repository	str
repo_url	the location of the repository	str
index	the repository index order	int	-1

example usage

# Search the Multiverse before any other repositories
add_package_repo('Multiverse', 'https://github.com/mesosphere/multiverse/archive/version-2.x.zip', 0)

remove_package_repo()

Remove a repository from the list of package sources.

parameters

parameter	description	type	default
repo_name	the name of the repository	str

example usage

# No longer search the Multiverse
remove_package_repo('Multiverse')

get_package_repos()

Retrieve a dictionary describing the configured package source repositories.

parameters

None

example usage

# Which repository am I searching through first?
repos = get_package_repos()
print("First searching " + repos['repositories'][0]['name'])

run_command()

Run a command on a remote host via SSH.

parameters

parameter	description	type	default
host	the hostname or IP to run the command on	str
command	the command to run	str
username	the username used for SSH authentication	str	core
key_path	the path to the SSH keyfile used for authentication	str	None

example usage

# I wonder what /etc/motd contains on the Mesos master?
exit_status, output = run_command(master_ip(), 'cat /etc/motd')

run_command_on_master()

Run a command on the Mesos master via SSH.

parameters

parameter	description	type	default
command	the command to run	str
username	the username used for SSH authentication	str	core
key_path	the path to the SSH keyfile used for authentication	str	None

example usage

# What kernel is our Mesos master running?
exit_status, output = run_command_on_master('uname -a')

run_command_on_agent()

Run a command on a Mesos agent via SSH, proxied via the Mesos master.

This method uses the same parameters as run_command()

run_dcos_command()

Run a command using the dcos CLI.

parameters

parameter	description	type	default
command	the command to run	str

example usage

# What's the current version of the Jenkins package?
stdout, stderr, return_code = run_dcos_command('package search jenkins --json')
result_json = json.loads(stdout)
print(result_json['packages'][0]['currentVersion'])

copy_file()

Copy a file via SCP.

parameters

parameter	description	type	default
host	the hostname or IP to copy the file to/from	str
file_path	the local path to the file to be copied	str
remote_path	the remote path to copy the file to	str	.
username	the username used for SSH authentication	str	core
key_path	the path to the SSH keyfile used for authentication	str	None
action	'put' (default) or 'get'	str	put

example usage

# Copy a datafile onto the Mesos master
copy_file(master_ip(), '/var/data/datafile.txt')

copy_file_to_master()

Copy a file to the Mesos master.

parameters

parameter	description	type	default
file_path	the local path to the file to be copied	str
remote_path	the remote path to copy the file to	str	.
username	the username used for SSH authentication	str	core
key_path	the path to the SSH keyfile used for authentication	str	None

example usage

# Copy a datafile onto the Mesos master
copy_file_to_master('/var/data/datafile.txt')

copy_file_to_agent()

Copy a file to a Mesos agent, proxied through the Mesos master.

This method uses the same parameters as copy_file()

copy_file_from_master()

Copy a file from the Mesos master.

parameters

parameter	description	type	default
remote_path	the remote path of the file to copy	str
file_path	the local path to copy the file to	str	.
username	the username used for SSH authentication	str	core
key_path	the path to the SSH keyfile used for authentication	str	None

example usage

# Copy a datafile from the Mesos master
copy_file_from_master('/var/data/datafile.txt')

copy_file_from_agent()

Copy a file from a Mesos agent, proxied through the Mesos master.

parameters

parameter	description	type	default
host	the hostname or IP to copy the file from	str
remote_path	the remote path of the file to copy	str
file_path	the local path to copy the file to	str	.
username	the username used for SSH authentication	str	core
key_path	the path to the SSH keyfile used for authentication	str	None

example usage

# Copy a datafile from an agent running Jenkins
service_ips = get_service_ips('marathon', 'jenkins')
for host in service_ips:
    assert copy_file_from_agent(host, '/home/jenkins/datafile.txt')

get_service()

Retrieve a dictionary describing a named service.

parameters

parameter	description	type	default
service_name	the name of the service	str
inactive	include inactive services?	bool	False
completed	include completed services?	bool	False

example usage

# Tell me about the 'jenkins' service
jenkins = get_service('jenkins')

get_service_framework_id()

Get the framework ID of a named service.

parameters

parameter	description	type	default
service_name	the name of the service	str
inactive	include inactive services?	bool	False
completed	include completed services?	bool	False

example usage

# What is the framework ID for the 'jenkins' service?
jenkins_framework_id = get_framework_id('jenkins')

get_service_task()

Get a dictionary describing a named service task.

parameters

parameter	description	type	default
service_name	the name of the service	str
task_name	the name of the task	str
inactive	include inactive services?	bool	False
completed	include completed services?	bool	False

example usage

# Tell me about marathon's 'jenkins' task
jenkins_tasks = get_service_task('marathon', 'jenkins')

get_service_tasks()

Get a list of task IDs associated with a named service.

parameters

parameter	description	type	default
service_name	the name of the service	str
inactive	include inactive services?	bool	False
completed	include completed services?	bool	False

example usage

# What's marathon doing right now?
service_tasks = get_service_tasks('marathon')

get_marathon_task()

Get a dictionary describing a named Marathon task.

parameters

parameter	description	type	default
task_name	the name of the task	str
inactive	include inactive services?	bool	False
completed	include completed services?	bool	False

example usage

# Tell me about marathon's 'jenkins' task
jenkins_tasks = get_marathon_task('jenkins')

get_marathon_tasks()

Get a list of Marathon tasks.

parameters

parameter	description	type	default
inactive	include inactive services?	bool	False
completed	include completed services?	bool	False

example usage

# What's marathon doing right now?
service_tasks = get_marathon_tasks()

get_service_ips()

Get a set of the IPs associated with a service.

parameters

parameter	description	type	default
service_name	the name of the service	str
task_name	the name of the task to limit results to	str	None
inactive	include inactive services?	bool	False
completed	include completed services?	bool	False

example usage

# Get all IPs associated with the 'chronos' task running in the 'marathon' service
service_ips = get_service_ips('marathon', 'chronos')
print('service_ips: ' + str(service_ips))

service_healthy()

Check whether a specified service is currently healthy.

parameters

parameter	description	type	default
service_name	the name of the service	str

example usage

# Is the 'jenkins' service healthy?
if service_healthy('jenkins'):
    print('Jenkins is healthy!')

wait_for_service_endpoint()

Checks the service url returns HTTP 200 within a timeout if available it returns true on expiration it returns false.

parameters

parameter	description	type	default
service_name	the name of the service	str
timeout_sec	how long in seconds to wait before timing out	int	120

example usage

# will wait
wait_for_service_endpoint("marathon-user")

wait_for_service_endpoint_removal()

Checks the service url returns HTTP 500 within a timeout if available it returns true on expiration it returns time to remove.

parameters

parameter	description	type	default
service_name	the name of the service	str
timeout_sec	how long in seconds to wait before timing out	int	120

example usage

# will wait
wait_for_service_endpoint_removal("marathon-user")

wait_for()

Waits for a function to return true or times out.

parameters

parameter	description	type	default
predicate	the predicate function	fn
timeout_seconds	how long in seconds to wait before timing out	int	120
sleep_seconds	time to sleep between multiple calls to predicate	int	1
ignore_exceptions	ignore exceptions thrown by predicate	bool	True
inverse_predicate	if True look for False from predicate	bool	False

example usage

# simple predicate
def deployment_predicate(client=None):
  ...

wait_for(deployment_predicate, timeout)

# predicate with a parameter
def service_available_predicate(service_name):
  ...

wait_for(lambda: service_available_predicate(service_name), timeout_seconds=timeout_sec)

time_wait()

Waits for a function to return true or times out. Returns the elapsed time of wait.

parameters

parameter	description	type	default
predicate	the predicate function	fn
timeout_seconds	how long in seconds to wait before timing out	int	120
sleep_seconds	time to sleep between multiple calls to predicate	int	1
ignore_exceptions	ignore exceptions thrown by predicate	bool	True
inverse_predicate	if True look for False from predicate	bool	False

example usage

# simple predicate
def deployment_predicate(client=None):
  ...

time_wait(deployment_predicate, timeout)

# predicate with a parameter
def service_available_predicate(service_name):
  ...

time_wait(lambda: service_available_predicate(service_name), timeout_seconds=timeout_sec)

elapse_time()

returns the time difference with a given precision.

parameters

parameter	description	type	default
start	the start time	time
end	end time, if not provided current time is used	time	None
precision	the number decimal places to maintain	int	3

example usage

# will wait
elapse_time("marathon-user")

get_task()

Get information about a task.

This method uses the same parameters as get_tasks()

get_tasks()

Get a list of tasks, optionally filtered by task ID.

parameters

parameter	description	type	default
task_id	task ID	str
completed	include completed tasks?	True

example usage

# What tasks have been run?
tasks = get_tasks()
for task in tasks:
    print("{} has state {}".format(task['id'], task['state']))

get_active_tasks()

Get a list of active tasks, optionally filtered by task name.

parameters

parameter	description	type	default
task_id	task ID	str
completed	include completed tasks?	False

example usage

# What tasks are running?
tasks = get_active_tasks()
for task in tasks:
    print("{} has state {}".format(task['id'], task['state']))

task_completed()

Check whether a task has completed.

parameters

parameter	description	type	default
task_id	task ID	str

example usage

# Wait for task 'driver-20160517222552-0072' to complete
while not task_completed('driver-20160517222552-0072'):
    print('Task not complete; sleeping...')
    time.sleep(5)

wait_for_task()

Wait for a task to be reported running by Mesos. Returns the elapsed time of wait.

parameters

parameter	description	type	default
service	framework service name	str
task	task name	str
timeout_sec	timeout	int	120

example usage

wait_for_task('marathon', 'marathon-user')

wait_for_task_property()

Wait for a task to be report having a specific property. Returns the elapsed time of wait.

parameters

parameter	description	type	default
service	framework service name	str
task	task name	str
prop	property name	str
timeout_sec	timeout	int	120

example usage

wait_for_task_property('marathon', 'chronos', 'resources')

wait_for_task_property_value()

Wait for a task to be reported having a property with a specific value. Returns the elapsed time of wait.

parameters

parameter	description	type	default
service	framework service name	str
task	task name	str
prop	property name	str
value	value of property	str
timeout_sec	timeout	int	120

example usage

wait_for_task_property_value('marathon', 'marathon-user', 'state', 'TASK_RUNNING')

wait_for_dns()

Wait for a task dns. Returns the elapsed time of wait.

parameters

parameter	description	type	default
name	dns name	str
timeout_sec	timeout	int	120

example usage

wait_for_dns('marathon-user.marathon.mesos')

delete_zk_node()

Delete a named ZooKeeper node.

parameters

parameter	description	type	default
node_name	the name of the node	str

example usage

# Delete a 'universe/marathon-user' ZooKeeper node
delete_zk_node('universe/marathon-user')

get_zk_node_data()

Get data for a Zookeeper node.

parameters

parameter	description	type	default
node_name	the name of the node	str

example usage

# Get data for a 'universe/marathon-user' ZooKeeper node
get_zk_node_data('universe/marathon-user')

deployment_wait()

Waits for Marathon Deployment to complete or times out.

parameters

parameter	description	type	default
timeout	max time to wait for deployment	int	120

example usage

# assuming a client.add_app() or similar
deployment_wait()

delete_all_apps()

Deletes all apps running on Marathon.

parameters

None.

example usage

delete_all_apps()

delete_all_apps_wait()

Deletes all apps running on Marathon and waits for deployment to finish.

parameters

None.

example usage

delete_all_apps_wait()

partition_master()

Separates the master from the cluster by disabling inbound and/or outbound traffic.

parameters

parameter	description	type	default
incoming	disable incoming traffic?	bool	True
outgoing	disable outgoing traffic?	bool	True

example usage

# Disable incoming traffic ONLY to the DC/OS master.
partition_master(True, False)

reconnect_master()

Reconnect a previously partitioned master to the network

parameters

None.

example usage

# Reconnect the master.
reconnect_master()

get_agents()

Retrieve a list of all agent node IP addresses.

parameters

None

example usage

# What do I look like in IP space?
nodes = get_agents()
print("Node IP addresses: " + nodes)

get_private_agents()

Retrieve a list of all private agent node IP addresses.

parameters

None

example usage

# What do I look like in IP space?
private_nodes = get_private_agents()
print("Private IP addresses: " + private_nodes)

get_public_agents()

Retrieve a list of all public agent node IP addresses.

parameters

None

example usage

# What do I look like in IP space?
public_nodes = get_public_agents()
print("Public IP addresses: " + public_nodes)

partition_agent()

Separates the agent from the cluster by adjusting IPTables with the following:

sudo iptables -F INPUT
sudo iptables -I INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -I INPUT -p icmp -j ACCEPT
sudo iptables -I OUTPUT -p tcp --sport 5051  -j REJECT
sudo iptables -A INPUT -j REJECT

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# Partition all the public nodes
public_nodes = get_public_agents()
for public_node in public_nodes:
    partition_agent(public_node)

reconnect_agent()

Reconnects a previously partitioned agent by reversing the IPTable changes.

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# Reconnect the public agents
for public_node in public_nodes:
    reconnect_agent(public_node)

restart_agent()

Restarts an agent process at the host.

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# Reconnect the public agents
for public_node in public_nodes:
    restart_agent(public_node)

stop_agent()

Stops an agent process at the host.

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# Reconnect the public agents
for public_node in public_nodes:
    stop_agent(public_node)

start_agent()

Start an agent process at the host.

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# Reconnect the public agents
for public_node in public_nodes:
    start_agent(public_node)

delete_agent_log()

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# Delete agent logs on the public agents
for public_node in public_nodes:
    delete_agent_log(public_node)

kill_process_on_host()

Kill the process(es) matching pattern at ip. This will potentially kill infrastructure processes.

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str
pattern	A regular expression matching the name of the process to
kill	str

example usage

# kill java on the public agents
for public_node in public_nodes:
    kill_process_on_host(public_node, "java")

disconnected_agent()

Managed context which will disconnect an agent for the duration of the context then restore the agent

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# disconnects agent
with disconnected_agent(host):
        service_delay()

# agent is reconnected
wait_for_service_url(PACKAGE_APP_ID)

disconnected_master()

Managed context which will disconnect the master for the duration of the context then restore the master

parameters

None

example usage

# disconnects agent
with disconnected_master(host):
        service_delay()

# master is reconnected
wait_for_service_url(PACKAGE_APP_ID)

iptable_rules()

Managed context which will save the firewall rules then restore them at the end of the context for the host. It calls save_iptables before the context and restore_iptables and the end of the context.

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# disconnects agent
with iptable_rules(shakedown.master_ip()):
    block_port(host, port)
    time.sleep(7)

# firewalls restored
wait_for_service_url(PACKAGE_APP_ID)

restore_iptables()

Reverses and restores saved iptable rules. It works with save_iptables.

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# disconnects agent
restore_iptables(host)

save_iptables()

Saves the current iptables to a file on the host.

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# disconnects agent
save_iptables(host)

flush_all_rules()

Flushes the iptables rules for the host. sudo iptables -F INPUT. Consider using save_iptables prior to use.

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# disconnects agent
flush_all_rules(host)

allow_all_traffic()

Removes iptable rules allow full access. Consider using save_iptables prior to using. sudo iptables --policy INPUT ACCEPT && sudo iptables --policy OUTPUT ACCEPT && sudo iptables --policy FORWARD ACCEPT'

parameters

parameter	description	type	default
hostname	the hostname or IP of the node	str

example usage

# disconnects agent
allow_all_traffic(host)