Added an extra verbosity level to paasta_metastatus

[mjksmith]

Uses Krall's mesos slave analysis script to display free CPU and RAM on each host. Called by using paasta metastatus -vv (or --verbose --verbose).

[solarkennedy]

Can you adjust the help output to hint that there can be more?

[solarkennedy]

Can you add this to the itest to ensure we are invoking this new code path on a real box?

[solarkennedy]

Just for record keeping, this PR is waiting for itest + review feedback fixes.

[Rob-Johnson]

I don't really like the logic for 'extra' checks being printed in main(), here's my suggestion for how to implement what you need:

Add a boolean parameter like verbose_checks to get_mesos_status. Decide if this should be set to True or False on the value ofverbose in main.

Add a check named assert_slave_stats to the list of checks performed by get_mesos_state.

def assert_slave_stats(mesos_state):
    header_string = "%12s %36s %9s' % ('Hostname', 'CPU free', 'RAM free')"
    slave_outputs = ['%40s %8.2f %9.2f' % (slave['hostname'], slave['free_resources']['cpus'], slave['free_resources'],slave['mem']) 
                     for slave in get_extra_mesos_slave_data(mesos_state)]
    check_output = [header_string].extend(slave_outputs)
    return ("\n".join(check_output),True)

N.B the second value of the tuple returned by the assert_* functions indicates if this check is 'healthy'. If there is anything that this check shows that is worth paging on (I don't think there is), then we can add logic to decide if we should return True/False accordingly.

modify get_mesos_state to perform this check based on the verbose_checks parameter.

get_mesos_state will end up looking something like:

def get_mesos_status(mesos_state, run_verbose_checks=False):
    """Gathers information about the mesos cluster.
       :return: tuple of a string containing the status and a bool representing if it is ok or not
    """

    cluster_results = run_healthchecks_with_param(mesos_state, [assert_quorum_size, assert_no_duplicate_frameworks])

    metrics = get_mesos_stats()
    check_functions =  [
        assert_cpu_health,
        assert_memory_health,
        assert_tasks_running,
        assert_slave_health
    ]

    if run_verbose_checks:
       check_functions.append(assert_slave_stats)
    metrics_results = run_healthchecks_with_param(metrics, check_functions)

    return cluster_results + metrics_results

By doing this, we move the logic for 'whats verbose' away from main, and the 'verbose' checks become regular healthchecks

I'm happy to pair with you on this if you like, too.

[Rob-Johnson]

why do you need the whole slave dict as the value here? how about:

dict((slave['id'], {}) for slave in mesos_state['slaves'])

that way, in the below logic, you're won't be modifying the slave dict.

[Rob-Johnson]

can you check what happens when flags is empty?

Then paasta_metastatus exits with..

Will flags just be an empty string?

[Rob-Johnson]

fix n ship - great stuff 👍