Merge coll sysinfo metadata log

[portante]

Merge collect sysinfo and metadata log

Merge pbench-collect-sysinfo and pbench-metadata-log operations and behaviors into the Tool Meister framework.

To make it easier to review, this PR was broken into 4 commits:

• All the changes to the agent/bench-scripts directory without unit tests (14 files)
• All the changes to the agent/util-scripts and lib/pbench/agent hierarchies (23 files)
• UML Sequence Diagram updates (1 file)
• All testing related updates (100+ files)

Once reviewed, we merged them back down to a single commit.

Description of combined changes

The last set of ssh commands issued by the pbench-agent that are not involved in orchestrating the Tool Meisters, outside of those issued by the bench-scripts, are for hostname -s data collection on all the registered remote tool hosts.

The Tool Meisters are now given the responsibility for collecting information about their environment, including the version of the pbench agent, and a more comprehensive collection of host name information, returning that with their startup acknowledgement payload. This change also addresses a previous bug where registered hosts labels were not being used in the on-disk data capture directories as they were before.

This collected data set is now recorded by the Tool Data Sink, after restructuring the action flow so that clients send action playloads to the Tool Data Sink, which then forwards to the Tool Meisters. This gives the Tool Data Sink the ability to collect all the Tool Meister information and write it to disk before acknowledging to the client that startup is complete.

We add an entirely new section in the metadata.log file which is per-host/per-tool, recording the tool options, and the output from the tool install checks (if any). This section supercedes the previous tool options in the per-host sections, though we leave those options there for compatibility for now.

We add graceful handling of bad PUT requests from Tool Meisters to the Tool Data Sink.

With this refactoring, we also collapse all the "sysinfo", "init", and "end" actions into the respective pbench-tool-meister-start and pbench-tool-meister-stop interfaces to simplify the CLI behaviors.

To make this possible, we add a new Client API used by the CLI interfaces pbench-tool-meister-start and pbench-tool-meister-stop.

We have also undertaken a major refactoring of pbench-sysinfo-dump to remove the dependency on base for environment variables. As a result of this work, we drop the stockpile configuration entirely, as it is no longer required.

We have enhanced the Tool Meister tests to use persistent tools, and enhanced the unit test framework to properly clean up the Tool Meister test environment when tests fail.

We have updated the UML sequence diagram describing the Tool Meister operation to add the init/end/sysinfo steps, and the fact that the Tool Data Sink is the gatekeeper of actions, forwarding them on to the Tool Meisters.

Notes

Okay, this is where Python 3 falls down: replacing command line utilities available via bash with Python 3 modules.

Turns out you can't use shutil.copytree() inside a container to replace cp -RL.

That code attempts to use os.setxattr() at the lowest level to copy all the attributes properly. But when not running as a real root user in a container, you can't copy all attributes, and a "Permission denied" exception is raised.

The original pbench-metadata-log code just used cp -rL and it worked both in and out of a container. So we just invoke that command directly again from the new Python code in the tool_data_sink.py module.

[portante]

Still a work-in-progress, as we need to implement the pbench-metadata-log functionality in as a callable API perhaps.

[Maxusmusti]

Re-tested with local and remote node, transient tools + persistent tools, works as expected

[dbutenhof]

Doesn't appear that GitHub thinks anything significant has changed since my last review, so let's get it in.