Refactor tool-meister-start orchestration

[portante]

We needed a way to specify what IP address and port to use when the Redis server and Tool Data Sink (really its WSGI server) bind for their respective services, while also being able to specify what address and port the Tool Meister(s) should connect to in environments where those address and ports are different (software-defined-networking, or NAT'd, or some other reason).

We solve this need by adding a new option, --orchestrate, along with a new --tool-data-sink option, and adjust the behaviors a bit.

The new --orchestrate option accepts two keywords, create and existing, which allow the caller to direct tool-meister-start to either be responsible for creating the Redis server, Tool Data Sink, and Tool Meister instances, or use existing instances. The --redis-server (or the PBENCH_REDIS_SERVER environment variable) and --tool-data-sink (or the PBENCH_TOOL_DATA_SINK environment variable) options give the caller the ability to specify the IP/port combinations to use in either operating case.

When --orchestrate=existing is specified, the --redis-server and --tool-data-sink options (or environment variables) must be specified.

The format for specifying the IP/ports to use via the --redis-server and --tool-data-sink options can be one of the following:

• <host ip>:<port>: used both for listening and connections
• <listen host ip>:<port>;<connection host ip>:<port>: the 1st host/port pair is used for listening for connections, the 2nd host/port pair is used to specify how to make connections to the instances

The defaults are:

• Redis: bind & connect, ${_pbench_full_hostname}:17001
  • Also binds to localhost
• TDS: bind & connect, ${_pbench_full_hostname}:8080

Prior to this change, pbench-tool-meister-start had one very long main method. This refactoring breaks out the orchestration pieces required to start the Redis server, Tool Data Sink, and the Tool Meisters, into their own methods and classes, reducing main to something a bit more reasonable.

As part of this work, a new BaseServer abstraction, sub-classed for the RedisServer and ToolDataSink, now handles how the Redis server and Tool Data Sink instance tracking works, either when they are created by tool-meister-start, or created ahead of time by the caller.

[lgtm-com]

This pull request introduces 2 alerts when merging 0903571 into cf66c53 - view on LGTM.com

new alerts:

• 1 for Missing call to `__init__` during object initialization
• 1 for Wrong number of arguments in a call

[portante]

@RobertKrawitz, okay here is the fixed code to handle specifying the listen (binding) vs connection IP/ports for both the Tool Data Sink and the Redis Server as created by pbench-tool-meister-start.

Please take a moment to review the PR description where we describe how to use PBENCH_REDIS_SERVER_CFG and PBENCH_TOOL_DATA_SINK_CFG contents.

[portante]

I need to test this in a real container, still, but it is a start.

[ndokos]

Light first pass: typos and some questions. Need to go through main() in more detail though.

[Maxusmusti]

Looks awesome so far, just a couple of design questions

[webbnh]

More to come....

[webbnh]

Done! (For this round....)

[portante]

Okay, since some of the code-review comments were asking about validating host names and IP addresses, and we had issue #2201 come up during this window of work, I decided to add a commit to solve #2201 first. It is now the first commit before any other changes.

I'll follow that up with another refactoring so that the pbench-tool-meister-start host name processing can leverage it as well.

[dbutenhof]

The levels of validate-hostname / validate_hostname are a bit convoluted, but I like the idea.

I see Webb pointed out a few problems that need to be resolved; but pending that, it's looking good.

[portante]

@Maxusmusti, @ndokos, this PR still have 2 changes requested from the two of you. Is that still the case?

[webbnh]

I assume that you are going to open an issue for the IPv6 address-with-port format, rather than add it to this PR, so, modulo the documentation nit below (and Nick's concerns), I think this is good to go.

[portante]

Okay, ready for another pass. I had to rebase anyways, and these changes were straightforward to make.

[dbutenhof]

Yeah, we need to clean up the IPv6 addressing, but since we're not using IPv6 I guess that can be deferred. (Do we even have IPv6 in RDU2 or BOS labs?)

[Maxusmusti]

Passed usual local/remote node testing with expected results

[webbnh]

Looks great, let's ship it! (Please don't make me review the entire thing again....)

What? Oh, them?...no, pay no attention to those comments.....

[dbutenhof]

I say go.