network destroy sometimes leave port used

[vega-paul]

Network stop sometimes does not stop all processes and leaves a port still in use (4647)

To stop the network we use

	@cd "${TESTS_DIR}" && ./vegacapsule network destroy --home-path "${NETWORK_HOME_PATH}/" || echo "Network already stopped"

This should be enough but occasionaly a port is left bound, shutdown fails.....

2023/02/04 12:10:42 Stopping network...
2023/02/04 12:10:42 Couldn't connect to nomad, skipping network shutdown...
2023/02/04 12:10:42 Cleaning up network...
2023/02/04 12:10:42 Network has been successfully cleaned up.

As part of ST cleanup we also run the following cmds in addition to above, but this should not be necessary.

	@pkill -9 vega || echo "No vega orphan is running"
	@pkill -9 visor || echo "No visor orphan is running"
	@pkill -9 data-node || echo "No data-node orphan is running"
	@pkill -9 tendermint || echo "No tendermint orphan is running"
	@pkill -9 vegawallet || echo "No vegawallet orphan is running"
	@pkill -9 faucet || echo "No faucet orphan is running"
	@pkill -9 toxiproxy || echo "No toxiproxy orphan is running"

With a bound port, this means the following nomad start fails

==> Loaded configuration from /Users/paulwebb/repos/system-tests/vegacapsule/nomad_config.hcl
==> Starting Nomad agent...
==> Error starting agent: server setup failed: Failed to start RPC layer: listen tcp 0.0.0.0:4647: bind: address already in use
    2023-02-06T12:19:02.976Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=""
    2023-02-06T12:19:02.976Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=""
    2023-02-06T12:19:02.977Z [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
    2023-02-06T12:19:02.977Z [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
    2023-02-06T12:19:02.977Z [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
    2023-02-06T12:19:02.977Z [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
    2023-02-06T12:19:02.977Z [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
    2023-02-06T12:19:02.978Z [ERROR] nomad: failed to initialize TLS listener: error="listen tcp 0.0.0.0:4647: bind: address already in use"
    2023-02-06T12:19:02.978Z [INFO]  nomad: shutting down server
    2023-02-06T12:19:02.978Z [ERROR] nomad: failed to start RPC layer: error="listen tcp 0.0.0.0:4647: bind: address already in use"
    2023-02-06T12:19:02.978Z [ERROR] agent: error starting agent: error="server setup failed: Failed to start RPC layer: listen tcp 0.0.0.0:4647: bind: address already in use"
Error: failed to execute binary /Users/paulwebb/.vegacapsule/nomad_1.3.1 with error: exit status 1
Usage:
  ./vegacapsule nomad [flags]
  ./vegacapsule nomad [command]

Available Commands:
  logscollector Starts a log collection program that should be run as a logging sidecar inside Nomad job.

Flags:
  -h, --help                       help for nomad
      --install-path string        Install path for the Nomad binary. Uses GOBIN environment variable by default.
      --nomad-config-path string   Allows to use Nomad configuration

Global Flags:
      --home-path string   Specify the location of network home directory (default "/Users/paulwebb/.vegacapsule/testnet")

Use "./vegacapsule nomad [command] --help" for more information about a command.

[gordsport]

@vega-paul
is this the same as:

• Error: failed to start network: network not bootstrapped #378

i.e. need to run make cleanup

[gordsport]

Closing as a duplicate of:

• Error: failed to start network: network not bootstrapped #378