resmgr: disallow talking to untested runtimes by default.

[klihub]

Disallow talking to/proxying for unknown runtimes, unless explicitly told to do so on the command line. This could save us quite a bit of time from chasing ghosts.

Inspired by an idea from @jukkar over a lunch discussion. Thanks !

[marquiz]

Heh 👌
/lgtm

[askervin]

A bit mixed feelings about this... a warning would have been easier for me, but at the same time I know that nobody would notice the warning, except for us when we are after a ghost already.

And on the other hand the scenario where we should to take back this change and allow docker by default, seems to be highly unlikely, says the crystal ball.

As a downside, we will need to add a couple of sentences about docker in our documentation, yet getting rid of it completely would be even nicer.

So after all, I suppose this is ok.

[klihub]

A bit mixed feelings about this... a warning would have been easier for me, but at the same time I know that nobody would notice the warning, except for us when we are after a ghost already.

There are mainly two reasons I chose to go with this approach:

1. What you also state above: nobody seems to read logs/warnings and if they do, they don't seem to pay attention.
2. We really want usage of untested runtimes to be opt-in. It should not work out of the box.

And on the other hand the scenario where we should to take back this change and allow docker by default, seems to be highly unlikely, says the crystal ball.

As a downside, we will need to add a couple of sentences about docker in our documentation, yet getting rid of it completely would be even nicer.

I think we should remove all documentation reference to docker. And add a single paragraph stating that:

1. Using untested runtimes is highly discouraged.
2. People should not expect feature parity with containerd/crio when using untested runtimes.
3. --allow-untested-runtimes must be passed on the command line to cri-resmgr, otherwise it will refuse to talk to untested runtimes

So after all, I suppose this is ok.

I added a mention of the necessary command line option to the rejection error message.

[klihub]

As a downside, we will need to add a couple of sentences about docker in our documentation, yet getting rid of it completely would be even nicer.

I wiped out all existing instructions related to docker (I could identify with a simple git grep) from our documentation and added a single paragraph along the lines I outlined above above.