Clarify how to use Runner Groups correctly

[toast-gear]

There is a lot of confusion on how runner groups work and how to correctly leverage them with ARC. Associating a repository with a runner group provides access to runners in that runner group in addition to the runners associated with Default runner group and runners in any other additional runner groups that the repository is also associated with.

Runner groups are NOT designed as a means for locking a repository to using runners not in the Default runner group i.e. custom runner groups only. If you have runners you need to protect you need by putting them in runner groups and give each set a unique label. it is totally legitamate that if you have a workflow which targets a common label like self-hosted or a custom common label then runners from the Default runner group pick up those queued jobs. If the repository is a member of 2 or more runner groups then each set of runners needs a unique label relative to each other and relative to the runners in your Default runner group.

Basically the entire system is designed around:

• Runner groups being a way of extending runner options, not restricting runner options
• The uniquenss of labels

[fheinecke]

This is an issue with labels as well as runner groups. It looks like there is not a way to restrict a runner to a specific job (or set of jobs) - only a way to restrict jobs to a specific runner.

This is a particularly big issue for running somewhat-untrusted CI workflows that need to be segregated from more trusted workflows such as production deployments. If a malicious actor is able to establish persistence from an untrusted workload (either by running a job in a non-ephemeral runner pod, or gaining access to the underlying node) then they could pivot to attack a security-critical workflow when it is sent to the same non-ephemeral runner, or a separate runner on the same underlying node.