Create project pool to be used by scalability build cluster

[oxddr]

Yesterday we've created a separate build cluster to be use by Prow to run scalability tests [1]. It's missing Boskos and tests which rely on project pool doesn't work.

Plan for now:

1. Disable dedidcated build cluster in kubemark-100 (it uses project pool)
2. Create new project pool
3. Setup Boskos in dedicated build cluster so projects can be picked by tests

[1] kubernetes/test-infra#12847 (comment)

[oxddr]

/assign oxddr

[oxddr]

Alternatively to maintaining a dedicated project pool (AFAIU it requires running Boskos in scalability build cluster, right @krzyzacy) we can only migrate tests with dedicated project. Especially those are ones, which cause most of troubles (they are long running and need lots of resources). This way we can minimize maintenance work on our side, but keep benefits of having a dedicated build cluster. A downside is that we run tests in two place. However this should be transparent most of the time AFAIU.

Opinions? @wojtek-t @mm4tt @mborsz @krzyzacy

[krzyzacy]

The maintenance cost of boskos is relatively low, it runs both internal and external and we rarely need to worry about that, except when we don't have enough projects in the pool. But that's your call :-)

[mm4tt]

+1 for migrating only the tests with the dedicated projects for now.
I don't see much gain in doing that for smaller tests (using boskos), as they are not affected by the kubernetes/test-infra#11594 (or at least run often enough to mitigate the problem).
There are tasks with higher priority than this that we should work on right now.

Let's keep this bug open though. We can come back here when we have less on our plates.

[krzyzacy]

sgtm - feel free to send follow up PRs, you just need to add cluster: scalability in the prowjobs that you want to run on your build cluster.

[oxddr]

/priority awaiting-more-evidence
/sig scalability

[oxddr]

/close

We decided not to migrate tests other than 5k performance and correctness to a dedicated build cluster. The effort (e.g. setting up Boskos) is not worth gain (not sure if there is any).

[k8s-ci-robot]

@oxddr: Closing this issue.

In response to this:

/close

We decided not to migrate tests other than 5k performance and correctness to a dedicated build cluster. The effort (e.g. setting up Boskos) is not worth gain (not sure if there is any).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.