Per-labelset queues

[raskyld]

Clear and concise description of the problem

The endpoint to fetch pipelines queues currently returns all pipelines regardless of their labels.
This is problematic for #999 because I would like to be able to create different auto-scaled agent pools.

Suggested solution

Add filters in queues list endpoint by label set so I get a view of only the pipelines my agent pools can handle.

Alternative

Optional / Nice-to-have: Add an option to label prometheus time series about queues with the same labels.

Additional context

Use cases

Different autoscaler agent pool

Additional Information: #999

A simple use-case is that I would like to have a pool of Instances (Virtual Machines) for Docker backend runners that use a pre-built docker image and just start the agent with the good parameters, the first start would be a cold boot because the volume would need to be initiated etc but the instance would be powered-off and re-spinned when needed with the same volume (hot boot).

Another pool for the Local backend runners would recreate (cold boot) a complete VM at each runs for cases where containers are not fitting and we want to run directly inside a VM.

Finally, I would use Cloud Providers' managed Kubernetes for the Kubernetes backend and fine-tunes the settings for Woodpecker.
If I take @scaleway as an example, we have Kapsule and it would be really easy to auto-scale the cluster horizontally by adding nodes as the number of container / the average per node CPU / memory starts to saturate.

But to make that work well I need to be able to get queues status per label backend so it auto-scales based on the right metrics (the queue length of the respective backends)

Validations

• Checked that the feature isn't part of the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[raskyld]

Once I finished my current PR on the autoscaler project I will probably give it a try

[raskyld]

Sadly we won't be able to take advantage of database indexing feature because from what I see the model is a pure JSON string: https://github.com/woodpecker-ci/woodpecker/blob/main/server/model/task.go#L33 :/

What would you recommend guys?

• Making the filter server-side (fetch all Tasks but filter in the server back-end) which means the performance impact is server-side
• Rework the Task model so we can off-load the selection to the database which is good practice but would require a migration of the data base and some rework

[anbraten]

Finally had the time to have a look at this one. I currently see to options:

• add some filter to the queue endpoint and simply filter by label (the index shouldn't be that much of a problem for the beginning IMO as you would probably only have <100 pending tasks for normal instances)
• use the queue tasks list for the autoscaler and interpret the different labels at the autoscaler.

[anbraten]

This should now be possible woodpecker-ci/autoscaler#134