Add resource requests and limits for web, plugins, and CH

[fuziontech]

Adding sane values pulled from product (and reduced pretty significantly)

For example on prod we give plugins 4cpu and 16gb of ram per task

[tiina303]

This would be a major upgrade. It requires us to also update all our docs & increases the minimum deployment price by quite a bit.

[fuziontech]

It definitely would be a major update - but it would at provide a more consistent and less unexpectedly buggy experience.

It would also require bumping up the minimum size for a deployed cluster. I am for it because of the consistency of experience it would give users and it would also unlock the node autoscaling on DO for k8s.

[tiina303]

So here's a proposal:

1. We change our default to be a more sized up autoscalable version (vs the current minimal version), we'll also enable hpa & size up the pvc's. I'd go a step further and default enable prometheus too then.
2. We add documentation for how to get the cheapest possible self-hosted instance (with a big note that they probably just want to use cloud), we used to have a cheap self-hosted version (Heroku) so maybe we should have a way to get that with ClickHouse, but it's also ok for us to just tell everyone who's price conscious to move to the cloud instead.

[guidoiaquinti]

+1 to add those. Once we agree on the values let's remember to update also our the cluster requirements section of our docs.

[hazzadous]

I going to close this to reduce the noise in this repo, if it's really important feel free to open again.