We have to move monitoring on it's own VM

[alexgarel]

Describe the bug
All the monitoring infrastructure is on preprod 200 and this is absolutly not a good idea.
Every-time we want to update the preprod data (the zfs clone containing sto), we have to reboot the machine.

Expected behavior

I propose to put monitoring on it's own container, because it is far better when monitoring is independent !

(I propose to gain on 105 or 103 machines resources).

[alexgarel]

@CharlesNepote and @cquest do you agree ?

[alexgarel]

#81 also have the same issue

[alexgarel]

@CharlesNepote could we provision a QEMU VM, possibly on ovh1.

I'm not sure how much resource we need. But I can look into it.

[alexgarel]

I would give:

• at least 150 Go disk (it is currently around 15G, but this will grow a lot (because of elasticsearch indexing logs))
• possibly 8 cpus (2 for ES, 1 for influx, 1 for prometheus, 1 for the rest, 3 for the querying tools grafana / ES)
• 25 Go (ES needs 12Go, prometheus 3Go, the rest is divided between remaining + a margin of 5Go)

@CharlesNepote could we provision this on ovh1 as a QEMU VM ?

It will then be quite easy to move the monitoring stack there.

[cquest]

Own VM or separate hardware outside of the free+OVH bare metals ?

[alexgarel]

@cquest do you have a proposition ?

If we want to lend a server outside, I would take one either in OVH, different DC or at Scaleway

• scaleway dedibox start-2L is around 50€/month (Vat incl.)
• ovh elite vps is around this or more…

@CharlesNepote ?

[CharlesNepote]

Let's start with OVH1 I guess. We still have 400 GB of disk and 190 GB RAM free on OVH1, let's use it!

Be aware it is easy to increase disk space of the VM while decreasing is not. So I would start with a machine with 80 GB and increase it when needed.

Own VM or separate hardware outside of the free+OVH bare metals ?

@cquest: why a separate machine?

[alexgarel]

I think we'd better start on OVH1 and then move if someone sponsors us a VM.

[alexgarel]

created the ticket for the VM, I will now create the VM.

[alexgarel]

It's done:

• documentation PR (report) : docs: Monitoring move #164
• deployment PR: ci: deploy to new vm openfoodfacts-monitoring#58