Provide a "System metrics" entry in the status page, to show disk space, RAM usage, and database size

[klonos]

This feature request was inspired by this discussion in Zulip around "Minimal Backdrop", which touched on topics around what would be the very minimum system specs where Backdrop core would run, and how well it would perform.

Something like this might be helpful to allow determining requirements when:

• moving a locally-built site to an actual hosting provider
• moving a live site to a different provider
• upgrading the hosting plan of an existing site (when certain specs seem to have been reaching capacity/limits)

[klonos]

I'm creating this issue here after @cellear's request, and looking for feedback around its potential usefulness and whether it is core-worthy, or something best placed in contrib.

PR with a sandbox that will allow people to play with this and view the code coming up soon...

[cellear]

I'm @cellear, and I approve this message.

Seriously, I think built-in metrics like this would be super useful. Hosting providers use tools like Sumo Logic and Drutiny to pull out information like this, but if we could give people a bit of that just by looking at the status report...how cool is that!

[klonos]

PR and sandbox available here: backdrop/backdrop#4254

Some thoughts/observations:

• Certain things such as disk usage, db size, and PHP memory size have native PHP functions/ways to grab them, but other things like RAM usage need to be parsed via commands that are operating-system-specific.
• There are ways to grab stats for instances of Backdrop installed on Windows, but our telemetry shows that over 95% of sites run on linux. Should we bother? Would these system metrics be of value in cases where for example the site is being built on a Windows machine, but is then to be moved to a linux host?
• Certain metrics, such as disk space and db size are fairly "persistent", in that they usually grow over time. So it makes sense to provide those as an indication (site admins can keep an eye for when these are running low, or in the case of the db when it grows too big). Other metrics though, such as system RAM usage and CPU load, only make sense if monitored over a period of time (because the server may be getting spikes of usage at some points, but otherwise sailing smoothly the rest of the time).
• Is it useful to show actual disk usage (which apparently includes operating system files etc.), or should we be showing something like media asset size instead? (how much space the documents/videos/images uploaded on the site take)

[klonos]

...should we be showing something like media asset size instead?

PR updated to show the size of the various files directories:

[klonos]

...OK, I'm gonna stop here for now, and wait for feedback/thoughts.

In the meantime, I'll spin up a Windows installation when I get a chance, and try things there. ...see if we can pull the same kind of metrics reliably on Windows.

[klonos]

...more thoughts:

• Compared to other entries in the status report page, this section is lengthy/dominant. Can anybody suggest a good way to provide some sort of "summary", so that everything else can be hidden away in a "more" toggle?
• Dashboard block to go with this?

[bugfolder]

As this is growing (and it's really nice to have!), should it perhaps be a separate report at admin/reports/system-metrics? The nature of the Status Report seems to be stuff that takes up only a line or so, and most "more" links only expand by another line or few. This listing is as long as, for example, the Telemetry separate report (admin/reports/telemetry).

[indigoxela]

If this went into core, I'd like to have a way to turn it off. And ideally it would be off by default.

Background to this: on shared hosting most of this is irrelevant info. One usually chooses a hosting plan and the available space is predefined (by quota). Even if the disk space per se were bigger (given that the site has access to that info, given that the main disk were relevant, given that...) - we already know how much we can have. Available RAM ... that's the hoster's job, not the job of a CMS. PHP memory usage changes with every request. CPU load is also something that changes.

The only info that might be relevant is the DB size, possibly public and private files ... but hardly ever needed.

Still I vote for this to be in contrib.

[docwilmot]

I'd also vote for contrib. This seems to be needed only in very limited situations

moving a locally-built site to an actual hosting provider
moving a live site to a different provider
upgrading the hosting plan of an existing site (when certain specs seem to have been reaching capacity/limits)

Question though, does the info reflect the current instance of backdrop, or the entire server? The sandbox site says:

RAM (system): 32% used (9.2 GB) | 21.26 GB free of 31.42 GB total
PHP memory usage: 16.17 MB | 16.24 MB (peak) | 12 MB (true)

Surely one barebones Backdrop site doesnt consume 9.2GB of RAM?

[klonos]

Thanks for the feedback everyone 🙏🏼

As this is growing (and it's really nice to have!), should it perhaps be a separate report at admin/reports/system-metrics? The nature of the Status Report seems to be stuff that takes up only a line or so...

Yeah 🤔

This seems to be needed only in very limited situations

FTR, I also feel that this should most likely be in contrib...

Having said that, when choosing a hosting plan in certain popular providers, the various available options include info about CPU/disk/RAM. For example:

• Linode: https://www.linode.com/pricing
  
• https://www.digitalocean.com/pricing/droplets#basic-droplets
  
• https://www.vultr.com/pricing
  

Anyway, when it comes to deciding on a plan, I personally often go with one of the smallest/cheapest plans, thinking that if the site is not performing well, then I can always upgrade to a bigger plan. Perhaps others have other ways to be deciding, but the drive for this feature was the discussion in Zulip around what specs are needed in order to run Backdrop. I was hoping that this would be an indicator that would help with that.

Surely one barebones Backdrop site doesnt consume 9.2GB of RAM?

Yeah, I thought that too, but that system RAM is not about the RAM that Backdrop needs, rather than what is being consumed by the server that runs it (or your computer if the site is local) ...which makes me question whether that's useful, since different OS and different configurations may mean different consumption/needs ...on the other hand, reporting only on the currently-used PHP RAM might be misleading 🤔

[klonos]

...I'm thinking that perhaps we can only display some of these metrics, in existing sections where possible:

• the PHP memory consumption could be shown in a "more" link under the existing "PHP" section
• the db size in the existing "MySQL Database" section (which should really be called "Database" by the way - I know that we don't support SQLite, Postgres etc., but we also support MariaDB)
• the files directory total size could go in a new section, with the breakdown of temp/public/private in a "more" toggle

Anyway, I may file a separate PR with that, but the point is that the above metrics can be retrieved using php-native functions or code - the rest of the metrics (such as system RAM/disk and CPU) cannot be reliably retrieved in a OS-agnostic way or in a way that would make them useful, so perhaps leave these for contrib(?) 🤔

[findlabnet]

I'd also vote for contrib. This seems to be needed only in very limited situations.

Or even less frequently.

[kiamlaluno]

Anyway, I may file a separate PR with that, but the point is that the above metrics can be retrieved using php-native functions or code - the rest of the metrics (such as system RAM/disk and CPU) cannot be reliably retrieved in a OS-agnostic way or in a way that would make them useful, so perhaps leave these for contrib(?)

I vote for those three metrics to be added in core. The other metrics should be in a contributed module.

[indigoxela]

@klonos do you prefer to work on the two newly created issues and close this one here? Or is this issue still relevant? If so, what's its purpose?

[klonos]

Hey @indigoxela ...sorry for the late reply. The other two issues was an attempt to de-scope and split this issue to more manageable tasks, the only thing left that should be done as part of this issue here is to determine if there are any non-db and non-php related metrics worth to be shown in the site status page.

From previous comments here, I believe that the only helpful thing remaining which we could reliably grab info for is the size of the files directory. Since this thread here has many comments now, I think I'll file a 3rd separate issue for this (also incorporating comments/concerns that @indigoxela has expressed in the PR), and I will then close the issue here.