Introducing monitoring

[BentiGorlich]

• add monitoring capabilities for troubleshooting purposes. This cannot be enabled or disabled from the UI, you have to go into the .env files for this
• Add monitoring capabilities for the biggest factors: curl requests (AP), twig rendering (frontend) and of course database querying
• 2 sources can start an execution context: an incoming request and a started message
• added an admin UI for inspecting requests and their response times, with an overview grouped by route name

The main goal of this PR is to add monitoring capabilities, so we can see what our messengers are doing in the background. Additionally I think the graph in the overview (even though it is from a dev instance) shows pretty well that we have to tackle our twig rendering. My guess is that that is a big problem on larger instances, like fedia.io.

Some screenshots:

[BentiGorlich]

Just a warning: In about 2 days the gathered stats take up about 16GB on my relatively small server that mainly gets a lot of AP requests, but not that much user requests

table_name	table_size	indexes_size	total_size
"public"."monitoring_query"	14 GB	599 MB	14 GB
"public"."monitoring_twig_render"	1338 MB	294 MB	1633 MB
"public"."monitoring_execution_context"	345 MB	52 MB	396 MB
"public"."monitoring_curl_request"	14 MB	4792 kB	18 MB

[blued-gear]

Maybe the stats then should get their own DB connection configuration. This way admins can store the data somewhere where the fill up is not much of a problem or even choose a DB which allows compression.

[melroy89]

This is really exciting !

But yeah, you definitely want this is a separate DB.. Or maybe even in a timeseries optimized DB. Especially due to the sizes.

[BentiGorlich]

I think I can bring down the size a lot if I separate the query string out to another table and only save a reference to that in the "instances" of the query. Also I want to make some things optional. At the moment it does record the parameters of each query as well, which might be overkill or just unnecessary, so I want it to be optional

[BentiGorlich]

And it is quite a lot of data, so you might not want to monitor it for that long anyway. Otherwise you just don't get through it :D

[melroy89]

And it is quite a lot of data, so you might not want to monitor it for that long anyway. Otherwise you just don't get through it :D

We need monitoring of the monitoring, that is monitoring. 😅

[BentiGorlich]

Ok, so I pushed a few changes:

• to save space in the DB:
  • make parameter storing optional via env var
  • similar queries/statements will not take up space for each call, but will be saved to a separate table and referenced by hash
• Make the overview chart also display aggregated stats based on the current filter
• Add a dropdown to switch between the total time and the mean time in the overview chart
• Add a gradient to the twig renders either based on the percentage of the total duration or the parent duration

I have it live on gehirneimer, without saving the parameters to the DB. Lets see how much space it is taking up in 2 days :)

[BentiGorlich]

Now after 3,5 days we have this:

table_name	table_size	indexes_size	total_size
"public"."monitoring_query"	5397 MB	1518 MB	6915 MB
"public"."monitoring_twig_render"	1906 MB	418 MB	2324 MB
"public"."monitoring_execution_context"	706 MB	106 MB	812 MB
"public"."monitoring_curl_request"	27 MB	9472 kB	36 MB
"public"."monitoring_query_string"	19 MB	888 kB	20 MB

still quite a bit and I have parameter logging disabled

[BentiGorlich]

I think making query logging and twig render logging optional this is a lot more viable for just use in general, instead of a developer tool.

[blued-gear]

Would you also add some markdown documentation about this feature?

[blued-gear]

I checked out the branch and ran the migrations. The monitoring page shows Query monitoring is enabled, but only the execution time. Twig rendering is enabled, but only the execution time. Curl request monitoring is enabled, but only the execution time.. I did not update the environment yet, so I thought it should be disabled. Also, no data is recorded to the DB.

[blued-gear]

It would be useful to have a button for deleting all recorded monitoringdata.

[BentiGorlich]

It would be useful to have a button for deleting all recorded monitoringdata.

I thought about that, but decided against it, because if the monitoring ran for a while, then the request will probably time out. It takes a while to delete all this data. I just forgot to add a command deleting it 😅

[BentiGorlich]

I checked out the branch and ran the migrations. The monitoring page shows Query monitoring is enabled, but only the execution time. Twig rendering is enabled, but only the execution time. Curl request monitoring is enabled, but only the execution time.. I did not update the environment yet, so I thought it should be disabled. Also, no data is recorded to the DB.

The monitoring in general is disabled, so you should not have seen a link in the admin options, but like mentioned in the docs and the .env some settings default to true

[blued-gear]

onitoring ran for a while, then the request will probably time out. It takes a while to delete all this data. I just forgot to add a command deleting it 😅

A command is fine too.

[blued-gear]

I checked out the branch and ran the migrations. The monitoring page shows Query monitoring is enabled, but only the execution time. Twig rendering is enabled, but only the execution time. Curl request monitoring is enabled, but only the execution time.. I did not update the environment yet, so I thought it should be disabled. Also, no data is recorded to the DB.

The monitoring in general is disabled, so you should not have seen a link in the admin options, but like mentioned in the docs and the .env some settings default to true

I noticed it afterward. The link is indeed not shown in the adminpanel. But as the docs reference the URL, it would be good to show on the page that monitoring is generally disabled in such a case.

[blued-gear]

The new command needs a quick documentation in our list of commands.