DPE-1796 Implement support for general/slowquery logs and logrotation of text logs

[shayancanonical]

Issue

1. We are rendering the custom mysqld config as a string (making it very difficult to maintain and trace)
2. We do not have general and slow query logs enabled
3. We are not rotating mysql text logs (error, general and slowquery)

Solution

1. Use python's in-built configparser to generate the mysqld custom config (an ini file) -> this has the added benefit of validation
2. Enable general and slow query logs
3. Use cron to rotate logs every minute. The logrotate's postrotate script will invoke a custom event in the charm (using juju-run in v2.9.X and juju-exec in >= v3.1.X) to execute the FLUSH statement. This is because we would like to avoid keeping database credentials on the machine

[carlcsaposs-canonical]

Issue
We do not have general and slow query logs enabled

I might be misremembering—I thought we didn't want to enable slow query logs but we did want to rotate them if they were enabled (cc @delgod)

[codecov]

Codecov Report

Merging #329 (d44da0e) into main (f0abc40) will decrease coverage by 0.09%.
The diff coverage is 63.88%.

@@            Coverage Diff             @@
##             main     #329      +/-   ##
==========================================
- Coverage   64.04%   63.96%   -0.09%     
==========================================
  Files          15       16       +1     
  Lines        2940     2997      +57     
  Branches      385      390       +5     
==========================================
+ Hits         1883     1917      +34     
- Misses        934      956      +22     
- Partials      123      124       +1     

Files	Coverage Δ
src/charm.py	51.60% <80.00%> (+0.57%)	⬆️
lib/charms/mysql/v0/mysql.py	72.74% <73.07%> (-0.14%)	⬇️
src/flush_mysql_logs.py	65.38% <65.38%> (ø)
src/mysql_vm_helpers.py	61.73% <40.00%> (-1.53%)	⬇️

[paulomach]

Left some non-blockers - want to see your opinion before approving

[paulomach]

nice work

[taurus-forever]

LGTM.

delete_file_or_directory_in_unit() could play us bad story... today it is in tests only, but tomorrow someone will use it in production. File removal "rm -f" and folder removal "rm -rf" have significantly different risks.

[taurus-forever]

Does it worth to ensure path is not "/" or "."? Otherwise: sudo rm -rf /

In general I would separate file removal, and avoid recursive call in this case.
I prefer find /path/ -mindepth 1 -mtime +5 -delete for the deep cleanup.
In this particular case, I would separate file and folder removal.
P.S. Better safe then sorry (c) this.

[shayancanonical]

ah, it would be terrible to have rm -rf run by mistake. changed to using find in a0052bf (with maxdepth=0 to match exact paths, and no mtime as the files the test deletes will be recently created)

[delgod]

Issue
We do not have general and slow query logs enabled

I might be misremembering—I thought we didn't want to enable slow query logs but we did want to rotate them if they were enabled (cc @delgod)

In my personal opinion:

• general log should be enabled, rotated and streamed to COS.
• slow log should be disabled by default in Charm because it is disabled by default in MySQL (maybe we should have the option to enable it in future). but when the slow log is enabled - it should be rotated.

in case of disagreement, the decision maker here is @taurus-forever

[taurus-forever]

Issue
We do not have general and slow query logs enabled

I might be misremembering—I thought we didn't want to enable slow query logs but we did want to rotate them if they were enabled (cc @delgod)

In my personal opinion:

• general log should be enabled, rotated and streamed to COS.
• slow log should be disabled by default in Charm because it is disabled by default in MySQL (maybe we should have the option to enable it in future). but when the slow log is enabled - it should be rotated.

in case of disagreement, the decision maker here is @taurus-forever

My opinion:

• slow query log is useful application level tool. DBA enables it to improve DB performance (indexes, caches, ...).
• IMHO, slow query log is disabled in production by default, but enabled on X% of servers to monitor X% of "traffic".
  IMHO2, Queries longer then X seconds have to be killed (and not reported to slow query log).
  This is a DBA level decisions...
• Canonical Data is an operator provider, not a DBA => we should "ship MySQL defaults"(c) unless we need to change them.
• Enabling "slow_query_log=ON" is a premature optimization we are granting to Application/DBA without the request and (what is most important) without the ability to overwrite us (as we ship it as mysql.conf.d/z-custom-mysqld.cnf).

Mohamed, let's agree to keep slow query log disabled for now, by default (at least until we provide configuration options to disable it => mysql tuning). Tnx!