Finer control over log deletion logic with a new config option

[dyrock]

Small logs that rotate more frequently are wiped out and the loss of diagnostic logs make it harder to debug. This pull request brings a new deletion mechanism with a config option of max space allowed for all types of log files. (proxy.config.log.auto_delete_files_space_limits_mb)

The deletion will happen on the oldest file from the log type with biggest ratio=(total size used by the type)/(max space allowed for the type). Log types that are not configured with the max space allowed will not be considered for deletion. (If the config option is empty, then traffic server won't delete any files.)

Test case:
Config:
proxy.config.log.max_space_mb_for_logs INT 25
proxy.config.log.auto_delete_files_space_limits_mb STRING diags.log=7,traffic.out=8,squid.log=10
Files:
10 traffic.out (1M each), 10 diags.log (1M each), and 10 squid.log (5M each)

Verified that 1) it does delete files and 2) deletion happens in order and 3) deletion works not just for the first time.

[bryancall]

Seems odd to have this option proxy.config.log.auto_delete_files_space_limits_mb be the only one that is combined for the logs while all the other options are split out into their own config:

proxy.config.log.rolling_enabled: 3
proxy.config.log.rolling_interval_sec: 900
proxy.config.log.rolling_offset_hr: 0
proxy.config.log.rolling_size_mb: 6000
proxy.config.diags.logfile.rolling_enabled: 3
proxy.config.diags.logfile.rolling_interval_sec: 900
proxy.config.diags.logfile.rolling_size_mb: 6000
proxy.config.output.logfile.rolling_enabled: 3
proxy.config.output.logfile.rolling_interval_sec: 900
proxy.config.output.logfile.rolling_size_mb: 6000

I suggest adding a configuration option (3 options) for each of the log types.

[SolidWallOfCode]

We can split it if you like.

[SolidWallOfCode]

What should the new option be? I like the ability to set a size limit for rotated files, rather than just a flag of delete or not. In practice, though, the numbers are really dimensionless weights to provide space based priorities. E.g. if A is set to 100MB and B to 200MB, that means that the rotated files will be clipped such that A takes about half as much space as B as files are deleted to get under the headroom limit.

[dyrock]

@bryancall proxy.config.log.auto_delete_files_space_limits_mb might be better in terms of control over custom logs with user-defined names?

[bryancall]

@dyrock custom log overrides should be done in yaml, like they are now for the rotation options

[SolidWallOfCode]

This was discussed at length during the summit and @dyrock will make the agreed upon changes.