distage-testkit: Smart memoization group merging.

[Caparow]

fix #1067

[codecov-commenter]

Codecov Report

Merging #1072 into develop will increase coverage by 0.27%.
The diff coverage is 92.59%.

@@             Coverage Diff             @@
##           develop    #1072      +/-   ##
===========================================
+ Coverage    55.65%   55.93%   +0.27%     
===========================================
  Files          380      381       +1     
  Lines         6917     6940      +23     
  Branches       362      358       -4     
===========================================
+ Hits          3850     3882      +32     
+ Misses        3067     3058       -9     

[neko-kai]

A thought: we can limit the amount of logs for re-reading the same config files by memoizing config reads with the same testBaseName (for example using a mutable map and getOrElseUpdate)

[Caparow]

A thought: we can limit the amount of logs for re-reading the same config files by memoizing config reads with the same testBaseName (for example using a mutable map and getOrElseUpdate)

A good idea!
Will do

[neko-kai]

Another thought, we can replace testRunnerLogLevel with just a switch debugOutput: Boolean and set level Debug on true, Info on false, and force level to Debug if property DebugProperties.\izumi.distage.testkit.debug`is set to true. Then we can also remove the separatedebugLoggerand just log withIzLogger.debug`.

[Caparow]

Another thought, we can replace testRunnerLogLevel with just a switch debugOutput: Boolean and set level Debug on true, Info on false, and force level to Debug if property DebugProperties.\izumi.distage.testkit.debug``is set to true. Then we can also remove the separatedebugLoggerand just log with`IzLogger.debug`.

But by this implementation we will lose control over the test log level. It's okay to have info level always, but sometimes we need to force warn level to get rid of the noise.

[neko-kai]

@Caparow Hmm, actually earlyLogger is used just once - when loading config, and currently there's no way to disable config loader printing - afterwards logger settings are read from config. Hmm, we may alter the log levels of the debug messages themselves (promote Debug to Info) instead of chaging the logger. We may also have three values for debugOutput = Silent/Normal/Verbose

[Caparow]

@Caparow Hmm, actually earlyLogger is used just once - when loading config, and currently there's no way to disable config loader printing - afterwards logger settings are read from config. Hmm, we may alter the log levels of the debug messages themselves (promote Debug to Info) instead of chaging the logger. We may also have three values for debugOutput = Silent/Normal/Verbose

You still can silence the config loader. Early logger uses testRunnerLogLevel. Also, a late logger is inherited from early logger. And there is no point to add new log level ADT (at least I don't know why we'll need this).