Copying logfiles by MaxSize.rollPaths is slow #756
Description
Hey! Thanks for the great lib and the continuous effort with maintaining it.
I want to ask about
When there are bunch of already existing log files with a lot of logs, it takes significant amount of time for the MaxSize to copy all the files. As far as I can see, it will move N -> N+1, N-1 -> N, ..., current file -> 1. Copying the files is blocking operation that blocks the execution and causes program to hang for some amount of time.
In my rudimentary example, there is only maxSize used. Perhaps using day or hour could help, but nevertheless, the underlying issue would still be present.
reproduction snippet
I've used dd if=/dev/zero of=logs/app.log bs=1M count=1024 to mimic already existing log files. Second log stops the execution for the couple of seconds needed to copy the logfiles:
it took PT0.078805S to log message in iteration 0
it took PT7.695495S to log message in iteration 1
it took PT0.031225S to log message in iteration 2
it took PT0.020617S to log message in iteration 3
it took PT0.016023S to log message in iteration 4
//> using scala "3.7.2"
//> using dep "com.outr::scribe::3.17.0"
//> using dep "com.outr::scribe-file::3.17.0"
import scribe.Logger
import scribe.format.*
import scribe.file.*
import scribe.file.path.MaxSize
import java.time.Instant
import java.time.Duration
import scala.sys.process.*
import java.io.File
object Main extends App {
val size = MaxSize.OneHundredMeg * 5
val file = "app" % maxSize(max = size, separator = ".") % ".log"
val fileWriter = FileWriter("logs" / file)
Logger.root
.clearHandlers()
.withHandler(writer = fileWriter)
.replace()
// emulating already existing logfiles
new File("logs/").mkdirs()
val _ = "dd if=/dev/zero of=logs/app.log bs=1M count=1024".!!
(1 to 10).foreach { i =>
val _ = s"dd if=/dev/zero of=logs/app.$i.log bs=1M count=1024".!!
}
var i = 0
val payload = "0" * 1024 * 1024 + '\n'
while (i < 5) {
val before = Instant.now()
scribe.info(payload)
val after = Instant.now()
val between = Duration.between(before, after)
println(s"it took ${(between)} to log message in iteration $i")
i += 1
}
}