CBG-3925: Add FileLogger RotationInterval

[bbrks]

CBG-3925

• Add FileLogger RotationInterval option to force a periodic log rotation.
• Interval is opt-in and can be set anywhere between 15 minutes and 7 days.
• Logs will rotate even if no log lines have been written, producing an empty file. This makes it very obvious that log coverage is present and empty, rather than just missing.

Integration Tests

• GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGateway-Integration/2465/

[github-actions]

Redocly previews

• Admin API
• Public API
• Metric API
• Diagnostic API

[torcolvin]

LGTM, just minor comments.

[torcolvin]

nit: I think this reads just a bit easier to see rotationTicker.C since I had to look a back to see if this was a normal ticker.

[bbrks]

I ended up writing it this way because rotationTicker can be nil if you haven't enabled a rotation interval. The select panics trying to access C on a nil ticker.

You can however, safely select over a nil channel, and this avoids having to write a separate conditional select if you have log rotation enabled.

[torcolvin]

Suggested change

	// log instead of error because it's likely this is going to fail on Windows for this test.
	// log instead of error because it's likely this is going to fail on Windows for this test if lumberjack has not finished rotation. The `time.Sleep` at the end of the tests tries to mitigate this.

I wonder if the windows build machines are set up to clear temporary files, this could end up filing up the disks on the machine. We could consider writing to cwd if the builds do a fresh checkpoint of sync_gateway or do a git clean -fdx

Any reason here to use os.MkdirTemp instead of t.TempDir ?

[bbrks]

t.TempDir fails because it asserts there was no error from RemoveAll. I originally wrote it like that.

[bbrks]

I battled with this for a day and ended up on this solution. The sleep doesn't prevent this error, because Lumberjack keeps the current file handle open for some reason, even after calling Close()

On *nix this doesn't matter because the OS will still let us remove the file/directory.