Following restart node continues to log "Commitlog shutdown complete" messages #11508
Comments
|
@michoecho , I probably mis-understood the commitlog call sequence and put the logging into wrong place. The intention was to print something into logs when database shuts down its commitlog on scylla stop/abort. Feel free to remove this log line if you think it's not applicable there. |
|
I see it in my tests as well. |
|
@xemul I think you put it in the place you intended. The problem is, the main database commitlog isn't the only commitlog in the program. Hinted handoff reuses the implemention for it's own little transient commitlogs, which are destroyed and recreated (or just shut down and reopened, I don't know how exactly it works) every 10s, and this causes the log spam. |
The intention was for these logs to be printed during the database shutdown sequence, but it was overlooked that it's not the only place where commitlog::shutdown is called. Commitlogs are started and shut down periodically by hinted handoff. When that happens, these messages spam the log. Fix that by adding INFO commitlog shutdown logs to database::stop, and change the level of the commitlog::shutdown log call to DEBUG. Fixes scylladb#11508
|
A fix (#11536) is already queued. |
The intention was for these logs to be printed during the database shutdown sequence, but it was overlooked that it's not the only place where commitlog::shutdown is called. Commitlogs are started and shut down periodically by hinted handoff. When that happens, these messages spam the log. Fix that by adding INFO commitlog shutdown logs to database::stop, and change the level of the commitlog::shutdown log call to DEBUG. Fixes #11508 Closes #11536 (cherry picked from commit 9b6fc55)
|
Backported to 5.1. Earlier releases are unaffected. |
Installation details
Kernel Version: 5.15.0-1019-aws
Scylla version (or git commit hash):
2022.2.0~rc1-20220902.a9bc6d191031with build-id074a0cb9e6a5ab36ba5e7f81385e68079ab6eedaCluster size: 5 nodes (i3.4xlarge)
Scylla Nodes used in this run:
OS / Image:
ami-03d21402486bbce67(aws: eu-west-1)Test:
longevity-mv-si-4days-testTest id:
371f29c2-5d87-4a5a-a487-a974ebf7edd1Test name:
enterprise-2022.2/longevity/longevity-mv-si-4days-testTest config file(s):
Issue description
>>>>>>>
During
RollingConfigChangeInternodeCompressionwe change the internode compression settings (duh!) and restart the nodes in the cluster in a rolling fashion.During the shutdown process we get the expected log line informing about the commitlog shutdown:
node 1 db log:
However this message gets logged after the restart every few seconds:
node 1 db log:
This continues on all the db nodes, through 7+ hours of the test, taking up ~85% of the db nodes log output:
<<<<<<<
$ hydra investigate show-monitor 371f29c2-5d87-4a5a-a487-a974ebf7edd1$ hydra investigate show-logs 371f29c2-5d87-4a5a-a487-a974ebf7edd1Logs:
Jenkins job URL
The text was updated successfully, but these errors were encountered: