-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
core: enabling logCollector by default for coredump collection #11163
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we require a similar change in cluster-on-pvc.yaml
?
also a thought,
we were discussing adding something similar in krew
plugin so can we add something in krew and let users decide if they require this advance debugging?
also, can we make periodicity hourly
since this is upstream?
above are just my thought, feel free to add your comment and skip the changes
PLease add a commit message in the commit explaining why we need it by default. |
@parth-gr updated |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@@ -65,8 +65,8 @@ For more details on the mons and when to choose a number other than `3`, see the | |||
* `disable`: is set to `true`, the crash collector will not run on any node where a Ceph daemon runs | |||
* `daysToRetain`: specifies the number of days to keep crash entries in the Ceph cluster. By default the entries are kept indefinitely. | |||
* `logCollector`: The settings for log collector daemon. | |||
* `enabled`: if set to `true`, the log collector will run as a side-car next to each Ceph daemon. The Ceph configuration option `log_to_file` will be turned on, meaning Ceph daemons will log on files in addition to still logging to container's stdout. These logs will be rotated. (default: false) | |||
* `periodicity`: how often to rotate daemon's log. (default: 24h). Specified with a time suffix which may be 'h' for hours or 'd' for days. **Rotating too often will slightly impact the daemon's performance since the signal briefly interrupts the program.** | |||
* `enabled`: if set to `true`, the log collector will run as a side-car next to each Ceph daemon. The Ceph configuration option `log_to_file` will be turned on, meaning Ceph daemons will log on files in addition to still logging to container's stdout. These logs will be rotated. The coredump files will be generated in `/var/lib/systemd/coredump` directory on the host where the pod is running in case a daemon terminates with a segfault. (default: `true`) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we sure this is the location for all underlying OSes/kernels? What happens when the kernel.core_pattern
is not pointing to the coredumpctl
/ just a path/directory.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
one more place to update
@@ -65,8 +65,8 @@ For more details on the mons and when to choose a number other than `3`, see the | |||
* `disable`: is set to `true`, the crash collector will not run on any node where a Ceph daemon runs | |||
* `daysToRetain`: specifies the number of days to keep crash entries in the Ceph cluster. By default the entries are kept indefinitely. | |||
* `logCollector`: The settings for log collector daemon. | |||
* `enabled`: if set to `true`, the log collector will run as a side-car next to each Ceph daemon. The Ceph configuration option `log_to_file` will be turned on, meaning Ceph daemons will log on files in addition to still logging to container's stdout. These logs will be rotated. (default: false) | |||
* `periodicity`: how often to rotate daemon's log. (default: 24h). Specified with a time suffix which may be 'h' for hours or 'd' for days. **Rotating too often will slightly impact the daemon's performance since the signal briefly interrupts the program.** | |||
* `enabled`: if set to `true`, the log collector will run as a side-car next to each Ceph daemon. The Ceph configuration option `log_to_file` will be turned on, meaning Ceph daemons will log on files in addition to still logging to container's stdout. These logs will be rotated. The coredump files will be generated in `/var/lib/systemd/coredump` directory on the host depending on the underlying OS location for coredumps where the pod is running in case a daemon terminates with a segfault. (default: `true`) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, just a small suggestion
* `enabled`: if set to `true`, the log collector will run as a side-car next to each Ceph daemon. The Ceph configuration option `log_to_file` will be turned on, meaning Ceph daemons will log on files in addition to still logging to container's stdout. These logs will be rotated. The coredump files will be generated in `/var/lib/systemd/coredump` directory on the host depending on the underlying OS location for coredumps where the pod is running in case a daemon terminates with a segfault. (default: `true`) | |
* `enabled`: if set to `true`, the log collector will run as a side-car next to each Ceph daemon. The Ceph configuration option `log_to_file` will be turned on, meaning Ceph daemons will log on files in addition to still logging to container's stdout. These logs will be rotated. In case a daemon terminates with a segfault, the coredump files will be commonly be generated in `/var/lib/systemd/coredump` directory on the host, depending on the underlying OS location. (default: `true`) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure, updated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, would be great if you could also enable the log collector in the tests, should just need to add it here.
enabling logCollector by default will enable the coredump generation in case a process terminates with a segmentation fault. Signed-off-by: gauravsitlani <gauravsitlani@riseup.net>
@travisn sure, just added it there. Let me know if it looks good |
Looks good thanks, i'll just wait to approve after the CI finishes |
core: enabling logCollector by default for coredump collection (backport #11163)
Signed-off-by: gauravsitlani gauravsitlani@riseup.net
Description of your changes:
To enable log collector by default for getting coredump for troubleshooting in case of segfaults.
Which issue is resolved by this Pull Request:
Resolves #10788 #11151
Checklist:
skip-ci
on the PR.