verbose_logging_timeout_sec_max #15798

oleiman · 2023-12-20T06:47:06Z

This PR introduces verbose_logging_timeout_sec_max, which, if set, causes expiration times included in set_log_level requests to be clamped before timeouts are calculated.

Includes an assortment of minor improvements to the logic around setting log levels and managing timeouts around those settings.

Fixes https://github.com/redpanda-data/core-internal/issues/938
Fixes https://github.com/redpanda-data/core-internal/issues/931
Fixes https://github.com/redpanda-data/core-internal/issues/930

Backports Required

Release Notes

Improvements

verbose_logging_timeout_max node config, to prevent setting TRACE and DEBUG log level indefinitely

Added option to v1/loggers to provide the log levels in addition to the names of the loggers. Use v1/loggers?include-levels=true Signed-off-by: Michael Boquard <michael@redpanda.com> Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

src/v/redpanda/admin/api-doc/config.def.json

vbotbuildovich · 2023-12-20T08:45:02Z

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/43112#018c8628-5a3e-4b3c-8840-b1dffe1713f0

michael-redpanda

looks fantastic, just a couple of notes

src/v/redpanda/admin/api-doc/config.def.json

src/v/config/node_config.cc

src/v/config/node_config.h

src/v/redpanda/admin/server.h

oleiman · 2023-12-20T16:50:53Z

CI Failures:

CI Failure (wrong value on nodes) in ClusterConfigAliasTest.test_aliasing #15117 (previously known)

Added message to set_log_level response that reports previous level of the logger. Signed-off-by: Michael Boquard <michael@redpanda.com> Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Signed-off-by: Michael Boquard <michael@redpanda.com> Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman · 2023-12-20T19:18:14Z

force push contents:

rename config to add units
config defaults to nullopt to preserve existing behavior
add get_log_level to rptest.Admin
tests check get_log_level (though it's still useful to monitor logs)
nits

michael-redpanda · 2023-12-21T13:32:38Z

/ci-repeat 5
dt-repeat=5
skip-redpanda-build
tests/rptest/tests/log_level_test.py

michael-redpanda

couple more questions but looks great!

src/v/config/node_config.cc

src/v/redpanda/admin/server.h

Max duration for TRACE or DEBUG level logging. If null (default) we impose no limit. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Allows setting an expiration that exceeds `verbose_logging_timeout_max`. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

- Adds get_log_level - Improve set_log_level - Allows setting expiration=0 - Adds force query parameter Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman · 2023-12-21T16:39:51Z

force push for better operator<(level_reset&, level_reset&) semantics and bounded_property for new config

michael-redpanda

lgtm

src/v/redpanda/admin/server.h

This allows us to keep the default level for each logger around in the level reset map and distinguish between loggers with expiring settings and those without when the log reset timer fires. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

- Clamps timeouts to node config - inclusive of 0-valued setting (no timeout) - Respects force query param to override configured max - Log the timeout (after clamping) at INFO level Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman · 2023-12-21T17:10:27Z

force push condition

vbotbuildovich · 2023-12-21T20:28:53Z

/backport v23.3.x

vbotbuildovich · 2023-12-21T20:28:54Z

/backport v23.2.x

vbotbuildovich · 2023-12-21T20:28:54Z

/backport v23.1.x

vbotbuildovich · 2023-12-21T20:29:49Z

Oops! Something went wrong.

Workflow run logs.

vbotbuildovich · 2023-12-21T20:29:51Z

Failed to create a backport PR to v23.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-15798-v23.2.x-896 remotes/upstream/v23.2.x
git cherry-pick -x 2404523d7824bc4b19e2097901dd8da59f11897a cc792b7a5889e8d3d83a4a690d4950747d8f92bb ab79ddc842d1ea5e9d74bd486af4b726cdc8ecdf 194ab70ae968380036ed07f9aaf1a8f30006cc63 d15cd37b20b3d4f7a693c00218710d13349b67c2 88ffdea66e915f02a29f84f32e767ea18d702bbf d1f95563a7964381d17bc55bf453f7ab1efb369c 1cc5583e2899ca07ada60e9a16dd0bb7692a1cb5 4d4c975dc41d7ba047e6afc47fc55be31c0ea219

Workflow run logs.

vbotbuildovich · 2023-12-21T20:29:55Z

Failed to create a backport PR to v23.1.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-15798-v23.1.x-301 remotes/upstream/v23.1.x
git cherry-pick -x 2404523d7824bc4b19e2097901dd8da59f11897a cc792b7a5889e8d3d83a4a690d4950747d8f92bb ab79ddc842d1ea5e9d74bd486af4b726cdc8ecdf 194ab70ae968380036ed07f9aaf1a8f30006cc63 d15cd37b20b3d4f7a693c00218710d13349b67c2 88ffdea66e915f02a29f84f32e767ea18d702bbf d1f95563a7964381d17bc55bf453f7ab1efb369c 1cc5583e2899ca07ada60e9a16dd0bb7692a1cb5 4d4c975dc41d7ba047e6afc47fc55be31c0ea219

Workflow run logs.

admin: Added levels to v1/loggers

2404523

Added option to v1/loggers to provide the log levels in addition to the names of the loggers. Use v1/loggers?include-levels=true Signed-off-by: Michael Boquard <michael@redpanda.com> Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman requested a review from michael-redpanda December 20, 2023 06:47

oleiman self-assigned this Dec 20, 2023

github-actions bot added the area/redpanda label Dec 20, 2023

oleiman marked this pull request as draft December 20, 2023 07:04

oleiman commented Dec 20, 2023

View reviewed changes

src/v/redpanda/admin/api-doc/config.def.json Show resolved Hide resolved

michael-redpanda reviewed Dec 20, 2023

View reviewed changes

src/v/redpanda/admin/api-doc/config.def.json Outdated Show resolved Hide resolved

src/v/config/node_config.cc Outdated Show resolved Hide resolved

src/v/config/node_config.h Outdated Show resolved Hide resolved

src/v/redpanda/admin/server.h Outdated Show resolved Hide resolved

michael-redpanda added 2 commits December 20, 2023 09:37

admin: Added response to set_log_level

cc792b7

Added message to set_log_level response that reports previous level of the logger. Signed-off-by: Michael Boquard <michael@redpanda.com> Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

admin: Added get_log_level

ab79ddc

Signed-off-by: Michael Boquard <michael@redpanda.com> Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman force-pushed the issue/ci-938 branch from f5f888f to df50d3b Compare December 20, 2023 19:16

oleiman requested a review from michael-redpanda December 20, 2023 19:18

oleiman marked this pull request as ready for review December 20, 2023 19:18

michael-redpanda reviewed Dec 21, 2023

View reviewed changes

src/v/config/node_config.cc Outdated Show resolved Hide resolved

src/v/redpanda/admin/server.h Outdated Show resolved Hide resolved

piyushredpanda added this to the v23.3.1-rc6 milestone Dec 21, 2023

oleiman added 3 commits December 21, 2023 08:38

node_config: Introduce verbose_logging_timeout_sec_max

194ab70

Max duration for TRACE or DEBUG level logging. If null (default) we impose no limit. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

admin: Introduce force parameter for PUT /v1/config/log_level

d15cd37

Allows setting an expiration that exceeds `verbose_logging_timeout_max`. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

rptest/admin: Admin.*_log_level improvements

88ffdea

- Adds get_log_level - Improve set_log_level - Allows setting expiration=0 - Adds force query parameter Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman force-pushed the issue/ci-938 branch from df50d3b to 3816a0f Compare December 21, 2023 16:38

oleiman requested a review from michael-redpanda December 21, 2023 16:40

michael-redpanda previously approved these changes Dec 21, 2023

View reviewed changes

src/v/redpanda/admin/server.h Outdated Show resolved Hide resolved

oleiman added 3 commits December 21, 2023 09:09

dt: Tests for verbose_logging_timeout_max

4d4c975

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman dismissed michael-redpanda’s stale review via 4d4c975 December 21, 2023 17:10

oleiman force-pushed the issue/ci-938 branch from 3816a0f to 4d4c975 Compare December 21, 2023 17:10

oleiman requested a review from michael-redpanda December 21, 2023 17:10

michael-redpanda approved these changes Dec 21, 2023

View reviewed changes

michael-redpanda merged commit 7b3096e into redpanda-data:dev Dec 21, 2023
21 checks passed

This was referenced Dec 21, 2023

[v23.3.x] verbose_logging_timeout_max #15856

Closed

[v23.2.x] verbose_logging_timeout_max #15857

Open

vbotbuildovich mentioned this pull request Dec 21, 2023

[v23.1.x] verbose_logging_timeout_max #15858

Closed

oleiman mentioned this pull request Dec 21, 2023

[v23.3.x] verbose_logging_timeout_max #15859

Merged

oleiman changed the title ~~verbose_logging_timeout_max~~ verbose_logging_timeout_sec_max Dec 21, 2023

renovate bot mentioned this pull request May 4, 2024

feat(github-release)!: Update redpanda-operator to v24.1.6 otosky/home-ops#1232

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

verbose_logging_timeout_sec_max #15798

verbose_logging_timeout_sec_max #15798

oleiman commented Dec 20, 2023 •

edited

vbotbuildovich commented Dec 20, 2023

michael-redpanda left a comment

oleiman commented Dec 20, 2023

oleiman commented Dec 20, 2023

michael-redpanda commented Dec 21, 2023

michael-redpanda left a comment

oleiman commented Dec 21, 2023

michael-redpanda left a comment

oleiman commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

verbose_logging_timeout_sec_max #15798

verbose_logging_timeout_sec_max #15798

Conversation

oleiman commented Dec 20, 2023 • edited

Backports Required

Release Notes

Improvements

vbotbuildovich commented Dec 20, 2023

michael-redpanda left a comment

Choose a reason for hiding this comment

oleiman commented Dec 20, 2023

oleiman commented Dec 20, 2023

michael-redpanda commented Dec 21, 2023

michael-redpanda left a comment

Choose a reason for hiding this comment

oleiman commented Dec 21, 2023

michael-redpanda left a comment

Choose a reason for hiding this comment

oleiman commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

vbotbuildovich commented Dec 21, 2023

oleiman commented Dec 20, 2023 •

edited