Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable log4j2 shutdown hook [HZ-3543] #26006

Closed

Commits on Nov 20, 2023

  1. Disable log4j2 shutdown hook

    A common issue, especially when Hazelcast runs in Kubernetes pods with log4j2 logging,
    is that shutdown logs are missing. The reason is that log4j2 registers its own shutdown
    hook, alongside Hazelcast's own shutdown hook. Since the JVM does not guarantee
    any ordering of shutdown hook execution, it can be the case that the log4j shutdown
    hook shuts down loggers while Hazelcast is still performing shut down activities.
    This results in missing important logs from the shutdown of Hazelcast members that
    can be useful for troubleshooting.
    This change introduces explicit shutdown of the log4j2 LogManager after Hazelcast Node
    has been shut down: this does no harm when the log4j shutdown hook is enabled (as far as
    I navigated the log4j code & tested the changes locally). When the shutdown hook is disabled,
    it allows for clean logging of shutdown information.
    Example of Hazelcast shutdown logging with GRACEFUL shutdown hook policy:
    - without this change:
     ^C2023-11-20 16:26:02,988 [ INFO] [hz.ShutdownThread] [c.h.i.i.Node]: [127.0.0.1]:5701 [dev] [5.4.0-SNAPSHOT] Running shutdown hook... Current node state: ACTIVE
     - with this change:
     ^C2023-11-20 16:26:31,183 [ INFO] [hz.ShutdownThread] [c.h.i.i.Node]: [127.0.0.1]:5701 [dev] [5.4.0-SNAPSHOT] Running shutdown hook... Current node state: ACTIVE
     2023-11-20 16:26:31,185 [ INFO] [hz.ShutdownThread] [c.h.c.LifecycleService]: [127.0.0.1]:5701 [dev] [5.4.0-SNAPSHOT] [127.0.0.1]:5701 is SHUTTING_DOWN
     2023-11-20 16:26:31,191 [ INFO] [hz.ShutdownThread] [c.h.i.i.Node]: [127.0.0.1]:5701 [dev] [5.4.0-SNAPSHOT] Shutting down connection manager...
     2023-11-20 16:26:31,195 [ INFO] [hz.ShutdownThread] [c.h.i.i.Node]: [127.0.0.1]:5701 [dev] [5.4.0-SNAPSHOT] Shutting down node engine...
     2023-11-20 16:26:31,205 [ INFO] [hz.ShutdownThread] [c.h.i.i.NodeExtension]: [127.0.0.1]:5701 [dev] [5.4.0-SNAPSHOT] Destroying node NodeExtension.
     2023-11-20 16:26:31,206 [ INFO] [hz.ShutdownThread] [c.h.i.i.Node]: [127.0.0.1]:5701 [dev] [5.4.0-SNAPSHOT] Hazelcast Shutdown is completed in 18 ms.
    vbekiaris committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    c95a423 View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2023

  1. Adds cluster property to control logger shutdown

    Default is false, so Hazelcast will not explicitly shut down the logging
    framework during graceful shutdown. When true, Hazelcast will
    shutdown the backing logging framework as part of its graceful
    shutdown procedure.
    vbekiaris committed Dec 19, 2023
    Configuration menu
    Copy the full SHA
    5c20529 View commit details
    Browse the repository at this point in the history