Move health indicator logging from indicators to a central location

[philwebb]

See #43492 for background.

We'd like to centralize the logging when a health indicator fails so that it can be changed easily. This will involve:

• Removing the existing logging from AbstractHealthIndicator and AbstractReactiveHealthIndicator (the logExceptionIfPresent method).
• Updating Health so that the exception can be obtained.
• Centralizing logging to the code the calls the health indicators.

[josephabonasara]

Here is a PR with possible fix: PR 43595

[marcusvoltolim]

My solution to this might help with brainstorming:

import java.util.List;
import java.util.Map;
import java.util.Optional;
import com.fasterxml.jackson.annotation.JsonGetter;
import com.github.seratch.jslack.api.model.Field;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.reflect.FieldUtils;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.actuate.autoconfigure.health.HealthEndpointProperties;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthComponent;
import org.springframework.boot.actuate.health.HealthEndpointGroups;
import org.springframework.boot.actuate.health.ReactiveHealthContributor;
import org.springframework.boot.actuate.health.ReactiveHealthContributorRegistry;
import org.springframework.boot.actuate.health.ReactiveHealthEndpointWebExtension;
import org.springframework.boot.actuate.health.Status;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.autoconfigure.condition.ConditionalOnWebApplication;
import org.springframework.stereotype.Component;
import reactor.core.publisher.Mono;

import static org.springframework.boot.autoconfigure.condition.ConditionalOnWebApplication.Type.REACTIVE;

@Slf4j
@ConditionalOnProperty("management.health.custom-extension.enabled")
@ConditionalOnWebApplication(type = REACTIVE)
@Component
public final class CustomReactiveHealthEndpointWebExtension extends ReactiveHealthEndpointWebExtension {

    private final Optional<MultiLogger> splunkLoggerOpt;
    private final String slackUrl;
    private final String hostname;

    CustomReactiveHealthEndpointWebExtension(final ReactiveHealthContributorRegistry registry,
                                              final HealthEndpointGroups groups,
                                              final HealthEndpointProperties properties,
                                              final Optional<MultiLogger> splunkLoggerOpt,
                                              final @Value("${management.health.custom-extension.slack-url:}") String slackUrl,
                                              final @Value("${management.health.custom-extension.hostname:}") String hostname) {
        super(registry, groups, properties.getLogging().getSlowIndicatorThreshold());
        this.splunkLoggerOpt = splunkLoggerOpt;
        this.slackUrl = slackUrl;
        this.hostname = hostname;
    }

    @Override
    protected Mono<? extends HealthComponent> getHealth(final ReactiveHealthContributor contributor, final boolean includeDetails) {
        return super.getHealth(contributor, includeDetails)
            .doOnNext(healthComponent -> {
                if (healthComponent.getStatus() != Status.UP) {
                    var healthErrorLog = new HealthErrorLog(contributor, healthComponent);

                    splunkLoggerOpt.ifPresentOrElse(
                        multiLogger -> multiLogger.error(healthErrorLog, null),
                        () -> log.error("Health indicator failed, details: {}", healthErrorLog)
                    );
                    if (!slackUrl.isBlank()) {
                        SlackUtil.send(slackUrl, SLACK_ALERT_COLOR, "HEALTH ALERT - " + hostname, "Health indicator failed!", healthErrorLog.buildSlackFields());
                    }
                }
            });
    }

    private record HealthErrorLog(@JsonGetter("_type") String type, String name, String status, Map<?, ?> details) {

        private static final String LOG_TYPE = "health-error";

        private HealthErrorLog(final ReactiveHealthContributor contributor, final HealthComponent healthComponent) {
            this(LOG_TYPE, getHealthIndicatorName(contributor), healthComponent.getStatus().toString(), healthComponent instanceof Health health ? health.getDetails() : Map.of());
        }

        public List<Field> buildSlackFields() {
            return List.of(
                new Field("name", name, true),
                new Field("status", status, true),
                new Field("details", details.toString(), false)
            );
        }

        /**
         * Workaround to get name from package-private class
         *
         * @see org.springframework.boot.actuate.health.HealthIndicatorReactiveAdapter
         * @see org.springframework.boot.actuate.health.CompositeHealthContributorReactiveAdapter
         */
        private static String getHealthIndicatorName(final ReactiveHealthContributor contributor) {
            try {
                var delegate = FieldUtils.readField(contributor, "delegate", true);
                return delegate.getClass().getSimpleName();
            } catch (Exception ignored) {
                return contributor.getClass().getSimpleName();
            }
        }

    }

}

Something that would help a lot and would be very simple to change, is to pass the name parameter in the method below, this would make it easier to overwrite and avoid the use of Reflection to obtain the name of the HealthIndicatorReactiveAdapter

org.springframework.boot.actuate.health.HealthEndpointSupport

protected abstract T getHealth(C contributor, boolean includeDetails);