Keycloak won't pick up metrics-listener

[torbjornsk]

Description

I'm trying to deploy the metrics spi to Keycloak 10.0.1 using Docker. I have tried multiple ways of deploying (copy file to standalone/deployments, deploy-command from cli, module add from cli), but it doesn't seem to work as expected. The logs state that the jar is deployed and everything is fine, but still metrics-listener does not appear in the provider list. To make things even more strange, the "metrics" provider appears, along with the metrics endpoint. So it seems only parts of the content of the jar is deployed, with the logs giving no indication of anything being wrong while deploying.

I'm quite stuck at this point, and close to giving up on the whole thing. Any points as to where/how to debug this?

[nicopadu]

We fixed this in our case by updating realm file imported by Docker:

    "eventsEnabled": true,
    "eventsListeners": ["jboss-logging", "metrics-listener"],
    "enabledEventTypes": [],
    "adminEventsEnabled": true,

[torbjornsk]

I'm not sure how that would help, as the metrics-listener provider is never registered in the first place.

[nicopadu]

After adding these 2 files in Docker image, when it initializes metrics-listener is registered.

/opt/jboss/keycloak/standalone/deployments/keycloak-metrics-spi-2.1.0.jar
/opt/jboss/keycloak/standalone/deployments/keycloak-metrics-spi-2.1.0.jar.dodeploy

We turn it on by using our modified realm file, also provided in docker image

[torbjornsk]

I tried adding this to my Dockerfile:

COPY keycloak-metrics-spi-2.1.0.jar /opt/jboss/keycloak/standalone/deployments/keycloak-metrics-spi-2.1.0.jar
RUN touch /opt/jboss/keycloak/standalone/deployments/keycloak-metrics-spi-2.1.0.jar.dodeploy

Logs state this:

14:58:30,836 INFO [org.jboss.as.server] (ServerService Thread Pool -- 35) WFLYSRV0010: Deployed "keycloak-metrics-spi-2.1.0.jar" (runtime-name : "keycloak-metrics-spi-2.1.0.jar")

But still, there is no provider for metrics-listener defined when I check the admin gui. Logs also states this when trying to use the provider:

14:58:48,029 ERROR [org.keycloak.events.EventBuilder] (default task-2) Event listener 'metrics-listener' registered, but provider not found

[leesplk]

I think this is actually a bug in the keycloak deployment code. The provider is supposed to be populated in KeycloakProviderDependencyProcessor. getKeycloakProviderDeploymentInfo(), but it just ... doesn't.

[leesplk]

Gah, spoke too soon. Its slightly more complicated. By the docs, metrics-spi is missing a required file "META-INF/services/org.keycloak.provider.Spi" as described in the keycloak server docs.

However, that still doesn't work. Near as I can tell, the solution is in this discourse thread

I don't fully understand how this is supposed to work, but at least some fault is on this provider.

[NCrustand]

@leesplk or @torbjornsk: Did you find a solution to this problem? I'm experiencing the same thing. The jar is picked up and deployed, and the endpoint is working, but I get the Event listener 'metrics-listener' registered, but provider not found and have no Keycloak metrics. I tried taking a look at the discourse link provided above, but couldn't figure anything out.

[leesplk]

yes, sort of. Its complicated to describe the underlying problem (ie, i'm not sure my diagnosis is 100% correct) but my discovery was that enabling the builtin jboss-logging SPI prevented the registration of any subsequent event listener. My solution was:

1. not enable jboss-logging event listener
2. steal the jboss-logging id and use it for this plugin (we started with this plugin and augmented it, but from the current context its the static ID - just change it from "metrics-listener" to "jboss-logging".

The advantage to stealing the jboss-logging id is that keycloak will automatically attach that specific EventListener to new realms (i don't think this is documented, but i've verified its true by code inspection and practical experience).

[ae-govau]

I'm running into what I think is the same issue with Keycloak 16.1.0 and Keycloak-Metrics-SPI 2.5.3.

I get the metrics page, but nothing except JVM metrics. In the logs I see the same message as @NCrustand , Event listener 'metrics-listener' registered, but provider not found.

We also do have jboss-logging enabled.

Am I to understand that we can have logging, or we can have metrics, but we can't have both?

[ae-govau]

I think I've managed to figure out the issue. When we enabled jboss-logging to output useful logging events, I searched the internet and found this post:
https://lists.jboss.org/pipermail/keycloak-user/2017-February/009498.html

From that post:

# Configure jboss-logging event listener
/subsystem=keycloak-server/spi=eventsListener:add(default-provider=jboss-logging)
/subsystem=keycloak-server/spi=eventsListener/provider=jboss-logging:add(enabled=true)
# Propgate success events to INFO instead of DEBUG
# This allows to track successful logins in log analysis
/subsystem=keycloak-server/spi=eventsListener/provider=jboss-logging:write-attribute(name=properties.success-level,value=info)
/subsystem=keycloak-server/spi=eventsListener/provider=jboss-logging:write-attribute(name=properties.error-level,value=warn)

We copy/pasted that, and it had been working fine. However today while debugging this issue I was reading up on SPI and read that:

Here we have two providers defined for the SPI myspi. The default-provider is listed as myprovider. However it is up to the SPI to decide how it will treat this setting. Some SPIs allow more than one provider and some do not. So default-provider can help the SPI to choose.

(https://access.redhat.com/documentation/en-us/red_hat_single_sign-on/7.2/html/server_installation_and_configuration_guide/manage_subsystem_configuration#config_spi_providers)

Thus it seemed plausible that defining default-provider (of jboss-logging) above might make an SPI decide to only use that provider instead of all. Sure enough, removing the <default-provider>jboss-logging</default-provider> from our XML seems to have fixed our issue.

I updated our pre-start script to the following and now everything seems to be happy:

# delete old eventsListener as it may have had default-provider set
/subsystem=keycloak-server/spi=eventsListener:remove
/subsystem=keycloak-server/spi=eventsListener:add

# add metrics listener
/subsystem=keycloak-server/spi=eventsListener/provider=metrics-listener:add(enabled=true)

# add logging listener
/subsystem=keycloak-server/spi=eventsListener/provider=jboss-logging:add(enabled=true)
/subsystem=keycloak-server/spi=eventsListener/provider=jboss-logging:write-attribute(name=properties.success-level,value=info)
/subsystem=keycloak-server/spi=eventsListener/provider=jboss-logging:write-attribute(name=properties.error-level,value=warn)

We now get both events written to logs, and metrics.