Invalid connection pool Prometheus metric format (monitor, mpMetrics)

[sdaschner]

Open Liberty version 18.0.0.4 emits datasource metrics in the Prometheus format under /metrics, if the monitor-1.0, and mpMetrics-1.1 features are enabled.

However, the metrics for the default datasource violate the Prometheus format:

vendor:connectionpool_data_source[_default_data_source]_connection_manager_connection_handles
vendor:connectionpool_data_source[_default_data_source]_connection_manager_create_total
...

This produces an error when Prometheus attempts to scrape the metrics under /metrics:

invalid metric type "_default_data_source]_connection_manager_in_use_time_total_seconds gauge"

Tested with Open Liberty versions: 18.0.0.4-20181123-2100 and 18.0.0.4-20181217-2101.

The invalid metrics format renders the whole Prometheus target unusable, unfortunately.

[donbourne]

@sdaschner , thanks very much for reporting this.

Could you please cut/paste the info from your server.xml relevant to your datasource configuration as well?

[sdaschner]

Sure:

This is the server.xml of my example project; the database properties are injected from bootstrap.properties:

<?xml version="1.0" encoding="UTF-8"?>
<server description="Config example">

    <featureManager>
        <feature>javaee-8.0</feature>
        <feature>mpMetrics-1.1</feature>
        <feature>monitor-1.0</feature>
    </featureManager>

    <!-- Postgres config-example-db definition -->
    <dataSource id="DefaultDataSource" jdbcDriverRef="postgresql-driver"
                type="javax.sql.ConnectionPoolDataSource" transactional="true">

        <properties serverName="config-example-db" portNumber="5432"
                    databaseName="postgres"
                    user="${config-example.db.username}"
                    password="${config-example.db.password}"/>

    </dataSource>

    <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="9080" httpsPort="9443"/>
    <keyStore id="defaultKeyStore" password="Liberty"/>

    <basicRegistry id="basic" realm="MicroProfileMetrics">
        <user name="admin" password="admin"/>
        <user name="nonadmin" password="guest"/>
    </basicRegistry>

    <administrator-role>
        <user>admin</user>
    </administrator-role>

</server>

And a file postgres-driver.xml that resides under /defaults/ and sets up the driver:

<?xml version="1.0" encoding="UTF-8"?>
<server description="OpenLiberty Java EE 8 Server">

    <jdbcDriver id="postgresql-driver"
                javax.sql.XADataSource="org.postgresql.xa.PGXADataSource"
                javax.sql.ConnectionPoolDataSource="org.postgresql.ds.PGConnectionPoolDataSource"
                libraryRef="postgresql-library"/>

    <library id="postgresql-library">
        <fileset id="PostgreSQLFileset" dir="/opt/wlp/lib"
                 includes="postgresql-9.4.1212.jar"/>
    </library>

</server>

[fwji]

The cause of the issue is the datasource with id 'DefaultDataSource''s auto-generated jndi name. It gets normalized to data_source[_default_data_source]_connection_manager in Prometheus, which broke the Prometheus format. We need to do some sanitizations of the jndi name to make sure the output will be in compliance with Prometheus format. A temporary workaround is to specify a jndiName field when configuring datasource in the server.xml. For example, the following datasource config will work

<dataSource id="ds2" jdbcDriverRef="DerbyEmbedded" jndiName="jdbc/exampleDS2">
	<properties createDatabase="create" databaseName="exampleDS2"/>
	<connectionManager maxIdleTime="1s" maxPoolSize="100" minPoolSize="1" reapTime="2s"/>
</dataSource>

Thanks for catching the issue as it's a scenario missed from our tests. We'll target the fix for 19.0.0.1

[sdaschner]

Thanks for the quick response! Yes, the workaround solves this.