Channel metrics refactor

[barspi]

This PR fixes a few issues with the current genration of metrics counters and metrics events for the "channel set": BaseChannel, ChannelAdaptor, QServer etc.
At the same time, the previously hardcoded msg counter metrics, are now configurable by means of a new interface and class:

public class ISOMsgCounter implements ISOMsgMetrics

A thorough ISOMsgCounterTest is included.

The PR includes some features:

• BaseChannel has a single reference to an ISOMsgCounter which wraps channel metrics configuration and recording.
  The old references to specific Counter msg[In|Out]Counter were not appropriate for metrics that vary their tag values (e.g. a new mti="xyz") because it's effectively a new counter in Micrometer, that was overridding the old reference, and in the case of QServer that reference was always null. This also a bit incorrect when removing meters because only the most-recent one, at best. would be removed.

• The metrics are now generated at one point of code only: the BaseChannel class in its send() and receive() methods.

• The default, zero-config option behaves the same as before, generating simple counter metrics like
  jpos_isomsg_total{name="some-channel",direction="in",mti="2110",type="client"} 6802.0
  So if you do nothing with your current system, it will keep working as it is already doing.

• But now, explicit configurable counters can be attached to a channel, either programmatically or through xml in the <channel> configuration of ChannelAdaptor and QServer.

• The ChannelAdaptor#newChannel() parser was leveraged to recognize the new <metrics> element. This is useful because newChannel() is used by both ChannelAdaptor and QServer so it's a single point of code to do this.

• Global defaults for valid tags, and tag values taken from isofields (read more below). These global tags are configurable through env variables, and the defaults are the zero-config ones maitining current behavior.

• ISOMsgCounter keep track of all their counters and unregisters (remove) them correctly upon request, such as when the ChannelAdaptor or QServer are being stopped and undeployed.

Default tags

Prometheus mandates that all metrics of the same name must have the same set of tags/labels.
If someone attempts to create/register a new Meter with a pre-existing name, but with a different set of tag keys, the returned Meter will work, but it will "almost silently" fail to register for scraping by Prometheus. It's an issue very difficult to detect programmatically, and it was fixed in PR #671 by throwing an exception.

(Note: the invariant refers to the tag keys, not their values which can obviously vary)

Still, some effort was done into having "global default tags" that all channel counters can automatically and safely use. At the same time, they can be configured through env vars, and also custom counter names can be used, so that a specific channel using a custom counter name (different from the default jpos.isomsg) is free to use its own set of tags.

The env variables, and their defaults are

metrics.channel.tags="name, type, direction"
metrics.channel.fields="mti"

These refer just to the tag key, some of the values will be set dynamically for every ISOMSg recorded.
The fields tags are supposed to take their values from specific ISOMsg fields with a syntax explained below.

Configuration

There are several ways to configure msg counter metrics in a channel through XML.

1) do nothing

<channel class=..... >
    ... normal stuff ....
</channel>

This is the zero-config version. Your current configurations will continue to generate counters as explained above.

2) "none"

<channel class=..... >
    ... normal stuff ....
    <metrics type="none" />
</channel>

This effectively turns off isomsg counting for that channel, maybe saving some CPU cycles, Prometheus scraping, etc.

3) Specific "class"

<channel class=..... >
    ... normal stuff ....
    <metrics class="my.own.ISOMsgMetricsImpl">
        ... custom config ....
    </metrics>
</channel>

In case you want to get creative with your counters 😉

4) Customizable "counter"

<channel class=..... >
    <metrics type="counter"  logger="Q2">
        <property name="tags"  value="my_tag:my_value" />
        <property name="fields" value="itc, rc:39, scheme:113.66" />
    </metrics>
</channel>

This "counter" type will use the ISOMsgCounter implementation of ISOMsgMetrics.
It will generate metrics with the specific Prometheus labels:

• my_tag="my_value" -- static value in this case
• itc="2100.00" -- itc is an alias that computes MTI + ProcessingCode(txn type) + other optionals like function and reason code)
• rc="0000" -- rc is an alias for the value of field 39, but in this case I used explicit syntax)
• scheme="visa" -- generate the scheme tag, with its value taken from field 113.66 in case of jPOS-CMF

Now, of course this will fail with ConfigurationException because none of those tags are "recognized" in the default values for metrics.channel.tags and metrics.channel.fields.
So, if you want to use custom tags for your metrics, you have two options
1)you must configure those variables with the desired tags and isofield tags, and they will become globally available.
2) You can give your particular counter a different name such as

<channel class=..... >
    <metrics type="counter"  logger="Q2">
        <property name="name"  value="my_own_counter" />
        <property name="description"  value="This is my counter" />
        <property name="tags"  value="my_tag:my_value" />
        <property name="fields" value="ttt:itc, rc" />
    </metrics>
</channel>

And you will get Prometheus serving such as

# HELP my_own_counter_total  This is my counter
# TYPE my_own_counter_total counter
my_own_counter_total{direction="in",type="client",name="my-channel",ttt="2110.00",rc="0000"} 85713.0

A couple of notes:

• tags direction, type, and name will always be added because they are donde by the internal channel configuration. Even if you configure them in the XML such as direction:up the value will be overwritten when the metric is recorded.
• Since I gave my counter its own unique name, I can specify any other keys I want, disregarding the global defaults from the env.
• Notice how in this version I chose to give just the alias rc without making field 39 explicit. Also notice how I used the expression ttt:itc instead of just itc. This means "take the value using alias itc for the isomsg field logic, but give it the tag name ttt.

Final notes

Grafana dashboards and operational assumptions

The standard jPOS Grafana panels assume the default metrics configuration provided by jPOS, including metric names and label keys. Although customers are free to customize or extend metric generation, the jPOS team relies on the default counters and labels for baseline system monitoring, troubleshooting, and support. For this reason, we recommend preserving the default metrics and labels, even when additional custom metrics are introduced.

Metrics cardinality

Care must be taken to keep metric cardinality under control. Adding high-cardinality labels can significantly impact Prometheus performance and resource usage. The default jPOS metrics are designed to remain low-cardinality, and any customization should preserve this property whenever possible.

[ar]

If possible, don't use Environment directly, go through Configuration.