architecture.adoc

Architecture

This chapter describes the architectural overview of how metrics are setup, stored and exposed for consumption. This chapter also lists the various scopes of metrics.

See section required-metrics.adoc for more information regarding metrics that are required for each vendor.

See section app-programming-model.adoc for more information regarding the application metrics programming model.

Metrics Setup

Metrics that are exposed need to be configured in the server. On top of the pure metrics, metadata needs to be provided.

The following three sets of sub-resource (scopes) are exposed.

• base: metrics that all MicroProfile vendors have to provide

• vendor: vendor specific metrics (optional)

• application: application-specific metrics (optional)

Scopes

Required Base metrics

Required base metrics describe a set of metrics that all MicroProfile-compliant servers have to provide. Each vendor can implement the set-up of the metrics in the base scope in a vendor-specific way. The metrics can be hard coded into the server or read from a configuration file or supplied via the Java-API described in app-programming-model.adoc. The Appendix shows a possible data format for such a configuration. The configuration and set up of the base scope is thus an implementation detail and is not expected to be portable across vendors.

Section required-metrics.adoc lists the required metrics. This list also includes a few items marked as optional. These are listed here as they are dependent on the underlying JVM and not the server and thus fit better in base scope than the vendor one.

The implementation must tag base metrics with scope=base.

Required base metrics are exposed under /metrics?scope=base.

The base scope is used for, and only for, any metrics that are defined in MicroProfile specifications. Metrics in the base scope are intended to be portable between different MicroProfile-compatible runtimes at the same version.

Application metrics

Application specific metrics can not be baked into the server as they are supposed to be provided by the application at runtime. Therefore a Java API is provided. Application specific metrics are supposed to be portable to other MicroProfile implementations. That means that an application written to this specification which exposes metrics, can expose the same metrics on a different compliant server without change.

Details of this Java API are described in app-programming-model.adoc.

The implementation must automatically tag application metrics with scope=application, except where the metric already has a scope tag.

Metrics with application scope are exposed under /metrics?scope=application.

Applications may also define their own scope names by tagging metrics with another scope (other than base, vendor, application). Metrics with custom scopes are exposed under /metrics?scope=scope_name where scope_name is defined by the application. Scope names must match the regex [a-zA-Z_][a-zA-Z0-9_]*. If an illegal character is used, the implementation must throw an IllegalArgumentException.

Metrics with application-defined scopes are exposed under /metrics?scope=scope_name

Vendor specific Metrics

It is possible for MicroProfile server implementors to supply their specific metrics data on top of the basic set of required metrics.

Vendor specific metrics are exposed under /metrics?scope=vendor.

Examples for vendor specific data could be metrics like:

• OSGi statistics if the MicroProfile-enabled container internally runs on top of OSGi.

• Statistics of some internal caching modules

• Any other metrics that are generated by application frameworks, but not directly declared in application code, if these metrics are not based on any specification and therefore not expected to be portable between different runtimes that might support the same application framework.

Vendor specific metrics are not supposed to be portable between different implementations of MicroProfile servers, even if they are compliant with the same version of this specification.

Vendors are encouraged to use metric names consistent with the Open Telemetry Metrics Semantic Conventions where applicable.

Tags

Tags (also known as labels) play an important role in modern microservices and microservice scheduling systems (like e.g. Kubernetes). Application code can run on any node and can be re-scheduled to a different node at any time. Each container in such an environment gets its own ID; when the container is stopped and a new one started for the same image, it will get a different id. The classical mapping of host/node and application runtime on it, therefore no longer works.

Tags have taken over the role to, for example, identify an application (app=myShop), the tier inside the application (tier=database or tier=app_server) and also the node/container id. Metric value aggregation can then work over label queries (Give me the API hit count for app=myShop && tier=app_server).

In MicroProfile Metrics, tags add an additional dimension to metrics that share a common basis. For example, a metric named carCount can be further differentiated by the car type (sedan, SUV, coupe, and etc) and the colour (red, blue, white, black, and etc). Rather than incorporating this in the metric name, tags can be used to capture this information in separate metrics.

carCount{type=sedan,colour=red}
carCount{type=sedan,colour=blue}
carCount{type=suv,colour=red}
carCount{type=coupe,colour=blue}

For portability reasons, the key name for the tag must match the regex [a-zA-Z_][a-zA-Z0-9_]* (Ascii alphabet, numbers and underscore). If an illegal character is used, the implementation must throw an IllegalArgumentException. If a duplicate tag is used, the last occurrence of the tag is used.

The tags named _scope and _app are reserved. If an application attempts to create a metric with either of these tags, the implementation must throw an IllegalArgumentException.

The tag value may contain any UTF-8 encoded character.

Note	The REST endpoints provided by MicroProfile Metrics have different reserved characters based on the format. The characters are only escaped as needed when exposed through the REST endpoints. See rest-endpoints.adoc for more information on the reserved characters.

Tags can be supplied in two ways:

• At the level of a metric as described in app-programming-model.adoc.

• At the application server level by using MicroProfile Config and setting a configuration property of the name mp.metrics.tags. The implementation MUST make sure that an implementation of MicroProfile Config version at least 2.0 is available at runtime. If it is supplied as an environment variable rather than system property, it can be named MP_METRICS_TAGS and will be picked up too.

  • Tag values set through mp.metrics.tags MUST escape equal symbols = and commas , with a backslash \

Set up global tags via environment variable

export MP_METRICS_TAGS=app=shop,tier=integration,special=deli\=ver\,y

Global tags and tags registered with the metric are included in the output returned from the REST API.

Global tags MUST NOT be added to the MetricID objects. Global tags must be included in list of tags when metrics are exported.

Note

In application servers with multiple applications deployed, there is one reserved tag name: _app, which serves for distinguishing metrics from different applications and must not be used for any other purpose. For details, see section Usage of MicroProfile Metrics in application servers with multiple applications.

Metadata

Metadata can be specified for metrics in any scope. For base metrics, metadata must be provided by the implementation. Metadata is exposed by the REST handler.

Tip	While technically it is possible to expose metrics without (some) of the metadata, it helps tooling and also operators when correct metadata is provided, as this helps getting a context and an explanation of the metric.

The Metadata:

• name: The name of the metric.

• unit: a fixed set of string units

• type:

  • counter: a monotonically increasing numeric value (e.g. total number of requests received).

  • gauge: a metric that is sampled to obtain its value (e.g. cpu temperature or disk usage).

  • histogram: a metric which calculates the distribution of a value.

  • timer: a metric which aggregates timing durations and provides duration statistics.

• description (optional): A human readable description of the metric.

Metadata must not change over the lifetime of a process (i.e. it is not allowed to return the units as seconds in one retrieval and as hours in a subsequent one). The reason behind it is that e.g. a monitoring agent on Kubernetes may read the metadata once it sees the new container and store it. It may not periodically re-query the process for the metadata.

Important

In fact, metadata should not change during the life-time of the whole container image or an application, as all containers spawned from it will be "the same" and form part of an app, where it would be confusing in an overall view if the same metric has different metadata.

Metric Registry

The MetricRegistry stores the metrics and metadata information. There is one MetricRegistry instance for each of the predefined scopes listed in Scopes.

Metrics can be added to or retrieved from the registry either using the @Metric annotation (see Metrics Annotations) or using the MetricRegistry object directly.

A metric is uniquely identified by the MetricRegistry if the MetricID associated with the metric is unique. That is to say, there are no other metrics with the same combination of metric name and tags. However, all metrics of the same name must be of the same type and be identified by the same set of tag names otherwise an IllegalArgumentException will be thrown. This exception will be thrown during registration.

The metadata information is registered under a unique metric name and is immutable. All metrics of the same name must be registered with the same metadata information otherwise an "IllegalArgumentException" will be thrown. This exception will be thrown during registration.

MetricID

The MetricID consists of the metric’s name and tags (if supplied). This is used by the MetricRegistry to uniquely identify a metric and its corresponding metadata.

The MetricID:

• name: The name of the metric.

• tags (optional): A list of Tag objects. See also Tags.

Reusing Metrics

For metrics declared using annotations, it is allowed to reference one metric by multiple annotations. The prerequisite for this is that all such annotations must carry the same metadata and tag names. If multiple annotations declare the same metric but contain different metadata or tag names, an IllegalArgumentException must be thrown during startup.

Reusability does not apply to gauges though. The implementation must throw an IllegalArgumentException during startup if it detects multiple @Gauge annotations referring to the same gauge (with the same MetricID).

Example of reused counters

    @Counted(name = "countMe", absolute = true, tags={"tag1=value1"})
    public void countMeA() { }

    @Counted(name = "countMe", absolute = true, tags={"tag1=value1"})
    public void countMeB() { }

In the above examples both countMeA() and countMeB() will share a single Counter with registered name countMe and the same tags in application scope.

Metrics and CDI scopes

Depending on CDI bean scope, there may be multiple instances of the CDI bean created over the lifecycle of an application. In these cases, where multiple bean instances exist, only one instance of the corresponding metric will be created (per annotated method), and updates to that metric will be combined from all related invocations regardless of the bean instance where the invocation happens. For example, calls to a method annotated with @Counted will increase the value of the same counter no matter which bean instance is the one where the counted method is being invoked.

The only exception from this are gauges, which don’t support multiple instances of the underlying bean to be created, because in that case it would not be clear which instance should be used for obtaining the gauge value. For this reason, gauges should only be used with beans that create only one instance, in CDI terms this means @ApplicationScoped and @Singleton beans. The implementation may employ validation checks that throw an error eagerly when it is detected that there is a @Gauge on a bean that will probably have multiple instances.

Exposing metrics via REST API

Data is exposed via REST over HTTP under the /metrics base path in different data formats for GET requests:

• JSON format - used when the HTTP Accept header best matches application/json.

• OpenMetrics exposition format - used when the HTTP Accept header best matches application/openmetrics-text; version=1.0.0. Support for this format by implementations is optional.

• Prometheus text-based exposition format - used when the HTTP Accept header best matches text/plain; version=0.0.4. This format is also returned when no media type is requested (i.e. no Accept header is provided in the request)

Note	Implementations and/or future versions of this specification may allow for more export formats that are triggered by their specific media type. The Prometheus text-based exposition format will stay as fall-back.

Formats are detailed below.

Data access must honour the HTTP response codes, especially

• 200 for successful retrieval of an object

• 204 when retrieving a subtree that would exist, but has no content. E.g. when the application-specific subtree has no application specific metrics defined.

• 404 if a directly-addressed item does not exist. This may be a non-existing sub-tree or non-existing object

• 406 if the HTTP Accept Header in the request cannot be handled by the server.

• 500 to indicate that a request failed due to "bad health". The body SHOULD contain details if possible { "details": <text> }

The API MUST NOT return a 500 Internal Server Error code to represent a non-existing resource.

Table 1. Supported REST endpoints

Endpoint	Request Type	Supported Formats	Description
/metrics	GET	JSON, Prometheus, OpenMetrics	Returns all registered metrics
/metrics?scope=<scope_name>	GET	JSON, Prometheus, OpenMetrics	Returns metrics registered for the respective scope. Scopes are listed in Metrics Setup
/metrics?scope=<scope_name>&name=<metric_name>	GET	JSON, Prometheus, OpenMetrics	Returns metrics that match the metric name for the respective scope
/metrics	OPTIONS	JSON	Returns all registered metrics' metadata
/metrics?scope=<scope_name>	OPTIONS	JSON	Returns metrics' metadata registered for the respective scope. Scopes are listed in Metrics Setup
/metrics?scope=<scope_name>&name=<metric_name>	OPTIONS	JSON	Returns the metric’s metadata that matches the metric name for the respective scope

Note	The implementation must return a 406 response code if the request’s HTTP Accept header for an OPTIONS request does not match application/json.

Usage of MicroProfile Metrics in application servers with multiple applications

Even though multi-app servers are generally outside the scope of MicroProfile, this section describes recommendations how such application servers should behave if they want to support MicroProfile Metrics.

Metrics from all applications and scopes should be available under a single REST endpoint ending with /metrics similarly as in case of single-application deployments (microservices).

To help distinguish between metrics pertaining to each deployed application, a tag named _app should be added to each metric.

The value of the _app tag should be passed by the application server to the application via a MicroProfile Config property named mp.metrics.appName. It should be possible to override this value by bundling the file META-INF/microprofile-config.properties within the application archive and setting a custom value for the property mp.metrics.appName inside it.

It is allowed for application servers to choose to not add the _app tag at all. Implementations may differ in how they handle cases where metrics are registered with the same name from two or more applications running in the same server. This behavior is not expected to be portable across vendors.