OpenTelemetry / Observability (traces, metrics)

[mySupply-JEB]

This is meant to serve as a broad opening idea to implement some tracing and metrics calculation + exposing these to/from fx. Prometheus, Jaeger, etc. collector which utilizes the OTLP-standard (https://opentelemetry.io/) for observability within the applications that utilize the Phase4-stack.

What I would like to have implemented

It would be nice to be able to acquire data on the application's state over time, e.g.:

• When a document has been received (successfully + unsuccessfully)
• When a document has been sent (successfully + unsuccessfully)
• Size of incoming request(s)
• Size of outgoing request(s)
• Ingoing + outgoing document types, all relevant identifiers for the participants involving a document (C1 -> C4)
• Does the application have contact to it's database(s)
• Length (timewise) of request from C2 -> C3
• ... and likely many more things, feel free to add more

This is essentially a feature request with a quite broad scope but this would definitely be a nice enhancement for (I believe most of) those who are running their services in a somewhat public cloud setup, particularly for those who run containers. OpenTelemetry collectors can also be installed on dedicated servers (in non-public cloud environments) so this could be of use to anyone running Phase4 in some capacity.

SDK to utilize for implementation

I believe this could for example be implemented by using the Prometheus SDK for Java.

What do I wish to achieve

This would help developers and administrators in observing and diagnosing issues by being able to make however complicated you want queries for telemetric data - hence also avoiding the possible risk of attempting to gather such data/statistics based on an already production-running Phase4 database. Overall, "observability" is just a must in larger-scale environments that different enterprises likely have and is a "cloud-first" mindset which more and more services adhere to or at least offers to some degree.

[mySupply-JEB]

It would also be really nice if logs could be configured for structured logging, e.g. printing log lines that become very machine-read friendly so that these also can be queried on specific values for a parameterized entry in the log-line.

[phax]

https://github.com/phax/ph-telemetry as the basis for this - stay tuned