SkyWalking v10 - Service Hierarchy Relationship, the ownership cross layers

[wu-sheng]

SkyWalking v10 and eBPF Rover Next

SkyWalking v9 started in Dec 2021, and was first released on Apr 9th, 2022. It has been 6 released versions (9.0 - 9.6) and one more should be released soon(9.7.0).
Here is my v9 core proposal, which introduced the layer concepts and brought SkyWalking into the current full-stack APM monitoring.
After 9.7, we have built the whole UI, new Query protocol(v2), and query MQE(metric v3 and alert kernel), as well as good integration with Prometheus, OTEL, Loki, Telegraf, etc.
The data sources of SkyWalking APM have covered traditional agent installed service, VMs, cloud infra, k8s, etc.

For example, a Java service is built in a docker image and is going to be deployed in a k8s cluster, with a sidecar injected due to service mesh managed. The following services would be able to detect cross-layers

1. Java service, detected as Java agent installed.
2. A pod of k8s service is detected, due to k8s layer monitoring.
3. Side car perspective service is detected.
4. VM Linux monitoring for a general process, as the container of Java service is deployed on this specific k8s node.
5. Virtual databases, caches, and queues conjectured by agents, and also monitored through k8s monitoring, even traffic monitored by service mesh.

All these services have logic connections or are identical from a physical perspective, but currently, they may be just similar on name(s), no further metadata connection.

By those, we have a chance to move one step ahead to connect the dots of the whole infrastructure. This means, for the first time, we are going to establish the connections among services detected from various layers.

In the v10, I am proposing a new concept Service Hierarchy. Service Hierarchy defines the relationships of existing services in various layers. With more kinds of agent tech involved(such as eBPF) and deployment tools(such as operator and agent injector), we could inject relative service/instance metadata and try to build the connections, including,

• Agent injector injects the pod ID into the system env, then Java agent could report the relationship through system properties.
• Rover(eBPF agent) reveals its next iteration forward k8s monitoring rather than profiling. And add the capabilities to establish connections among k8s pods and service mesh srv.

Meanwhile, as usual with the new major version change, I would expect UI side changes as well. UI should have flexible capabilities to show hierarchy services from the service view and topology view. Also, we could consider a deeper view of the instance part as well.

About the protocol, there should be no breaking changes, but enhancements only. New query protocols(service-hierarchy and instance-hierarchy) are considered to be added, some new fields should be added on things like topology query and instance dependencies to list relative services/instances from other layers directly rather than an extra query.

[wu-sheng]

@kai @mrproliu and I discussed this for the first time in the pass two weeks. I am writing this officially, and trying to bring all @apache/skywalking-committers up to speed.

[wu-sheng]

@Fine0830 For the layer graph, I think this style is a good reference, which is from previous v10 planning

[wu-sheng]

After the 9.7 release is cut, this would be tracked at #11615

[trungtien]

When will Skywalking version 10 be released? I'm really looking forward to this new version. :D

[wu-sheng]

Next month.(May. 2024)

[wu-sheng]

Most are done. Unless you want to try BanyanDB 0.6, the current latest will be the release of v10.