Support AI/GPT plugins for specific scenarios

[wu-sheng]

With the quick raising of ChatGPT and other GPT4 models, I have run several tests in the ChatGPT to resolve several typical scenarios we knew for a long time in the APM field.

Restful API Formatting

Restful APIs usually includes parameter variable as a part of it, such as /product/{id}. Ideally, with OpenAPI Spec, SkyWalking could format those correctly, or with agent plugins(such as Java SpringMVC plugin), SkyWalking reached the codes directly with this metadata. But in many other cases, dev team don't have a proper definition of OpenAPI, and/or they don't install the agent such as in the service mesh case. The parameters are lost.

I tried this in the ChatGPT,

Q: Could you summary the parterns from following URIs? /products/1 , /products/2, /products/5, /products/2100, /products/2323, /file/2

ChatGPT:

Based on the provided URIs, here is a summary of the patterns observed:

/products/1: This URI pattern suggests a product resource with an ID of 1. It is likely that the number after "products/" represents a unique identifier for a specific product.

/products/2: Similar to the previous URI, this pattern represents a different product resource with an ID of 2.

/products/5: Following the same pattern, this URI indicates yet another product resource with an ID of 5.

/products/2100: Continuing the pattern, this URI suggests a product resource with a higher ID of 2100.

/products/2323: Similarly, this URI pattern represents a product resource with a unique ID of 2323.

/file/2: This URI pattern, distinct from the product resources, suggests a file resource with an ID of 2. It indicates that the API or web service may handle files separately from products.

In summary, the provided URIs appear to follow a consistent pattern where "/products/" is followed by a numeric ID to identify specific product resources. Additionally, there is a separate URI pattern for handling file resources, indicated by the "/file/" prefix followed by a numeric ID.

Q: If {var} could represent number and string in the patterns, could you make the pattern expressions?

ChatGPT

Certainly! If we consider that {var} can represent both numbers and strings in the patterns, here are the corresponding pattern expressions:

/products/{var} - This pattern expression represents a product resource where {var} can be replaced with any alphanumeric value, representing the ID of the product. It allows for both numbers and strings.

/file/{var} - This pattern expression represents a file resource where {var} can be replaced with any alphanumeric value, indicating the ID of the file. Again, it accommodates both numbers and strings.

Using these pattern expressions, you can match and handle a wide range of product and file resources by substituting {var} with the appropriate identifier, whether it's a number or a string.

You can see that, it has a perfect conclusion. We could dig deeper such as asking which algorithm it was using for recognizing those patterns.

Alerting Baseline Calculation

There was an issue proposed from jd.com about this very long time ago. It didn't work out, but we could have directly input/output from ChatGPT. We could ask about

Which algorithms are good to build the base time for alerting about the system HTTP success rate? Considering the different payloads between morning and afternoon.

Could you simulate the metrics of a HTTP server covering 7 days. In each day, you build 24 hours metrics, one value per minute. The significant peaks should be at Wed. 8am to 10am.

Could you run this program, and use STL to find the peak of the output?

We could use this way to quick choose the suitable algorithm, even finishing the codes to build the interaction.

Ideas

The AI(GPT models) has shown a significant positive impact on the APM industry. SkyWalking community should follow up on this, and adopt them. After some discussions with AIOps team(China and US), one of the important things to get this works on is that SkyWalking should provide a very well context and data format for the AI engine to understand.
I am going to build an AI-adopting module in the OAP, which would consume data from the kernel with a good understanding, such as service success rate with service metadata and specific time range, periodically endpoint grouping mechanism.

Make the AI/ChatGPT integration totally as an add-on, and always feed certain formats with well design mechanisms. Don't overthink the capabilities of AI. Human and/or AI are only good at context-based analysis. At least for now.

[Superskyyy]

Hi! Excited to see the LLM adventure starts given the rise of chatgpt!

After reading the proposal, I have few points/questions that could augment or maybe trigger some further discussions. @wu-sheng

For the first part of the url pattern matching case, are we saying that we propose some plugins that rely on the response of LLMs to get understanding of the endpoints OR is it that we build something that could do the clustering given the LLM's suggestion on algortihms? For both cases I can offer some suggestions[I will add them later]. (the algorithm sounds almost identical to log clustering but in a simplified way (given token boundry is defined by / and count is bound to a relatively small number, we can implement refering to the Drain algorithm that we use, which is essentially via an incremental decision tree with some tricks)

For the second part of metrics alerting baseline, this is the tricky part about LLMs like gpt4. They are not able to do reasoning for time series data analysis. Given the current technical limitation of transformer architecture, they are naturally incapable of doing long context reasoning especially time series data usually have large amount of data points and they cannot do it real time (incremental analysis). As a matter of fact, over 10 metrics algortihms have been evaluated on our side (aiops engine project) since last year, since we failed to use Ray last year, the new effort of integrating both log clustering and metrics forcasting into flink is already underway which is able to do near-real time feedback on multi-dimensional metrics with widely used algorithms. Although I think a lightweight module (STL for example, which essentially just removes all outliers) should also be added in the oap side as a way to implement automatic baselines)

Now, what if we want to try actually rely on LLMs? The problem I see is the great security risks of feeding data to LLMs, so users would want a local deployment of AI like below. (results are, as expected, not always reliable)
Below screenshot uses a locally deployed LLM 13billion parameter Vicuna model in a CPU only server, which acheives good enough results like chatgpt, but without sending any private data to third party.

[wu-sheng]

Hi, @Superskyyy What I am proposing is not which way to, is how to do within workflow.
As the ideas listing, for us, it is important to ship data in good context. Others are pluggable.

[wu-sheng]

The reason I said gpt/ai rather than ChatGPT is that, AI never would be our expertise.
I knew you are running AI engines for a while, it is still open ideas to adopt that, but with robust industry level solution is out, it hasn't to be built by users themselves.

[Superskyyy]

Hi, @Superskyyy What I am proposing is not which way to, is how to do within workflow. As the ideas listing, for us, it is important to ship data in good context. Others are pluggable.

Thanks for the explanation, I get it now. Indeed we need a well-defined mechanism to expose data for ingestion into pluggable AI features.

[wu-sheng]

The thing here, I want to push things moving fast, to benefit our users.
Industry level solution preferences are also a good direction for us too.
So, if your side is ready, please promote them how to use.

I would like to continue on how to ship formatted data to AI, rather than existing exporters for metrics. Make sense? @Superskyyy

[Superskyyy]

The thing here, I want to push things moving fast, to benefit our users. Industry level solution preferences are also a good direction for us too. So, if your side is ready, please promote them how to use.

I would like to continue on how to ship formatted data to AI, rather than existing exporters for metrics. Make sense? @Superskyyy

Makes sense, is there a reference link that I can take a look at what other apm vendors are doing about it?

The formatted data would benefit my side too, currently it is relying on a bunch of post-processing, if we can just package them in a better way before shipping that would be of great benefit.

[Superskyyy]

Do we plan to also define an interface for getting results back from AI plugins? My side can then convert accordingly.

[wu-sheng]

Yes, we are. That is also the scope of this. We would open APIs of typical scenarios through ai-pipeline module with codes and gRPC to provide data feeding and feedback in async.

[wu-sheng]

Makes sense, is there a reference link that I can take a look at what other apm vendors are doing about it?

I am afraid there isn't. Because, there's any oter open source apm doing so complex integrations.

I am going to write one for sure, if that works for you. It could be done in these weeks.

[Superskyyy]

That is awsome! Can't wait to see this, I have some pipelines that can directly plug-into this as poc once api is ready.

[wu-sheng]

@Superskyyy When is your side work good enough to represent? I think I didn't heart of updates for months just knowing there are things happening from there.

[Superskyyy]

@Superskyyy When is your side work good enough to represent? I think I didn't heart of updates for months just knowing there are things happening from there.

Algorithms are implemented outside of the aiops repo, one maintained by last year's ospp phd student for metrics anomaly detection and I took control of the upstream log clustering algorithm. https://github.com/Fengrui-Liu/StreamAD and https://github.com/logpai/Drain3. We didn't update the main repo since our ray integration was broken after trying.

Since I was busy with research papers for a while and the phd last year also was busy finding a job, we paused for few months in terms of online collaboration but instead bringing more people to collaboration in private.

We currently have 5 people working together on the AIOps project, so its moving in a much faster pace. I can say a showcase that connects to skywalking can be presented immediately once the input-output interfaces are out (as standalone services depending on grpc APIs to send back results that are directly visualizable for both metrics and logs.) I'd like to know in-advance the design of the AI module so we can make sure both sides fit onto the same input-output.

Feature on log automatic aggregation and anomaly detection: Streaming and batch both supported
Input: [Logs in any format, can be preprocessed by lal or not + service/instance/pod metadata, **user settings],
Output: [Required: The templateID for LogID, The template content for templateID if updated/created] [Optional: one dimensional metrics on each log template count through time]

Feature on metrics anomaly detection and forecastiong: Streaming and batch both supported
Input: [Selected minute-wise golden signals such as sucess rate, cpm metrics etc. can be multi-dimensional, can be single-dimensional, service/instance metadata, **user settings]
output: [Required (flexible): The predicted next values with timestamp (or upper bound/lower bound) || the identified anomaly in history (depend on mode) ] [Optional: a baseline value to adjust alarm threshold] [Optional: api to directly trigger new alert] <<These outputs can be directly used to visualize on UI to augment original metrics.
not_implemented but planned: human in the loop mechanism, this mechanism provide a way to mark metrics as anomaly from UI so machine learning model can learn on examples.

Both of the features are preferred in streaming since they are more useful and can achieve near-real time response.

[wu-sheng]

To be honest, we need something in official and running, otherwise, no one knows its existence and how to use it.

[Superskyyy]

To be honest, we need something in official and running, otherwise, no one knows its existence and how to use it.

Indeed, I didn't have the time to push it forward fast. Sorry about that, but it will be there soon.

[wu-sheng]

Restful API Formatting starts from #10884 as promised.

[wu-sheng]

Better GC and performance(40% at least) formating implementation are landed in 9.5.0 through #10923

[wu-sheng]

v9.5.0 is out for supporting Restful API Formatting. Wait for @Superskyyy R3 implementation to make this up and running. Then we could have doc update and blog for this.