AIOps
Artificial Intelligence for IT Operations(AIOps) is the application of AI and machine learning to IT and observability data. Opni AIOps offers the following features:
- Log Anomaly Detection
The goal of log anomaly detection is to apply machine-learning technologies to structure log data and detect anomalous logs. Opni AIOps incorporates an advanced deep learning and a log parsing method and offers:
- Pre-trained models for logs of the Kubernetes control plane, Rancher, and Longhorn.
- Workload Log Anomaly Detection. Ability to learn and self train models based on your workload logs (if 1+ GPU is available on the cluster)
This section will walk through a few key technologies that Opni Log Anomaly Detection based on.
Opni AIOps utilizes transformers to build a self-supervised learning model that formulates the masked word prediction task as masked language modeling. This model is then applied to log anomaly detection in unsupervised scenario, and it can also be further fine-tuned with downstream log-anomaly-detection task with ground-truth data.
- model input. raw unstructured logs
- tokenization. Normalize raw logs to lists of tokens. Some words are transferred to pre-defined tokens, for example,
https://rancher.combecomes<URL>. Tokenized sentences will always start with token<CLS>and end with padding token<PAD>, and it has a fixed number of tokens (such as 64 or 128), defined by a hyper parameter. - word masking. one token in each sentence is masked as
<MASK>for the model to predict. - build model. The model consists of N encoder layers, N decoder layers, a generator layer (linear + softmax). Typically N = 1 or 2.
- model output. predicted tokens of the
<MASK>tokens. - model training. model is trained with batches of training data. the loss function is defined by the accuracy of masked word prediction.
- model inference as unsupervised log anomaly detection task. The trained model will try to predict every token of an input sentence (except the leading
<CLS>token and the padding<PAD>token), the anomaly score is then calculated byanomaly_score = token_correct_predict / token_total
Log parsing is the first step of log analysis. It transforms unstructured log messages into structured events, namely log templates. Opni AIOps built the log parsing module on top of an open-sourced project Drain3, which is an online log template miner that can extract templates from streaming log messages.
Opni AIOps uses Nats to implement data pipeline including message queues and key-value storage. A python package Nats Wrapper is created to simplify the usage of nats in python services. Additionally, Protobuf is used to encapsulate data between the different microservices.