Adding kafka forwarder plugins #123
Conversation
|
Could you provide your use case? |
|
@wu-sheng |
| new(grpc_nativeclr.Forwarder), | ||
| new(grpc_nativeprocess.Forwarder), | ||
| new(grpc_nativeebpfprofiling.Forwarder), |
There was a problem hiding this comment.
Why did you need to delete the existing forwarders?
| // empty struct for set | ||
| type void struct{} | ||
|
|
||
| var empty void |
There was a problem hiding this comment.
I think these fields and struct are not used, why do you declare in there?
There was a problem hiding this comment.
Thanks for your contribution, please complete the following steps:
- Fix the comments.
- Using
make gen-docsto generate the documentation.- You need to create an E2E to verify the Kafka forwards are working well.
Sure will check and fix all the comments added..
| var message []*sarama.ProducerMessage | ||
| for _, e := range batch { | ||
| data := e.GetData().(*v1.SniffData_Meter) | ||
| rawdata, ok := proto.Marshal(data.Meter) | ||
| if ok != nil { | ||
| return ok | ||
| } | ||
| message = append(message, &sarama.ProducerMessage{ | ||
| Topic: f.Topic, | ||
| Value: sarama.ByteEncoder(rawdata), | ||
| }) | ||
| } | ||
| return f.producer.SendMessages(message) |
There was a problem hiding this comment.
The meter data needs to be sent to the consistent backend address. Please follow the native gRPC Meter forwarder to get more information.
There was a problem hiding this comment.
@mrproliu , Can you please elaborate a bit more on this as here we are sending to kafka and and then oap reads from the same topic
There was a problem hiding this comment.
Kafka has the partitioning mechanism. Some metrics like meter should always forward to the consistent partition
There was a problem hiding this comment.
Thanks @wu-sheng , We will check this
| # contributor license agreements. See the NOTICE file distributed with | ||
| # this work for additional information regarding copyright ownership. | ||
| # The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| # (the "License"); you may not use this file except in compliance with |
There was a problem hiding this comment.
Please don't add the files under the target directory.
|
Another thing matters, please provide the sizes of binary before and after these new forwarders. |
Log's Kafka forward doesn't mean anything actually. It is a legacy feature. In the very beginning of Satellite, the agent doesn't provide log reporter, and Satellite did instead. |
We have added kafka forwarder plugins for event, meters, metrics, profiling, management and jvm.