Replies: 1 comment
-
I don't know your Kafka issues, I am not a fan of that use case. But I can say something about following questions.
Because, as a monitoring system, if one OAP performance is even lower than an agent, then it means, you have larger OAP deployment scale than your application, which also means, the APM is not suitable for your cases.
There is not such limitation from SkyWalking side. That is your Kafka thing. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
question discription
1)Changing the way agents report data to Skywalking from GRPC to Kafka cluster mode.
2)OAP is in cluster mode with 3 nodes:
OAP node 172.31.91.218 role Aggregator
OAP node 172.30.15.183 role Receiver
OAP node 172.30.91.50 role Receiver
3)Kafka cluster mode with 3 nodes: 172.31.200.52:9092, 172.31.200.53:9092, 172.31.200.59:9092
Main Issues:
1)When sending 1000 requests to the monitored service, the cpm value displayed on the Skywalking-ui monitoring page is 1000 when data is reported in GRPC mode, and it is 350 when reported in Kafka cluster mode, significantly lower than 1000.
2)Modify the agent's configuration file of the monitored service, add the following configuration:
plugin.kafka.namespace=oap
Modify the application.yml configuration file of the oap node 172.31.91.218 in the kafka-fetcher module's namespace parameter
kafka-fetcher:
selector: ${SW_KAFKA_FETCHER:default}
default:
bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:172.31.200.52:9092,172.31.200.53:9092,172.31.200.59:9092}
namespace: ${SW_NAMESPACE:"oap"}
partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
enableNativeProtoLog: ${SW_KAFKA_FETCHER_ENABLE_NATIVE_PROTO_LOG:true}
enableNativeJsonLog: ${SW_KAFKA_FETCHER_ENABLE_NATIVE_JSON_LOG:true}
consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1}
kafkaHandlerThreadPoolSize: ${SW_KAFKA_HANDLER_THREAD_POOL_SIZE:-1}
kafkaHandlerThreadPoolQueueSize: ${SW_KAFKA_HANDLER_THREAD_POOL_QUEUE_SIZE:-1}
Question:
In the scenario of using a Kafka cluster to receive agent-reported data in an OAP cluster environment, why is it necessary to limit the agent data reporting to specific OAP nodes to ensure the correctness of cpm? Why does the cpm value decrease when using the default empty string as the namespace configuration?
Beta Was this translation helpful? Give feedback.
All reactions