Can Vector processing metrics data from Kafka to Prometheus directly

[u-kyou]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I am trying to find a tool which can support:
prometheus -(1)-> kafka -(2)-> VictoriaMetrics

(1) prometheus -vector-> kafka Succeed
vector's prometheus_remote_write source and kafka sink help me send data from prometheus to kafka already. The data in kafka is like:
{"name":"up","tags":{"instance":"10.32.140.2:9090","job":"prometheus","usage":"test-vector"},"timestamp":"2023-10-24T11:27:07.024Z","kind":"absolute","gauge":{"value":1.0}}

(2) kafka --> VictoriaMetrics Failed
I got error when I use kafka source and prometheus_remote_write sink:
ERROR vector::cli: Configuration error. error=Data type mismatch between prome_kafka_in (Log) and out_to_prome (Metric)

I noticed that there is a transform named log_to_metric. I tried but it did not work as expected.

Configuration

[sources.prome_kafka_in]
type = "kafka"
bootstrap_servers = "kafka_addr:9092"
topics = ["sre_prometheus_remote_write"]
group_id ="vector_prome_kafka"

#[transforms.log_to_metric]
#type = "log_to_metric"
#inputs = [ "prome_kafka_in" ]
#
#  [[transforms.log_to_metric.metrics]]
#  field = "gauge.value"
#  name = "up"
#  type = "gauge"

[sinks.out_to_prome]
type = "prometheus_remote_write"
inputs = [ "prome_kafka_in" ]
endpoint = "http://remote_write_addr"

Version

timberio/vector:0.33.0-debian

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Hi @Felix-YuJing . This seems more like a question than a bug report so I'll convert it to a GitHub Discussion.