failed to write data using prometheus remote_write

[GoneLikeAir]

What type of bug is this?

Unexpected error

What subsystems are affected?

Metasrv, Frontend, Datanode

Minimal reproduce step

1. deploy greptimedb cluster in k8s using helm chart, version 0.7.2
2. change frontend service type to nodeport
3. add remote write config to prometheus

What did you expect to see?

read nd write metrics data successfully

What did you see instead?

error log in metasrv and frontend, and failed to read and write

What operating system did you use?

CentOS Linux release 7.9.2009

What version of GreptimeDB did you use?

0.7.2

Relevant log output and stack trace

log in matesrv: 
`WARN meta_srv::handler::collect_stats_handler: Incomplete heartbeat data: HeartbeatRequest { header: Some(RequestHeader { protocol_version: 1, cluster_id: 0, member_id: 2, role: Datanode, tracing_context: {} }), peer: None, report_interval: None, region_stats: [], mailbox_message: None, duration_since_epoch: 0, node_epoch: 0 }, err: 0: Invalid heartbeat request: missing header or peer, at src/meta-srv/src/handler/node_stat.rs:130:14`

log in frontend:
`ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=90 ms`

[zyy17]

@GoneLikeAir Thank you for your report! I will confirm it based on your description.

[zyy17]

@GoneLikeAir, is this an issue that must occur? Based on my environment, I can't reproduce your problem. Can you give me more information, for example:

• the specific steps
• more key error log of other components, for example, datanode or frontend;

The following are my steps for using remote write and read:

• Install GreptimeDB cluster by the guide. After installation, I use kubectl port-forward to expose 4000 and 4002;

• Install node exporter:

docker run -d \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  quay.io/prometheus/node-exporter:latest \
  --path.rootfs=/host

• Install Prometheus

docker run --network=host \ -p 9090:9090 \ -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml \ prom/prometheus

And here is my prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
- job_name: node
  static_configs:
  - targets: ['localhost:9100']

remote_write:
- url: http://localhost:4000/v1/prometheus/write?db=public&physical_table=greptime_physical_table

remote_read:
- url: http://localhost:4000/v1/prometheus/read?db=public

• Use mysql to read the data:

mysql -h 127.0.0.1 -P 4002

[killme2008]

I think it's a compatible issue. Do you still have this problem? @GoneLikeAir

[killme2008]

I closed this issue. Feel free to reopen it if you still have some issues.