Prometheus remote write missing measurements name

[Nodraak]

I don't like copy pasting issues and not respecting the template, but really, all the needed info is here : Initial issue opened on Influxdb's Github. So I'll just quote him:

---

Metrics send from prometheus to influxdb with native influx remote write have missing measurement name.
All metrics are saved by InfluxDB to _ measurement.

> select * from _ where __name__='up' and kubernetes_io_hostname='ctl03' limit 1;
name: _
time                From To __name__ action addonmanager_kubernetes_io_mode alertmanager alertname alertstate annotation_io_kubernetes_container_hash annotation_io_kubernetes_container_ports annotation_io_kubernetes_container_restartCount annotation_io_kubernetes_container_terminationMessagePath annotation_io_kubernetes_container_terminationMessagePolicy annotation_io_kubernetes_pod_terminationGracePeriod annotation_kubernetes_io_config_seen annotation_kubernetes_io_config_source annotation_kubernetes_io_created_by annotation_prometheus_io_port annotation_prometheus_io_scrape annotation_scheduler_alpha_kubernetes_io_tolerations app backend beta_kubernetes_io_arch beta_kubernetes_io_os bind branch buildDate call client cluster_name code com_docker_stack_namespace com_docker_swarm_node_id com_docker_swarm_service_id com_docker_swarm_service_name com_docker_swarm_task_id com_docker_swarm_task_name com_mirantis_image_specs_calico_bird6Url com_mirantis_image_specs_calico_birdUrl com_mirantis_image_specs_calico_birdclUrl com_mirantis_image_specs_calico_confdUrl com_mirantis_image_specs_calico_felixImage com_mirantis_image_specs_docker_imgTag com_mirantis_image_specs_gerritChangeId com_mirantis_image_specs_gerritChangeNumber com_mirantis_image_specs_gerritPatchsetNumber com_mirantis_image_specs_gerritPatchsetRevision com_mirantis_image_specs_gerritProject com_mirantis_monitoring compiler container_image container_name container_version contentType controller controller_revision_hash cpu database device dialer_name direction dockerVersion endpoint engine engine_host event exe f64 fstype gitCommit gitTreeState gitVersion goVersion goversion grpc_code grpc_method grpc_service grpc_type handler host id image input instance integration interface interval io_kubernetes_container_logpath io_kubernetes_container_name io_kubernetes_docker_type io_kubernetes_pod_name io_kubernetes_pod_namespace io_kubernetes_pod_uid io_kubernetes_sandbox_id ip_version job                 k8s_app kernelVersion kubernetes_io_hostname le listener_name major method minor name namespace network node_attribute_master node_host node_id node_name node_role_kubernetes_io_master node_role_kubernetes_io_node operation_type osVersion output path pattern pid platform pod_name pod_template_generation pod_template_hash port process_name protocol_instance proxy quantile queue reason refid region  remote resource retentionPolicy revision role route_type rule_type scope scrape_job server service severity slice socket state state_prefix status stratum subresource sv table type unit url username verb version walPath
----                ---- -- -------- ------ ------------------------------- ------------ --------- ---------- --------------------------------------- ---------------------------------------- ----------------------------------------------- --------------------------------------------------------- ----------------------------------------------------------- --------------------------------------------------- ------------------------------------ -------------------------------------- ----------------------------------- ----------------------------- ------------------------------- ---------------------------------------------------- --- ------- ----------------------- --------------------- ---- ------ --------- ---- ------ ------------ ---- -------------------------- ------------------------ --------------------------- ----------------------------- ------------------------ -------------------------- ---------------------------------------- --------------------------------------- ----------------------------------------- ---------------------------------------- ------------------------------------------ -------------------------------------- --------------------------------------- ------------------------------------------- --------------------------------------------- ----------------------------------------------- -------------------------------------- ----------------------- -------- --------------- -------------- ----------------- ----------- ---------- ------------------------ --- -------- ------ ----------- --------- ------------- -------- ------ ----------- ----- --- --- ------ --------- ------------ ---------- --------- --------- --------- ----------- ------------ --------- ------- ---- -- ----- ----- -------- ----------- --------- -------- ------------------------------- ---------------------------- ------------------------- ---------------------- --------------------------- --------------------- ------------------------ ---------- ---                 ------- ------------- ---------------------- -- ------------- ----- ------ ----- ---- --------- ------- --------------------- --------- ------- --------- ------------------------------ ---------------------------- -------------- --------- ------ ---- ------- --- -------- -------- ----------------------- ----------------- ---- ------------ ----------------- ----- -------- ----- ------ ----- ------  ------ -------- --------------- -------- ---- ---------- --------- ----- ---------- ------ ------- -------- ----- ------ ----- ------------ ------ ------- ----------- -- ----- ---- ---- --- -------- ---- ------- -------
1513343319131000000         up                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          amd64                   linux                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      0                                                                                                                                         ctl03                                                                                                                                                                                                                                       kubernetes-cadvisor                       ctl03                                                                                                                               true                                                                                                                                                                                                                                           region1

Looks like labels from ALL metrics are merged, so each metric have labels (set to null) from all other metrics...

When Prometheus Remote Storage Adapter is in place:

> select * from up limit 1;
name: up
time                beta_kubernetes_io_arch beta_kubernetes_io_os instance job             kubernetes_io_hostname node_role_kubernetes_io_master node_role_kubernetes_io_node region  value
----                ----------------------- --------------------- -------- ---             ---------------------- ------------------------------ ---------------------------- ------  -----
1513585977606000000 amd64                   linux                 ctl03    kubernetes-node ctl03                  true                                                        region1 0

IMHO native support for prometheus remote write should behave similar to remote storage adapter, and use label __name__ as measurement name

InfluxDB: 1.4.2-1
Ubuntu: 16.04
Prometheus: 2.0.0

How to reproduce:

Configure remote write in prometheus:

remote_write:
  # docker_remote_write
  - url: http://INFLUXDB_HOST:8086/api/v1/prom/write?u=USER&p=PASS&db=DB

[brian-brazil]

That looks to be an issue on Influx's end.

[Nodraak]

I must admit that I have not looked at the kind of requests Prometheus sends to Influxdb ; but I guess that Influxdb stores what it receives, so I suppose it's up to Prometheus to send data in the proper format.

I'm don't mean to be aggressive, I honestly don't know if it's up to Influxdb or Prometheus to fix it, but given that the issue on Influxdb is opened since some time and does not have any answer, I thought I would open it here as well.

[brian-brazil]

Prometheus is what defines the format, it's up to Influx to alter it in any way it sees fit.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.