Scrape Commit failed from prometheus receiver

[kiahmed]

Describe the bug
did anyone see this or have a working example of prometheus receiver against a static_target ?

ERROR scrape/scrape.go:1088 Scrape commit failed {"kind": "receiver", "name": "prometheus", "scrape_pool": "node-collector", "target": "http://relay-node:12798/metrics", "err": "Permanent error: [Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid tsMap: cannot be empty map]"}
github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport.func1
github.com/prometheus/prometheus@v1.8.2-0.20210217141258-a6be548dbc17/scrape/scrape.go:1088
github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport
github.com/prometheus/prometheus@v1.8.2-0.20210217141258-a6be548dbc17/scrape/scrape.go:1153
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run
github.com/prometheus/prometheus@v1.8.2-0.20210217141258-a6be548dbc17/scrape/scrape.go:1039
[ec2-user@ip-172-31-7-205 aws-oltp-collector]$ docker logs --tail 10 aws-otc
github.com/prometheus/prometheus@v1.8.2-0.20210217141258-a6be548dbc17/scrape/scrape.go:1153
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run
github.com/prometheus/prometheus@v1.8.2-0.20210217141258-a6be548dbc17/scrape/scrape.go:1039

Steps to reproduce
use a prometheus static target for an instrumented app

What did you expect to see?
A clear and concise description of what you expected to see.

What did you see instead?
A clear and concise description of what you saw instead.

What version did you use?
Version: (e.g., v0.4.0, 1eb551b, etc)

What config did you use?
prometheus:
config:
global:
scrape_interval: 1m
scrape_timeout: 10s
scrape_configs:
- job_name: "relay-node-collector"
scrape_interval: 10s
sample_limit: 10000
static_configs:
- targets: ['relay-node:12100']

Environment
debian 10 and docker

Additional context
Add any other context about the problem here.

[khanhntd]

Hi @kiahmed , thanks for waiting so long to receive an answer. The reasons why you have this error is because the received metrics from the app is an invalid type. One of the way to get through this is to send you a debug adot collector to know which metrics and their type are causing this error. Let's me know if you are still having this issue.
Note: Since there are any debug level infor in this code, using logging debug is not feasible.

[gouthamve]

Hi @kiahmed, is this still an issue?

"invalid temporality and type combination" only exists on the remote write path now. I believe this has been fixed in the receiver end. @dashpole can we close this?