scraping should not be effected when reloading Prometheus without change any config

[codwu]

Bug Report

What did you do?

reload Prometheus without change any config

curl -X POST http://localhost:9090/-/reload

What did you expect to see?
there is no effect on target scraping

What did you see instead? Under which circumstances?
scrape interval become longer than I configured

Environment

• System information:

Linux 3.10.0-229.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc)
  build user:       root@149e5b3f0829
  build date:       20180314-14:15:45
  go version:       go1.10

• Prometheus configuration file:

global:
  scrape_interval:     30s 
  evaluation_interval: 30s

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

rule_files:

scrape_configs:
  - job_name: 'prometheus1'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'prometheus2'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'prometheus3'
    static_configs:
      - targets: ['localhost:9090']

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

level=info ts=2018-06-05T01:10:15.681143246Z caller=main.go:220 msg="Starting Prometheus" version="(version=2.2.1, branch=HEAD, revision=bc6058c81272a8d938c05e75607371284236aadc)"
level=info ts=2018-06-05T01:10:15.681218089Z caller=main.go:221 build_context="(go=go1.10, user=root@149e5b3f0829, date=20180314-14:15:45)"
level=info ts=2018-06-05T01:10:15.681243105Z caller=main.go:222 host_details="(Linux 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 centos7b (none))"
level=info ts=2018-06-05T01:10:15.681261809Z caller=main.go:223 fd_limits="(soft=1024, hard=4096)"
level=info ts=2018-06-05T01:10:15.683833415Z caller=main.go:504 msg="Starting TSDB ..."
level=info ts=2018-06-05T01:10:15.684348421Z caller=web.go:382 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-06-05T01:10:15.688001338Z caller=main.go:514 msg="TSDB started"
level=info ts=2018-06-05T01:10:15.688036386Z caller=main.go:588 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2018-06-05T01:10:15.689628626Z caller=main.go:491 msg="Server is ready to receive web requests."
level=info ts=2018-06-05T01:13:01.657278126Z caller=main.go:588 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2018-06-05T01:13:34.633327981Z caller=main.go:588 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2018-06-05T01:14:02.592713121Z caller=main.go:588 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2018-06-05T01:14:23.356799561Z caller=main.go:588 msg="Loading configuration file" filename=prometheus.yml

[codwu]

scrape interval become correct after this scraping

[codwu]

this is probably where the bug is

prometheus/scrape/manager.go

Lines 87 to 94 in bc6058c

	// Cleanup and reload pool if config has changed.
	for name, sp := range m.scrapePools {
	if cfg, ok := m.scrapeConfigs[name]; !ok {
	sp.stop()
	delete(m.scrapePools, name)
	} else if !reflect.DeepEqual(sp.config, cfg) {
	sp.reload(cfg)
	}

reflect.DeepEqual returned false but it should be true

[krasi-georgiev]

are you 100% there were no config changes?

[codwu]

I am pretty sure nobody changes the config.

[krasi-georgiev]

that is unexpected
can you paste the output if you add.
this will output all fields with their types so I can see why does it detect to be different.

please remove any sensitive information before pasting.

else if !reflect.DeepEqual(sp.config, cfg) { 
               fmt.Printf("%+v \n",sp.config)
               fmt.Printf("%+v \n",cfg)
 		sp.reload(cfg) 
 	} 

[codwu]

the output is below

level=info ts=2018-06-06T14:38:25.081519725Z caller=main.go:588 msg="Loading configuration file" filename=prometheus.yml
&{JobName:prometheus2 HonorLabels:false Params:map[] ScrapeInterval:30s ScrapeTimeout:10s MetricsPath:/metrics Scheme:http SampleLimit:0 ServiceDiscoveryConfig:{StaticConfigs:[0] DNSSDConfigs:[] FileSDConfigs:[] ConsulSDConfigs:[] ServersetSDConfigs:[] NerveSDConfigs:[] MarathonSDConfigs:[] KubernetesSDConfigs:[] GCESDConfigs:[] EC2SDConfigs:[] OpenstackSDConfigs:[] AzureSDConfigs:[] TritonSDConfigs:[] XXX:map[]} HTTPClientConfig:{}
 RelabelConfigs:[] MetricRelabelConfigs:[] XXX:map[]} 
&{JobName:prometheus2 HonorLabels:false Params:map[] ScrapeInterval:30s ScrapeTimeout:10s MetricsPath:/metrics Scheme:http SampleLimit:0 ServiceDiscoveryConfig:{StaticConfigs:[] DNSSDConfigs:[] FileSDConfigs:[] ConsulSDConfigs:[] ServersetSDConfigs:[] NerveSDConfigs:[] MarathonSDConfigs:[] KubernetesSDConfigs:[] GCESDConfigs:[] EC2SDConfigs:[] OpenstackSDConfigs:[] AzureSDConfigs:[] TritonSDConfigs:[] XXX:map[]} HTTPClientConfig:{}
 RelabelConfigs:[] MetricRelabelConfigs:[] XXX:map[]} 
&{JobName:prometheus3 HonorLabels:false Params:map[] ScrapeInterval:30s ScrapeTimeout:10s MetricsPath:/metrics Scheme:http SampleLimit:0 ServiceDiscoveryConfig:{StaticConfigs:[0] DNSSDConfigs:[] FileSDConfigs:[] ConsulSDConfigs:[] ServersetSDConfigs:[] NerveSDConfigs:[] MarathonSDConfigs:[] KubernetesSDConfigs:[] GCESDConfigs:[] EC2SDConfigs:[] OpenstackSDConfigs:[] AzureSDConfigs:[] TritonSDConfigs:[] XXX:map[]} HTTPClientConfig:{}
 RelabelConfigs:[] MetricRelabelConfigs:[] XXX:map[]} 
&{JobName:prometheus3 HonorLabels:false Params:map[] ScrapeInterval:30s ScrapeTimeout:10s MetricsPath:/metrics Scheme:http SampleLimit:0 ServiceDiscoveryConfig:{StaticConfigs:[] DNSSDConfigs:[] FileSDConfigs:[] ConsulSDConfigs:[] ServersetSDConfigs:[] NerveSDConfigs:[] MarathonSDConfigs:[] KubernetesSDConfigs:[] GCESDConfigs:[] EC2SDConfigs:[] OpenstackSDConfigs:[] AzureSDConfigs:[] TritonSDConfigs:[] XXX:map[]} HTTPClientConfig:{}
 RelabelConfigs:[] MetricRelabelConfigs:[] XXX:map[]} 
&{JobName:prometheus1 HonorLabels:false Params:map[] ScrapeInterval:30s ScrapeTimeout:10s MetricsPath:/metrics Scheme:http SampleLimit:0 ServiceDiscoveryConfig:{StaticConfigs:[0] DNSSDConfigs:[] FileSDConfigs:[] ConsulSDConfigs:[] ServersetSDConfigs:[] NerveSDConfigs:[] MarathonSDConfigs:[] KubernetesSDConfigs:[] GCESDConfigs:[] EC2SDConfigs:[] OpenstackSDConfigs:[] AzureSDConfigs:[] TritonSDConfigs:[] XXX:map[]} HTTPClientConfig:{}
 RelabelConfigs:[] MetricRelabelConfigs:[] XXX:map[]} 
&{JobName:prometheus1 HonorLabels:false Params:map[] ScrapeInterval:30s ScrapeTimeout:10s MetricsPath:/metrics Scheme:http SampleLimit:0 ServiceDiscoveryConfig:{StaticConfigs:[] DNSSDConfigs:[] FileSDConfigs:[] ConsulSDConfigs:[] ServersetSDConfigs:[] NerveSDConfigs:[] MarathonSDConfigs:[] KubernetesSDConfigs:[] GCESDConfigs:[] EC2SDConfigs:[] OpenstackSDConfigs:[] AzureSDConfigs:[] TritonSDConfigs:[] XXX:map[]} HTTPClientConfig:{}
 RelabelConfigs:[] MetricRelabelConfigs:[] XXX:map[]}

[krasi-georgiev]

nice , thanks , I already see the difference in StaticConfigs:[0].

@simonpasquier , @pgier do you want to take this one? Otherwise I can have a look tomorrow.

[pgier]

@krasi-georgiev yeah, I can take a look if you are not already working on it

[pgier]

This appears to be the culprit. The array index of the target is set on the source field after the config has been loaded.

prometheus/discovery/manager.go

Lines 294 to 298 in bc6058c

	func NewStaticProvider(groups []*targetgroup.Group) *StaticProvider {
	for i, tg := range groups {
	tg.Source = fmt.Sprintf("%d", i)
	}
	return &StaticProvider{groups}

[krasi-georgiev]

Thanks @pgier. Do you want to open a PR and I will test it. Should add a test to catch this.

[pgier]

Created #4245 with a fix. There are some tests already around config reloading, but I'll see if I can make a test for this specific case next week.

[krasi-georgiev]

thanks , ping when ready for review.

[pgier]

I added a test, although its kind of very specific to this case. Feel free to review.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.