[__meta_azure_machine_resource_group] returns the value in lowercase

[yanivroz]

Bug Report

What did you do?
Tried to scrape instances from a specific resource group in azure that contains uppercase characters in the resource group name

What did you expect to see?
Intances in the resource group are discovered

What did you see instead? Under which circumstances?
Instances were not discovered.
Only when I typed the resource group in all lowercase under the regex it was matched (even though the actual value contains both upper and lower case characters)

Environment
Azure

• System information:

Linux 4.15.0-34-generic x86_64

• Prometheus version:

prometheus, version 2.4.3 (branch: HEAD, revision: 167a4b4e73a8eca8df648d2d2043e21bdb9a7449)
  build user:       root@1e42b46043e9
  build date:       20181004-08:42:02
  go version:       go1.11.1

• Prometheus configuration file:

scrape_configs:
  - job_name:       'XXX'
    azure_sd_configs:
      - subscription_id: 'XXX'
        tenant_id: 'XXX'
        client_id: 'XXX'
        client_secret: 'XXX'
        port: 46000
        refresh_interval: '60s'

    relabel_configs:
            - source_labels: [__meta_azure_machine_resource_group]
              regex: 'myResourceGroupName'
              action: keep

• Logs:
  with regex all in lowercase (regex: 'myresourcegroupname').
  you can see 2 instances were discovered in the bottom 2 lines

level=info ts=2018-10-18T09:00:17.149312488Z caller=main.go:238 msg="Starting Prometheus" version="(version=2.4.3, branch=HEAD, revision=167a4b4e73a8eca8df648d2d2043e21bdb9a7449)"
level=info ts=2018-10-18T09:00:17.149639717Z caller=main.go:239 build_context="(go=go1.11.1, user=root@1e42b46043e9, date=20181004-08:42:02)"
level=info ts=2018-10-18T09:00:17.149894257Z caller=main.go:240 host_details="(Linux 4.15.0-34-generic #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 ubuntu (none))"
level=info ts=2018-10-18T09:00:17.150367396Z caller=main.go:241 fd_limits="(soft=1024, hard=1048576)"
level=info ts=2018-10-18T09:00:17.150962767Z caller=main.go:242 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2018-10-18T09:00:17.152621606Z caller=main.go:554 msg="Starting TSDB ..."
level=info ts=2018-10-18T09:00:17.17058081Z caller=web.go:397 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-10-18T09:00:17.209714294Z caller=main.go:564 msg="TSDB started"
level=info ts=2018-10-18T09:00:17.210164003Z caller=main.go:624 msg="Loading configuration file" filename=prometheus.yml
level=debug ts=2018-10-18T09:00:17.211811274Z caller=manager.go:158 component="discovery manager scrape" msg="Starting provider" provider=*azure.SDConfig/0 subs=[XXX]
level=info ts=2018-10-18T09:00:17.212114839Z caller=main.go:650 msg="Completed loading of configuration file" filename=prometheus.yml
level=info ts=2018-10-18T09:00:17.212381113Z caller=main.go:523 msg="Server is ready to receive web requests."
level=debug ts=2018-10-18T09:00:23.342276167Z caller=azure.go:358 component="discovery manager scrape" discovery=azure msg="Azure discovery completed"
level=debug ts=2018-10-18T09:00:42.234106305Z caller=scrape.go:794 component="scrape manager" scrape_pool=XXX target=http://10.0.0.13:46000/metrics msg="Scrape failed" err="context deadline exceeded"
level=debug ts=2018-10-18T09:00:44.765680626Z caller=scrape.go:794 component="scrape manager" scrape_pool=XXX target=http://10.0.0.14:46000/metrics msg="Scrape failed" err="context deadline exceeded" 

With regex set as the actual resource group name case (regex: 'myResourceGroupName')
no instances were discovered

level=info ts=2018-10-18T09:00:17.149312488Z caller=main.go:238 msg="Starting Prometheus" version="(version=2.4.3, branch=HEAD, revision=167a4b4e73a8eca8df648d2d2043e21bdb9a7449)"
level=info ts=2018-10-18T09:00:17.149639717Z caller=main.go:239 build_context="(go=go1.11.1, user=root@1e42b46043e9, date=20181004-08:42:02)"
level=info ts=2018-10-18T09:00:17.149894257Z caller=main.go:240 host_details="(Linux 4.15.0-34-generic #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 ubuntu (none))"
level=info ts=2018-10-18T09:00:17.150367396Z caller=main.go:241 fd_limits="(soft=1024, hard=1048576)"
level=info ts=2018-10-18T09:00:17.150962767Z caller=main.go:242 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2018-10-18T09:00:17.152621606Z caller=main.go:554 msg="Starting TSDB ..."
level=info ts=2018-10-18T09:00:17.17058081Z caller=web.go:397 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-10-18T09:00:17.209714294Z caller=main.go:564 msg="TSDB started"
level=info ts=2018-10-18T09:00:17.210164003Z caller=main.go:624 msg="Loading configuration file" filename=prometheus.yml
level=debug ts=2018-10-18T09:00:17.211811274Z caller=manager.go:158 component="discovery manager scrape" msg="Starting provider" provider=*azure.SDConfig/0 subs=[XXX]
level=info ts=2018-10-18T09:00:17.212114839Z caller=main.go:650 msg="Completed loading of configuration file" filename=prometheus.yml
level=info ts=2018-10-18T09:00:17.212381113Z caller=main.go:523 msg="Server is ready to receive web requests."
level=debug ts=2018-10-18T09:00:18.880305596Z caller=azure.go:257 component="discovery manager scrape" discovery=azure msg="Found virtual machines during Azure discovery." count=122
level=debug ts=2018-10-18T09:00:23.342276167Z caller=azure.go:358 component="discovery manager scrape" discovery=azure msg="Azure discovery completed"

Using azure cli you can see the actual resource group does contain uppercase characters

az group list | grep name | grep -iw myResourceGroupName
    "name": "myResourceGroupName",

[simonpasquier]

Looking at the code, the resource group is explicitly lower-cased:

prometheus/discovery/azure/azure.go

Lines 230 to 233 in 5824d69

	return azureResource{
	Name: strings.ToLower(s[8]),
	ResourceGroup: strings.ToLower(s[4]),
	}, nil

The (lower-case) value is used later on to query Azure:

prometheus/discovery/azure/azure.go

Line 415 in 5824d69

result, err := client.vmssvm.List(r.ResourceGroup, *(scaleSet.Name), "", "", "")

I can't say what would happen if the resource group is kept as-is (would the 2nd query fail?). Also people are probably relying on the value being lower-case so it might be better to just update the Azure SD documentation to state that the label value is always lower-case?

[yanivroz]

Thanks for your quick response. It’s not clear to me why the resource group is explicitly lower-cased. Other meta labels such as __meta_azure_machine_name are not explicitly lower-cased, so why treat resource group differently? Seems like a bug to me. (Although Azure doesn’t let you create 2 resource groups with same name but different case. for example if I have a resource group named “test” I cannot create another group named “Test”)

[simonpasquier]

It's been like that since the Azure SD support has been added to Prometheus: 0988e3b#diff-7e618b6815f2ec80ed046c215305bb13R131

Again I can't explain why it is this way but changing it might have side effects.

[brian-brazil]

We are permitted to make breaking changes to SDs such as if we discover we didn't get something right, or the underlying SD has changed in some way.

[simonpasquier]

@yanivroz if you can, feel free to test what happens if strings.ToLower() is removed. I don't have access to an Azure environment personally.

[yanivroz]

@simonpasquier
I did as you suggested and changed:

	return azureResource{
		Name:          strings.ToLower(s[8]),
		ResourceGroup: strings.ToLower(s[4]),
	}, nil

To:

	return azureResource{
		Name:          s[8],
		ResourceGroup: s[4],
	}, nil

But that didn't work, the instances were not discovered regardless which case I used in the regex (not sure why)

So to work around this problem I just added the case-insensitive flag into the regex:
regex: '(?i:(myResourceGroupName))'
and that did the trick.

[simonpasquier]

@yanivroz thanks for the follow-up. In that case, you can close the issue.