azure service discovery does not report public IP

[dmitsh]

Bug Report

What did you do?
Run azure service discovery

What did you expect to see?
A label with VM's public IP should be added to the target VMs

What did you see instead? Under which circumstances?
The label with public IP was missing

Environment

• System information:

  Linux 4.15.0-50-generic x86_64

• Prometheus version:

  prometheus, version 2.9.2 (branch: upstream-master, revision: d6a4daa)
  go version: go1.12.4

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s
  evaluation_interval: 30s
  # scrape_timeout is set to the global default (10s).

scrape_configs:
- job_name: service-azure
  azure_sd_configs:
    - environment: AzurePublicCloud
      authentication_method: OAuth
      subscription_id: xxxxxx
      tenant_id: xxxxxx
      client_id: xxxxxx
      client_secret: xxxxxx
      port: 9100

[dmitsh]

The label for public IP is not added because azure SDK does not populate PublicIPAddressPropertiesFormat on network interface request.
A simple workaround would be to add another remote API call to network.PublicIPAddressesClient.
However that will double number of remote API read operation and cause throttling of Azure Resource Manager.

I've opened an issue Azure/azure-sdk-for-go#4829 for Azure SDK team.
Once addressed, I can proceed with fixing and resolving this issue.

[Deshke]

is this not fixed with #5475 ?

[dmitsh]

@Deshke unfortunately not. Azure does not populate public IP in getNetworkInterfaceByID().
There was another related issue which I fixed in #5587

[silencev]

@dmitsh
I'm not familiar with Azure SDK, is it possible to get VM's FQDN for workaround?

[dmitsh]

@dmitsh
I'm not familiar with Azure SDK, is it possible to get VM's FQDN for workaround?

@silencev you may want to take a look at https://github.com/dmitsh/prometheus/blob/ds-up-azure-discovery-wip/discovery/azure/azure.go#L504
The result is of type network.Interface. FQDN could be retrieved by following Interface->InterfacePropertiesFormat->InterfaceEndpoint->InterfaceEndpointProperties->Fqdn

[silencev]

I'm using relabel for workaround now.
composing the FQDN my-cname.centralus.cloudapp.azure.com by CNAME tag and VM region
CNAME tag is manually added :(

  - source_labels: [__meta_azure_machine_tag_CNAME, __meta_azure_machine_location]
    separator: .
    regex: (.+)
    replacement: ${1}${2}.cloudapp.azure.com:8080
    target_label: __address__

[gouthamve]

Please send us a PR with this feature, but if this lookup causes another RPC call to the API, we'd likely not add it because the Azure API limits are very low already and adding additional RPCs will make things worse.

[gavinkflam]

Hi, I worked on a fix and created a PR #13241. Please help to review the changes.