discovery 403 (forbidden) error

[haiakab]

Hi,
multiple subscriptions defined under our Azure account,
subscriptions have the same authentication (mode=UserAssignedManagedIdentity)
for all of the subscription, discovery is created as well - container is running, curl command returns Ok status (200).
scrapers were defined for the subscriptions with multiple metrics, but it's not working for some of the scrapers - getting 403 (forbidden) error for some of the subscriptions.
I see that this error is related to authentication- but the same auth is working for other subscriptions
what can be the issue?

error log:
Failed to scrape resource collection virtual-machine-landscape: Response status code does not indicate success: 403 (Forbidden).
System.Net.Http.HttpRequestException: Response status code does not indicate success: 403 (Forbidden).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendGetRequestAsync(String uriPath) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 63
at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.GetAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 44
at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryRepository.GetResourceDiscoveryGroupAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryRepository.cs:line 25
at Promitor.Agents.Scraper.Scheduling.ResourceDiscoveryGroupScrapingJob.ExecuteAsync(CancellationToken cancellationToken) in /src/Promitor.Agents.Scraper/ResourceDiscoveryGroupScrapingJob.cs:line 80

[tomkerkhove]

What permissions did you assign to the application?

[haiakab]

sure, although after increasing the verbosity just able to get these logs, hope this will help
prom_scrape.txt

[haiakab]

@tomkerkhove , hi any updates on this?

[tomkerkhove]

Hard to say, to be honest. Some authentication is misconfigured but not sure why. Is it for the correct subscription?

[haiakab]

@tomkerkhove , yes subscription is correct, also our identity has read access over that subscription

[tomkerkhove]

Unfortunately I'll need more logs then such as Azure Monitor or so, there's nothing more I can tell based on logs above