list databases REST API and SDK Container.ReadThroughputAsync do not work with disable local auth

[daveoshinsky]

We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.

Describe the bug
There is no REST API or SDK API that works properly with disable local auth to list databases under a Cosmos account. The same situation likely exists with the very similar REST API to list containers in a database.

To Reproduce
Attempt REST API
https://learn.microsoft.com/en-us/rest/api/cosmos-db/list-databases
with disable local auth. The REST API normally works with an "authorization" header based on primary master key. When local authorization is disabled, the list databases REST API will always fail with 401 (Unauthorized). Attempts were made to obtain a token and use a "bearer" header with the list databases REST API. That also fails with 401 (Unauthorized).

Expected behavior
There should be a way to list databases (and to list containers in a database) when disable local auth is enabled. There should be SDK API's (CosmosClient methods) to do the same, but no such API's exist.

Actual behavior
All attempts to list databases (via REST API) fail with 401 (Unauthorized) with disable local auth. Since listing databases fails, this has not been tested with the very similar REST API to list containers in a database, but the same result would very likely be seen there as well.

Environment summary
SDK Version: 3.42.0
OS Version: Windows 11

Additional context
This might be due to the same root cause as unresolved issue:
Azure/cosmos-explorer#1470
Web pages describing authorization with Cosmos DB REST API's
https://learn.microsoft.com/en-us/rest/api/cosmos-db/common-cosmosdb-rest-request-headers
and
https://learn.microsoft.com/en-us/rest/api/cosmos-db/access-control-on-cosmosdb-resources?redirectedfrom=MSDN
do not mention using a bearer token header with these REST API's. Without such an ability, the REST API's will always fail with 401 (Unauthorized) with disable local auth.
By contrast, the REST API to list database accounts
https://learn.microsoft.com/en-us/rest/api/cosmos-db-resource-provider/database-accounts/list?view=rest-cosmos-db-resource-provider-2024-05-15&tabs=HTTP
should work just fine with a bearer token and disable local auth. Why are the REST API's to list databases and containers not similarly able to function with a bearer token and disable local auth? It would also be a great feature for SDK CosmosClient to have the ability to list databases and containers, rather than having to write separate REST API code.

[daveoshinsky]

Note that 3.42.0 Container.ReadThroughputAsync also fails with disableLocalAuth:

3304 1554 08/26 17:00:52 2047664 allocating CosmosClient using MSI method
3304 1554 08/26 17:00:53 2047664 Container throughput failure with message 'Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 5300; ActivityId: 768e5b85-009c-4e1f-b35d-3fc728045d62; Reason: (Request blocked by Auth cosmosuscentral : The given request [POST /offers] cannot be authorized by AAD token in data plane. Learn more: https://aka.ms/cosmos-native-rbac.
ActivityId: 768e5b85-009c-4e1f-b35d-3fc728045d62, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, Windows/10.0.17763 cosmos-netstandard-sdk/3.34.4);
at Microsoft.Azure.Cosmos.GatewayStoreClient.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Cosmos.GatewayStoreClient.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Cosmos.GatewayStoreModel.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Azure.Cosmos.GatewayStoreModel.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Cosmos.Handlers.TransportHandler.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Cosmos.Handlers.TransportHandler.d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"GatewayCalls":{"(403, 5300)":1}},"name":"FeedIteratorCore ReadNextAsync","start datetime":"2024-08-26T17:00:53.250Z","duration in milliseconds":172.83870000000002,"children":[{"name":"QuerySpec to Stream","duration in milliseconds":19.4908},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":133.55100000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":119.6992,"data":{"System Info":{"systemHistory":[{"dateUtc":"2024-08-26T17:00:52.9702114Z","cpu":1.156,"memory":62011352.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32766,"minThreads":8,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":118.52690000000001,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":116.875,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":101.4478,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":100.1319,"children":[{"name":"Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request","duration in milliseconds":87.9444,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2024-08-26T17:00:53.3242080Z","DurationInMs":67.00030000000001,"RequestUri":"https://cosmosuscentral.documents.azure.com/offers","ResourceType":"Offer","HttpMethod":"POST","ActivityId":"768e5b85-009c-4e1f-b35d-3fc728045d62","StatusCode":"Forbidden","ReasonPhrase":"Forbidden"}]},"AuthProvider LifeSpan InSec":1.0610087,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"768e5b85-009c-4e1f-b35d-3fc728045d62","ResponseTimeUtc":"2024-08-26T17:00:53.4072071Z","StatusCode":403,"SubStatusCode":5300,"RequestCharge":0,"RequestUri":"//offers/","ErrorMessage":"Microsoft.Azure.Documents.DocumentClientException: Request blocked by Auth cosmosuscentral : The given request [POST /offers] cannot be authorized by AAD token in data plane. Learn more: https://aka.ms/cosmos-native-rbac.\r\nActivityId: 768e5b85-009c-4e1f-b35d-3fc728045d62, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, Windows/10.0.17763 cosmos-netstandard-sdk/3.34.4\r\n at Microsoft.Azure.Cosmos.GatewayStoreClient.d__9.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Cosmos.GatewayStoreClient.d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Cosmos.GatewayStoreModel.d__9.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at Microsoft.Azure.Cosmos.GatewayStoreModel.d__9.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":null,"BELatencyInMs":null}}}]}]}]}]}]}]},{"name":"Feed Response Serialization","duration in milliseconds":1.7765000000000002}]}'
3304 1554 08/26 17:00:53 2047664 DescribeCosmosDBContainer Exception Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 5300; ActivityId: 768e5b85-009c-4e1f-b35d-3fc728045d62; Reason: (Request blocked by Auth cosmosuscentral : The given request [POST /offers] cannot be authorized by AAD token in data plane. Learn more: https://aka.ms/cosmos-native-rbac.
ActivityId: 768e5b85-009c-4e1f-b35d-3fc728045d62, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, Windows/10.0.17763 cosmos-netstandard-sdk/3.34.4);
at Microsoft.Azure.Cosmos.GatewayStoreClient.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Cosmos.GatewayStoreClient.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Cosmos.GatewayStoreModel.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Azure.Cosmos.GatewayStoreModel.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Cosmos.Handlers.TransportHandler.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Cosmos.Handlers.TransportHandler.d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"GatewayCalls":{"(403, 5300)":1}},"name":"FeedIteratorCore ReadNextAsync","start datetime":"2024-08-26T17:00:53.250Z","duration in milliseconds":172.83870000000002,"children":[{"name":"QuerySpec to Stream","duration in milliseconds":19.4908},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":133.55100000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":119.6992,"data":{"System Info":{"systemHistory":[{"dateUtc":"2024-08-26T17:00:52.9702114Z","cpu":1.156,"memory":62011352.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32766,"minThreads":8,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":118.52690000000001,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":116.875,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":101.4478,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":100.1319,"children":[{"name":"Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request","duration in milliseconds":87.9444,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2024-08-26T17:00:53.3242080Z","DurationInMs":67.00030000000001,"RequestUri":"https://cosmosuscentral.documents.azure.com/offers","ResourceType":"Offer","HttpMethod":"POST","ActivityId":"768e5b85-009c-4e1f-b35d-3fc728045d62","StatusCode":"Forbidden","ReasonPhrase":"Forbidden"}]},"AuthProvider LifeSpan InSec":1.0610087,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"768e5b85-009c-4e1f-b35d-3fc728045d62","ResponseTimeUtc":"2024-08-26T17:00:53.4072071Z","StatusCode":403,"SubStatusCode":5300,"RequestCharge":0,"RequestUri":"//offers/","ErrorMessage":"Microsoft.Azure.Documents.DocumentClientException: Request blocked by Auth cosmosuscentral : The given request [POST /offers] cannot be authorized by AAD token in data plane. Learn more: https://aka.ms/cosmos-native-rbac.\r\nActivityId: 768e5b85-009c-4e1f-b35d-3fc728045d62, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, Windows/10.0.17763 cosmos-netstandard-sdk/3.34.4\r\n at Microsoft.Azure.Cosmos.GatewayStoreClient.d__9.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Cosmos.GatewayStoreClient.d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Cosmos.GatewayStoreModel.d__9.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at Microsoft.Azure.Cosmos.GatewayStoreModel.d__9.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":null,"BELatencyInMs":null}}}]}]}]}]}]}]},{"name":"Feed Response Serialization","duration in milliseconds":1.7765000000000002}]}
3304 1554 08/26 17:00:53 2047664 CosmosDBCoreIDA::CosmosDBCoreBackupCoordinator::initializeAllContainers() - Failed to describe cosmosuscentral/ToDoListres/bills with message 'Failed to describe container cosmosuscentral/ToDoListres/bills with message 'Response status code does not indicate success: Forbidden (403); Substatus: 5300; ActivityId: 768e5b85-009c-4e1f-b35d-3fc728045d62; Reason: (Request blocked by Auth cosmosuscentral : The given request [POST /offers] cannot be authorized by AAD token in data plane. Learn more: https://aka.ms/cosmos-native-rbac.
ActivityId: 768e5b85-009c-4e1f-b35d-3fc728045d62, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, Windows/10.0.17763 cosmos-netstandard-sdk/3.34.4);''

[daveoshinsky]

This issue should not have been closed. It remains an open problem. Both REST API to list databases and Container.ReadThroughputAsync (mentioned in the above comment) do not work at all with "disable local auth".

[daveoshinsky]

To clarify one thing - with the exact same user, MSI, RBAC, etc. but without "disable local auth", the above operations work properly. It is "disable local auth" specifically, that when added, causes all functionality mentioned above to stop working. It's likely that other functionality is also non-functional with "disable local auth", like "list containers in database" REST API as well as additional SDK method calls.

[kirankumarkolli]

@daveoshinsky its a service issue. This GitHub is for client issues only.
Can you please create a support ticket to follow-up?