[BUG] OpenSearch 2.11 401 errors even though valid Authorization header is sent

[ed-ud]

Describe the bug
After upgrade from 2.10 to 2.11, we are getting 401 errors with reprompt for login when using a browser to make requests.

To Reproduce
Steps to reproduce the behavior:

1. Go to any URL such as https://host.example.com:9200/_cat_nodes
2. Login (BasicAuth username/password - internal user)
3. Page is displayed
4. Refresh the page
5. User is re-prompted to login again
6. Page is displayed
7. Wait 5 minutes
8. Refresh the page
9. Page is displayed
10. Refresh the page
11. User is re-prompted to login again

Expected behavior
User should not be re-prompted for login over and over again.

Plugins
Stock set of plugins included with 2.11 "opensearch" package from repo:
https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/yum

Screenshots
N/A

Host/Environment (please complete the following information):

• OS: Rocky Linux
• Version: 9

Additional context
We have a 7-node cluster with 3 coord-only nodes and 4 dim nodes.
The problem didn't occur with 2.10... and started when the 2.11 update came in during a "yum upgrade".
I've reinstalled the cluster from scratch with 2.10 and the problem does not occur.
I've reinstalled the cluster from scratch with 2.11 and the problem occurs.
Using curl with username/password or with the admin cert works without issue in both 2.10 and 2.11.

When using Chrome and watching the requests, I can see the exact same Authorization header being sent during
a successful request, and when a request is made where the server returns a 401 error causing re-prompt for username/password.

I've tried both Chrome and Edge and they both see the same 401 errors.

It seems that if you wait for some number of minutes (5+) between page refreshes, you can avoid the bug, but that is not a suitable workaround. This oddity makes me think that there is some new "rate limit" sort of setting that is needed for 2.11.

[cwperks]

Hi @ed-ud , when using a browser to query opensearch directly, it will always challenge for a username and password since the URL does not contain username and password. After you fill out the prompt it will send the Authorization header with the request to OpenSearch. There is no session in the browser when querying opensearch directly, unlike when using OpenSearch dashboards and logging in with username and password. If you are using OpenSearch Dashboards, there is a Dev Tools section where you can make repeat requests against OpenSearch without having to re-login.

I don't believe the browser stores a cookie after you fill out the challenge box to remember the user on subsequent requests directly to <opensearch_host>:9200. Is it possible that there was a change in chrome that affected the behavior?

[ed-ud]

Yes, the browser does retain Basic Authentication credentials and submits an Authorization header with every request after login for the same URL or URL prefix, without re-prompting the user. That's just the way Basic Authentication works: https://datatracker.ietf.org/doc/html/rfc7617#section-2.2

As I said, the browser is sending the same Authorization header on successive requests, but in 2.11, OpenSearch ignores the valid header and replies with a 401 error (which causes the browser to re-prompt for credentials). OpenSearch 2.10 did not do this on successive requests with a valid Authorization header.

[scrawfor99]

[Triage] Hi @ed-ud, thank you for filing this issue. Looks like this could be an unintended change as part of 2.11 or that there is some other issue causing this behavior. Either way, someone will be able to look into this and hopefully identify the cause and fix.

Action items:

• Determine cause of behavior & whether it is expected
• Correct any introduced bugs

[ed-ud]

Any luck determining the cause of this bug?

[cwperks]

Could be related: opensearch-project/OpenSearch-Dashboards#5562 (comment)