[🐛 Bug]: 401 error with new http client while creating RemoteWebDriver with URL that uses basic auth (e.g. Selenoid GGR)

[dbudim]

What happened?

Received 401 error with new http client while creating RemoteWebDriver with host with basic auth (e.g. Selenoid GGR hub with quotas - http://user:pass@my.awesome.hub)

It happened if enabled jdk-http-client
https://www.selenium.dev/blog/2022/using-java11-httpclient/

Without System.setProperty("webdriver.http.factory", "jdk-http-client"); works fine

How can we reproduce the issue?

- Enable new http client https://www.selenium.dev/blog/2022/using-java11-httpclient/
- Try start RemoteWebDriver with URL with basic auth `http://user:pass@my.awesome.hub`

Relevant log output

Caused by: org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Unable to parse remote response: 401 Unauthorized
 
Host info: host: '28504ca6c95d', ip: '172.17.0.5'
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:130)
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:102)
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:67)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:156)
	at org.openqa.selenium.remote.TracedCommandExecutor.execute(TracedCommandExecutor.java:51)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:547)
	at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:242)
	at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:157)
	at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:139)
	at qa.airslate.selenium.core.browsers.hub.SelenoidBrowser.setupDriver(SelenoidBrowser.java:62)
	... 27 more
Caused by: org.openqa.selenium.json.JsonException: Unable to parse: 401 Unauthorized

	at org.openqa.selenium.json.Json.toType(Json.java:57)
	at org.openqa.selenium.json.Json.toType(Json.java:50)
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:127)
	... 36 more
Caused by: org.openqa.selenium.json.JsonException: Expected to read a START_MAP but instead have: NUMBER. Last 0 characters read: 
Build info: version: '4.5.0', revision: 'fe167b119a'
System info: os.name: 'Linux', os.arch: 'aarch64', os.version: '4.14.214-160.339.amzn2.aarch64', java.version: '11.0.10'
Driver info: driver.version: RemoteWebDriver
	at org.openqa.selenium.json.JsonInput.expect(JsonInput.java:307)
	at org.openqa.selenium.json.JsonInput.beginObject(JsonInput.java:229)
	at org.openqa.selenium.json.MapCoercer.lambda$apply$1(MapCoercer.java:64)
	at org.openqa.selenium.json.JsonTypeCoercer.lambda$buildCoercer$6(JsonTypeCoercer.java:145)
	at org.openqa.selenium.json.JsonTypeCoercer.coerce(JsonTypeCoercer.java:127)
	at org.openqa.selenium.json.Json.toType(Json.java:71)
	at org.openqa.selenium.json.Json.toType(Json.java:55)
	... 38 more

Operating System

linux

Selenium version

4.5.0

What are the browser(s) and version(s) where you see this issue?

104

What are the browser driver(s) and version(s) where you see this issue?

RemoteWebDriver

Are you using Selenium Grid?

no

[github-actions]

@dbudim, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[ldiary]

I solved the same issue by modifying the getPasswordAuthentication() part of JdkHttpClient with a replacement code below:

      Authenticator authenticator = new Authenticator() {
        @Override
        protected PasswordAuthentication getPasswordAuthentication() {
          if (getRequestorType() == RequestorType.SERVER) {
            String info = config.baseUri().getUserInfo();
            if (!Strings.isNullOrEmpty(info)) {
              String[] parts = info.split(":", 2);
              String username = parts[0];
              String password = parts.length > 1 ? parts[1] : null;
              return new PasswordAuthentication(username, password.toCharArray()
              );
            }
          }
          return new PasswordAuthentication(uap.username(), uap.password().toCharArray());
        }
      };

I was thinking it must have been handled correctly by AuthenticationFilter but for some reason, in my case, it always picks up the credentials for the proxy instead of the credentials for the server.

[pujagani]

Thank you for sharing the details and providing an alternate solution. I have used the same and it was also a part of existing asynchttpclient support. I have tested the changes to fix this in the commit 101b42d locally using the Grid's "username" and "password" command line parameters and run the following code snippet. Before the changes, I was able to reproduce the error.

 public static void main(String[] args) throws MalformedURLException, InterruptedException {
        System.setProperty("webdriver.http.factory", "jdk-http-client");
        WebDriver driver = new RemoteWebDriver(new URL("http://selenium:grid@localhost:4444"), new ChromeOptions());
        driver.get("https://www.google.com");
        driver.manage().window().maximize();
        driver.quit();
    }

[pujagani]

@dbudim Can you please try out the latest release of Selenium Java binding and provide feedback if it solves the problem? Accordingly, we can close this issue.

[dbudim]

In 4.6.0 new error😅
Seems new changes in JDK Http client about buffering affected that

Caused by: org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure. 
Host info: host: '19f8fa5bcfd1', ip: '172.17.0.5'
Build info: version: '4.6.0', revision: '79f1c02ae20'
System info: os.name: 'Linux', os.arch: 'aarch64', os.version: '4.14.243-185.433.amzn2.aarch64', java.version: '11.0.10'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, browserVersion: 106, enableVNC: true, enableVideo: false, goog:chromeOptions: {args: [--no-sandbox, --disable-webgl, --disable-extensions, --disable-gpu, --kiosk, --user-agent=Mozilla/5.0 (X...], extensions: []}, goog:loggingPrefs: org.openqa.selenium.logging..., name: airSlate (HF): airSlate_UI_..., pageLoadStrategy: none, sessionTimeout: 5m, videoName: airSlate_HF_Page__Login_166...}], desiredCapabilities=Capabilities {browserName: chrome, browserVersion: 106, enableVNC: true, enableVideo: false, goog:chromeOptions: {args: [--no-sandbox, --disable-webgl, --disable-extensions, --disable-gpu, --kiosk, --user-agent=Mozilla/5.0 (X...], extensions: []}, goog:loggingPrefs: org.openqa.selenium.logging..., name: airSlate (HF): airSlate_UI_..., pageLoadStrategy: none, sessionTimeout: 5m, videoName: airSlate_HF_Page__Login_166...}}]
Capabilities {}
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:559)
	at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:242)
	at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:157)
	at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:139)
	at qa.airslate.selenium.core.browsers.hub.SelenoidBrowser.setupDriver(SelenoidBrowser.java:62)
	... 26 more
Caused by: java.io.UncheckedIOException: java.io.IOException: SocketTube(3) [HttpClient-1-Worker-2] Too few bytes returned by the publisher (0/1550)
	at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:284)
	at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:122)
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:101)
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:67)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:156)
	at org.openqa.selenium.remote.TracedCommandExecutor.execute(TracedCommandExecutor.java:51)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:541)
	... 30 more
Caused by: java.io.IOException: SocketTube(3) [HttpClient-1-Worker-2] Too few bytes returned by the publisher (0/1550)
	at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:565)
	at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
	at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:280)
	... 37 more
Caused by: java.io.IOException: SocketTube(3) [HttpClient-1-Worker-2] Too few bytes returned by the publisher (0/1550)
	at java.net.http/jdk.internal.net.http.Http1Request$FixedContentSubscriber.onComplete(Http1Request.java:450)
	at java.net.http/jdk.internal.net.http.PullPublisher$Subscription$PullTask.run(PullPublisher.java:115)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
	at java.net.http/jdk.internal.net.http.PullPublisher$Subscription.request(PullPublisher.java:130)
	at java.net.http/jdk.internal.net.http.Http1Exchange$Http1BodySubscriber.request(Http1Exchange.java:135)
	at java.net.http/jdk.internal.net.http.Http1Exchange.requestMoreBody(Http1Exchange.java:537)
	at java.net.http/jdk.internal.net.http.Http1Exchange.lambda$sendBodyAsync$8(Http1Exchange.java:328)
	at java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
	at java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
	at java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)
	at java.net.http/jdk.internal.net.http.Http1Exchange.sendBodyAsync(Http1Exchange.java:328)
	at java.net.http/jdk.internal.net.http.Exchange.sendRequestBody(Exchange.java:410)
	at java.net.http/jdk.internal.net.http.Exchange.checkFor407(Exchange.java:367)
	at java.net.http/jdk.internal.net.http.Exchange.lambda$responseAsyncImpl0$7(Exchange.java:436)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:153)
	at java.base/java.util.concurrent.CompletableFuture.completeAsync(CompletableFuture.java:2591)
	at java.net.http/jdk.internal.net.http.Http1Exchange.getOutgoing(Http1Exchange.java:586)
	at java.net.http/jdk.internal.net.http.Http1Exchange$Http1Publisher$WriteTask.run(Http1Exchange.java:663)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:153)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:273)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:242)
	at java.net.http/jdk.internal.net.http.Http1Exchange$Http1Publisher$Http1WriteSubscription.request(Http1Exchange.java:703)
	at java.net.http/jdk.internal.net.http.SocketTube$InternalWriteSubscriber$WriteSubscription.requestMore(SocketTube.java:503)

[pujagani]

Thank you for sharing the feedback. I do not see when Grid is used without basic auth. The JDK HTTP Client works as expected. However, when I set up the Grid using basic auth and the same on the client side, I see the errors shared above. I will triage it further today.