[🐛 Bug]: Selenium remote driver is giving error with proxy authentication

[pvsharma]

What happened?

I am trying to launch a browser in BitBAr from my IDE setup. I have to bypass the proxy and then connect to remote server. After doing some google & stackoverlow help I wrote this code. But it does not seem to work with Java 11 but it connects the remote browser with Java 8. Can someone please help as I have spent lot of time undertanding the issue byt failed to get the solution

Selenium version used - 4.21.1 (used previous ones also) Jave version (11)

The errors I get are

getMessage(): Could not start a new session. Unable to parse remote response:

org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Unable to parse remote response:

Caused by: org.openqa.selenium.json.JsonException: Expected to read a START_MAP but instead have: END. Last 0 characters read:

How can we reproduce the issue?

public static void main(String[] args) throws IOException, InterruptedException, NoSuchFieldException {
        MutableCapabilities capabilities = new MutableCapabilities();
        
        // SET PROXY DETAILS
        String proxyHost = "someurl.com";
        int proxyPort = 8080;
        String proxyUser = "proxyUSer";
        String proxyPassword = "proxyPassword";
        String remoteDesktopURL = "https://us-west-desktop-hub.bitbar.com/wd/hub/";
        Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort));
        ClientConfig config = null;

        // CONFIG TO ADD CREDENTIALS
        try {
            config = ClientConfig.defaultConfig()
                    .readTimeout(Duration.ofSeconds(90))
                    .authenticateAs(new UsernameAndPassword(proxyUser, proxyPassword))
                    .proxy(proxy)
                    .baseUrl(new URL(remoteDesktopURL));
        } catch (MalformedURLException e) {
            throw new RuntimeException(e);
        }

        capabilities.setCapability("platformName", "macOS");
        capabilities.setCapability("browserName", "chrome");
        capabilities.setCapability("browserVersion", "120");

        //Setting BITBAR capabilities
        HashMap<String, String> bitbarOptions = new HashMap<String, String>();
        bitbarOptions.put("apiKey", "someKey");
        bitbarOptions.put("osVersion", "12");
        bitbarOptions.put("resolution", "2560x1920");
        bitbarOptions.put("seleniumVersion", "4");
        bitbarOptions.put("bitbar_testTimeout", "1800");

        // Creating a RemoteWebDriver instance with Bitbar capabilities
        try{
            driver = RemoteWebDriver.builder()
                    .config(config)
                    .oneOf(capabilities)
                    .setCapability("bitbar:options", bitbarOptions)
                    .build();
        }
        catch (Exception e){
 
            e.printStackTrace();
        }

        driver.get("https://www.bbc.co.uk/");

Relevant log output

StackTrace: 
org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Unable to parse remote response:  
Host info: host: 'HOR135F.local', ip: 'fdde:c05c:634:210a:cbf:d3cb:7650:59cc%en0'
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:100)
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:75)
	at org.openqa.selenium.remote.RemoteWebDriverBuilder.getRemoteDriver(RemoteWebDriverBuilder.java:394)
	at org.openqa.selenium.remote.RemoteWebDriverBuilder.build(RemoteWebDriverBuilder.java:366)
	at org.example.WithProxyWithVPN.main(WithProxyWithVPN.java:88)
Caused by: org.openqa.selenium.json.JsonException: Unable to parse: 
	at org.openqa.selenium.json.Json.toType(Json.java:169)
	at org.openqa.selenium.json.Json.toType(Json.java:152)
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:96)
	... 4 more
Caused by: org.openqa.selenium.json.JsonException: Expected to read a START_MAP but instead have: END. Last 0 characters read: 
Build info: version: '4.21.0', revision: '79ed462ef4'
System info: os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '14.5', java.version: '11.0.18'
Driver info: driver.version: unknown
	at org.openqa.selenium.json.JsonInput.expect(JsonInput.java:467)
	at org.openqa.selenium.json.JsonInput.beginObject(JsonInput.java:333)
	at org.openqa.selenium.json.MapCoercer.lambda$apply$1(MapCoercer.java:64)
	at org.openqa.selenium.json.JsonTypeCoercer.lambda$buildCoercer$6(JsonTypeCoercer.java:171)
	at org.openqa.selenium.json.JsonTypeCoercer.coerce(JsonTypeCoercer.java:146)
	at org.openqa.selenium.json.Json.toType(Json.java:206)
	at org.openqa.selenium.json.Json.toType(Json.java:167)
	... 6 more
Exception in thread "main" java.lang.NullPointerException

Operating System

macOS

Selenium version

4.21

What are the browser(s) and version(s) where you see this issue?

BitBar Chrome

What are the browser driver(s) and version(s) where you see this issue?

Chrome 123

Are you using Selenium Grid?

4.0

[github-actions]

@pvsharma, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[titusfortner]

I'm not sure what you mean by it does not work in Java 11, but works in Java 8. Selenium no longer works with Java 8, so there are going to be more differences than just the Java version number.

You have an advanced setup and I can see 3 places it might be getting caught up.

1. The code isn't referencing the Proxy properly
2. The proxy forwards the command to the service provider which rejects it
3. The service provider creates a session but the designated driver errors

So you need to start with checking if the service provider is getting your request, and then work from there. Another thing off the bat, back sure you're specifying the latest version of Selenium in case the provider is giving you 4.0, which may cause problems.

[pvsharma]

Thank you Titus, I will stick to Java 11 for time being.

IS there anything wrong, I am passing the proxy and the credentials. I got these code from the previous ticket #10231 (comment) . Should I be changing anything on this.

Also BitBar(Service provided) is not getting my request

I did a debug and see that I get 407, so the proxy is not getting authenticated. What is wrong in the above section of the code (// CONFIG TO ADD CREDENTIALS) in which I am trying to authenticate proxy.

[titusfortner]

You're putting credentials both in the proxy object and in the client config directly. I think you just want it in proxy, so try removing authenticateAs

[pvsharma]

THank you Titus, I am not adding authentication in proxy but just adding it in config. If I remove AuthenticateAs from config, then proxy will not be authenticated.

[titusfortner]

You're right, and it looks like what you have matches what is supposed to work: #10231 (comment)

One of the other Java devs might need to help out with this one.

[Vishwas-web]

hey @pvsharma, Configure the proxy directly within the Proxy object instead of using ClientConfig for authentication and then try.

[pvsharma]

Thank you @Vishwas-web By just using the Proxy object, I cannot authenticate with credentials and it fails hence after going through various channels, found this solution which works with earlier version but fails on the latest version #10231 (comment)