Java+Selenium+Firefox proxy server cannot be set.

[smatei]

Hi,

I have a problem configuring Firefox to use proxy from selenium and java. I used a similar example with the code snippet you provided here http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp

Here is the code I use:

`
import java.io.File;
import java.io.IOException;

    import org.apache.commons.lang.SystemUtils;
    import org.openqa.selenium.Proxy;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.firefox.FirefoxBinary;
    import org.openqa.selenium.firefox.FirefoxDriver;
    import org.openqa.selenium.firefox.FirefoxProfile;
    import org.openqa.selenium.firefox.GeckoDriverService;
    import org.openqa.selenium.remote.CapabilityType;
    import org.openqa.selenium.remote.DesiredCapabilities;

    import com.google.common.collect.ImmutableMap;

    public class TestProxy
    {
      public static void main(String[] args)
      { 
        if (args.length == 0)
        {
          System.out.println("usage: IP port");
          System.exit(-1);
        }

        String proxyServer = args[0];
        int proxyPort = Integer.parseInt(args[1]);

        try
        {
          TestProxy test = new TestProxy();
          test.TestDriver(proxyServer, proxyPort);
        } catch (IOException e)
        {
          e.printStackTrace();
        }    
      }

      private void TestDriver(String aHost, int aPort) throws IOException
      {
        FirefoxProfile firefoxProfile = new FirefoxProfile();

        firefoxProfile.setEnableNativeEvents(true);

        WebDriver driver = null;
        GeckoDriverService service = null;

        if (SystemUtils.IS_OS_LINUX)
        {
          FirefoxBinary firefoxBinary = new FirefoxBinary(new File("/home/ubuntu/firefox/firefox"));

          service = new GeckoDriverService.Builder(firefoxBinary)
              .usingDriverExecutable(new File("/usr/bin/geckodriver"))
              .usingAnyFreePort()
              .usingAnyFreePort()
              .withEnvironment(ImmutableMap.of("DISPLAY",":20"))
              .build();
          service.start();      
        }
        else if (SystemUtils.IS_OS_WINDOWS)
        {
          FirefoxBinary firefoxBinary = new FirefoxBinary(new File("C:\\Program Files\\Mozilla Firefox\\firefox.exe"));

          service = new GeckoDriverService.Builder(firefoxBinary)
              .usingDriverExecutable(new File("C:\\Windows\\geckodriver.exe"))
              .usingAnyFreePort()
              .usingAnyFreePort()
              .build();
          service.start();      
        }
        else
        {
          System.out.println("Unknown operation system");
          System.exit(-1);
        }

        DesiredCapabilities cap = new DesiredCapabilities();

        Proxy proxy = new Proxy();

        String proxyServer = aHost + ":" + aPort;
        proxy.setHttpProxy(proxyServer)
             .setFtpProxy(proxyServer)
             .setSslProxy(proxyServer);
                   proxy.setProxyType(ProxyType.MANUAL);

        cap.setCapability(CapabilityType.PROXY, proxy);    

        driver = new FirefoxDriver(service, cap, null);

        driver.navigate().to("https://api.ipify.org/?format=text");

        System.out.println(driver.getPageSource());
      }
    }

`
Here are the environment details, on both windows and linux:

Selenium version: client-combined-3.0.0-beta4-nodeps.jar
Firefox version: Mozilla Firefox 49.0.1 (both on Windows and Linux)
Windows version: 8.1
Linux version: Ubuntu 15.04
geckodriver version: 0.11.1 (on both Windows and Linux)
java version: build 1.8.0_102-b14 on 64 bits (on both Windows and Linux)

The output on Windows is:

  1476874782195   geckodriver INFO    Listening on 127.0.0.1:14369
    Oct 19, 2016 1:59:42 PM org.openqa.selenium.remote.ProtocolHandshake createSession
    INFO: Attempting bi-dialect session, assuming Postel's Law holds true on the remote end
    log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
    log4j:WARN Please initialize the log4j system properly.
    1476874783017   mozprofile::profile INFO    Using profile path C:\Users\Stefan\AppData\Local\Temp\rust_mozprofile.j9tzNXSwtgD6
    1476874783019   geckodriver::marionette INFO    Starting browser C:\Program Files\Mozilla Firefox\firefox.exe
    1476874783021   geckodriver::marionette INFO    Connecting to Marionette on localhost:60470
    1476874783502   Marionette  INFO    Listening on port 60470
    1476874784363   Marionette  INFO    startBrowser 09aa9da8-d090-42af-be6d-3eec1ec1d4a2
    Oct 19, 2016 1:59:44 PM org.openqa.selenium.remote.ProtocolHandshake createSession
    INFO: Detected dialect: W3C
    <html><head><link rel="alternate stylesheet" type="text/css" href="resource://gre-resources/plaintext.css" title="Wrap Long Lines"></head><body><pre>8x.7x.22x.14x</pre></body></html>

Where 8x.7x.22x.14x is my local ip, not proxy IP.

Similar output on Linux:

  1476875492936   geckodriver     INFO    Listening on 127.0.0.1:22032
    Oct 19, 2016 11:11:33 AM org.openqa.selenium.remote.ProtocolHandshake createSession
    INFO: Attempting bi-dialect session, assuming Postel's Law holds true on the remote end
    log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
    log4j:WARN Please initialize the log4j system properly.
    1476875493387   mozprofile::profile     INFO    Using profile path /tmp/rust_mozprofile.9tk1zKafY0lI
    1476875493387   geckodriver::marionette INFO    Starting browser /home/ubuntu/firefox/firefox
    1476875493389   geckodriver::marionette INFO    Connecting to Marionette on localhost:38328
    (firefox:25100): GLib-GObject-CRITICAL **: g_object_ref: assertion 'object->ref_count > 0' failed
    (firefox:25100): GLib-GObject-CRITICAL **: g_object_unref: assertion 'object->ref_count > 0' failed
    (firefox:25100): GLib-GObject-CRITICAL **: g_object_ref: assertion 'object->ref_count > 0' failed
    (firefox:25100): GLib-GObject-CRITICAL **: g_object_unref: assertion 'object->ref_count > 0' failed
    1476875494041   Marionette      INFO    Listening on port 38328
    1476875495209   Marionette      INFO    startBrowser 1d7fe1dd-c381-4287-852d-1a2d87dc324e
    Oct 19, 2016 11:11:35 AM org.openqa.selenium.remote.ProtocolHandshake createSession
    INFO: Detected dialect: W3C
    (/home/ubuntu/firefox/plugin-container:25163): GLib-GObject-CRITICAL **: g_object_ref: assertion 'object->ref_count > 0' failed
    (/home/ubuntu/firefox/plugin-container:25163): GLib-GObject-CRITICAL **: g_object_unref: assertion 'object->ref_count > 0' failed
    (/home/ubuntu/firefox/plugin-container:25163): GLib-GObject-CRITICAL **: g_object_ref: assertion 'object->ref_count > 0' failed
    (/home/ubuntu/firefox/plugin-container:25163): GLib-GObject-CRITICAL **: g_object_unref: assertion 'object->ref_count > 0' failed
    <html><head><link rel="alternate stylesheet" type="text/css" href="resource://gre-resources/plaintext.css" title="Wrap Long Lines"></head><body><pre>5x.14x.2x.2xx</pre></body></html>

Where 5x.14x.2x.2xx is the ip of the Linux machine not the proxy.

Where is the problem? If I open the firefox "Connection Settings", the proxy option is "Use System Proxy Settings", not manual.

I have also tried a version with FirefoxProfile preferences, but no luck either.

firefoxProfile.setPreference("network.proxy.type", 1); firefoxProfile.setPreference("network.proxy.http", proxyServer); firefoxProfile.setPreference("network.proxy.http_port", proxyPort); firefoxProfile.setPreference("network.proxy.ssl", proxyServer); firefoxProfile.setPreference("network.proxy.ssl_port", proxyPort); firefoxProfile.setPreference("network.proxy.ftp", proxyServer); firefoxProfile.setPreference("network.proxy.ftp_port", proxyPort); firefoxProfile.setPreference("network.proxy.socks", proxyServer); firefoxProfile.setPreference("network.proxy.socks_port", proxyPort); firefoxProfile.setPreference("network.proxy.share_proxy_settings", true); firefoxProfile.setPreference("network.proxy.no_proxies_on", "localhost, 127.0.0.1");

[lukeis]

duplicate of mozilla/geckodriver#97

[smatei]

Hi @lukeis,

I have looked here mozilla/geckodriver#97

and in the geckodriver code

` GeckoDriver.prototype.setSessionCapabilities = function(newCaps) {
const copy = (from, to={}) => {
let errors = [];

        // Remove any duplicates between required and desired in favour of the
        // required capabilities
        if (from !== null && from.desiredCapabilities) {
          for (let cap in from.requiredCapabilities) {
            if (from.desiredCapabilities[cap]) {
              delete from.desiredCapabilities[cap];
            }
          }

          // Let's remove the sessionCapabilities from desired capabilities
          for (let cap in this.sessionCapabilities) {
            if (from.desiredCapabilities && from.desiredCapabilities[cap]) {
              delete from.desiredCapabilities[cap];
            }
          }
        }

        for (let key in from) {
          switch (key) {
            case "desiredCapabilities":
              to = copy(from[key], to);
              break;

            case "requiredCapabilities":
              if (from[key].proxy) {
                  this.setUpProxy(from[key].proxy);
                  to.proxy = from[key].proxy;
                  delete from[key].proxy;
              }
              for (let caps in from[key]) {
                if (from[key][caps] !== this.sessionCapabilities[caps]) {
                  errors.push(from[key][caps] + " does not equal " +
                      this.sessionCapabilities[caps]);
                }
              }
              break;

            default:
              to[key] = from[key];
          }
        }

        if (Object.keys(errors).length == 0) {
          return to;
        }

        throw new SessionNotCreatedError(
            `Not all requiredCapabilities could be met: ${JSON.stringify(errors)}`);
      };

      // clone, overwrite, and set
      let caps = copy(this.sessionCapabilities);
      caps = copy(newCaps, caps);
      logger.config("Changing capabilities: " + JSON.stringify(caps));
      this.sessionCapabilities = caps;
    };

`

It seems that it is looking for proxy only in 'requiredCapabilities'

` GeckoDriver.prototype.setUpProxy = function(proxy) {
logger.config("User-provided proxy settings: " + JSON.stringify(proxy));

      if (typeof proxy == "object" && proxy.hasOwnProperty("proxyType")) {
        switch (proxy.proxyType.toUpperCase()) {
          case "MANUAL":
            Preferences.set("network.proxy.type", 1);
            if (proxy.httpProxy && proxy.httpProxyPort){
              Preferences.set("network.proxy.http", proxy.httpProxy);
              Preferences.set("network.proxy.http_port", proxy.httpProxyPort);
            }
            if (proxy.sslProxy && proxy.sslProxyPort){
              Preferences.set("network.proxy.ssl", proxy.sslProxy);
              Preferences.set("network.proxy.ssl_port", proxy.sslProxyPort);
            }
            if (proxy.ftpProxy && proxy.ftpProxyPort) {
              Preferences.set("network.proxy.ftp", proxy.ftpProxy);
              Preferences.set("network.proxy.ftp_port", proxy.ftpProxyPort);
            }
            if (proxy.socksProxy) {
              Preferences.set("network.proxy.socks", proxy.socksProxy);
              Preferences.set("network.proxy.socks_port", proxy.socksProxyPort);
              if (proxy.socksVersion) {
                Preferences.set("network.proxy.socks_version", proxy.socksVersion);
              }
            }
            break;

          case "PAC":
            Preferences.set("network.proxy.type", 2);
            Preferences.set("network.proxy.autoconfig_url", proxy.proxyAutoconfigUrl);
            break;

          case "AUTODETECT":
            Preferences.set("network.proxy.type", 4);
            break;

          case "SYSTEM":
            Preferences.set("network.proxy.type", 5);
            break;

          case "NOPROXY":
          default:
            Preferences.set("network.proxy.type", 0);
            break;
        }
      } else {
        throw new InvalidArgumentError("Value of 'proxy' should be an object");
      }
    };`

So I have tried to pass them like this:

` DesiredCapabilities required = new DesiredCapabilities();

    JsonObject json = new JsonObject();
    json.addProperty("proxyType", "MANUAL");
    json.addProperty("httpProxy", aHost);
    json.addProperty("httpProxyPort", aPort);
    json.addProperty("sslProxy", aHost);
    json.addProperty("sslProxyPort", aPort);

    required.setCapability("proxy", json.toString());

    driver = new FirefoxDriver(service, cap, required);

`

but still no luck, I get some error.

{"error":"invalid argument","message":"Value of 'proxy' should be an object","stacktrace":""},null]

I will try to find out why.

[smatei]

Hi @lukeis,

Success! I was sending:

setCapability("proxy", json.toString());

Instead of this now I use successfully

setCapability("proxy", json);

That opens firefox asking me for the user and password for that proxy.

` DesiredCapabilities required = new DesiredCapabilities();

JsonObject json = new JsonObject();
json.addProperty("proxyType", "MANUAL");
json.addProperty("httpProxy", aHost);
json.addProperty("httpProxyPort", aPort);
json.addProperty("sslProxy", aHost);
json.addProperty("sslProxyPort", aPort);

required.setCapability("proxy", json);

driver = new FirefoxDriver(service, cap, required);

`

Now we have the prove that proxy works with Marionette and Firefox.

You should get rid of the webdriver example

` String PROXY = "localhost:8080";

    org.openqa.selenium.Proxy proxy = new org.openqa.selenium.Proxy();
    proxy.setHttpProxy(PROXY)
         .setFtpProxy(PROXY)
         .setSslProxy(PROXY);
    DesiredCapabilities cap = new DesiredCapabilities();
    cap.setCapability(CapabilityType.PROXY, proxy);
    WebDriver driver = new FirefoxDriver(cap);`

or at least mark it as deprecated and add the new example with Marionette desiredCapabilities.

[lukeis]

Feel free to submit a patch to the website itself, I would appreciate it :)

https://github.com/SeleniumHQ/www.seleniumhq.org

[gjundat95]

Work in Selenium 3.14.2, Firefox 62, C# .NET 4.5

FirefoxDriverService service = FirefoxDriverService.CreateDefaultService(@"GeckoDriver19", "geckodriver.exe");
service.FirefoxBinaryPath = @"C:\Program Files\Mozilla Firefox\firefox.exe";

    FirefoxOptions firefoxOptions = new FirefoxOptions();
    firefoxOptions.SetPreference("network.proxy.type", 1);
    firefoxOptions.SetPreference("network.proxy.socks", "127.0.0.1");
    firefoxOptions.SetPreference("network.proxy.socks_port", 1080);

    IWebDriver driver = new FirefoxDriver(service, firefoxOptions);

    driver.Navigate().GoToUrl("https://www.hbus.com/register");