OpenRefine 3.4.1: Reconciliation Service does not use proxy defined in openrefine.l4j.ini

[ToVie]

OpenRefine 3.4.1 on Windows (with or without integrated JRE) does not use the proxy set in openrefine.l4j.ini to connect to reconciliation service (https://lobid.org/gnd/reconcile)

To Reproduce

Steps to reproduce the behavior:

1. First, set proxy in openrefine.l4j.ini using the following parameters:
   -Dhttps.proxyHost=proxy-name.domain.ch
-Dhttps.proxyPort=8080
-Dhttp.proxyHost=proxy-name.domain.ch
-Dhttp.proxyPort=8080
-Dhttp.nonProxyHosts=localhost
2. Then, start OpenRefine using openrefine.exe, open project, select column "CDU - GND" for reconciliation
3. Then start reconciling using https://lobid.org/gnd/reconcile
4. Finally, OpenRefine throws the following error
   16:50:55.909 [ command] Failed to guess cell types for load {"q0":{"query":"Landquart (Fluss)","limit":3}} (0ms) org.apache.http.conn.HttpHostConnectException: Connect to lobid.org:443 [lobid.org/193.30.112.187] failed: Connection timed out:...

Current Results

The following messages are shown in OpenRefine (with verbosity "all"):

Additionally we did monitor the network traffic using wireshark and did find no connection from our localhost to our proxy but instead a connection request from our localhost to "emphytos.hbz-nrw.de" which bears the alias "lobid.org" with the IP 193.30.112.187.
This leads us to the conclusion that OpenRefine does not use the proxy settings specified in openrefine.l4j.ini.

Expected Behavior

OpenRefine connects to the Reconciliation service using the proxy specified in openrefine.l4j.ini using the parameters
-Dhttps.proxyHost=proxy-name.domain.ch
-Dhttps.proxyPort=8080
-Dhttp.proxyHost=proxy-name.domain.ch
-Dhttp.proxyPort=8080
-Dhttp.nonProxyHosts=localhost

Versions

• Operating System: Windows 10
• Browser Version: Edge or Firefox
• JRE or JDK Version: java version "1.8.0_281" Java(TM) SE Runtime Environment (build 1.8.0_281-b25) Java HotSpot(TM) 64-Bit Server VM (build 25.281-b25, mixed mode)
• OpenRefine: Version 3.4.1 [437dc4d]

Datasets

Dummy.openrefine.tar.gz

Additional context

OpenRefine runs behind a proxy that does not allow direct connections to the internet.
We tried both packages that are available for windows (windows with java and widows kit without java)
We also tested with the setting "-Djava.net.useSystemProxies=true", set the proxy-server in the java control panel and got the same results.
We also tried to run openrefine directly from the commandline and added the proxy parameters (we could not test refine.bat because the use of batch files is blocked by group policy).

[thadguidry]

@ToVie Hmm, curious, instead of through the object method (which there might indeed be a bug that we need to look into further where Apache HTTP client isn't picking them up for some reason)...
Can you try setting with a Windows environment variable at the system level (not user)?
http_proxy and https_proxy (lowercase should be fine for Apache HTTP Client library we use and should recognize them)
You can then verify that the proxy settings are passing through into OpenRefine's system Java environment by actually just doing a Custom text tranform on some column and just preview using GREL Clojure expression to pick them up:

(System/getenv "https_proxy")

where this Clojure expression lists all environment variables exposed to Java system:

(System/getenv)

[ToVie]

@thadguidry Thank you for your quick reply. Unfortunately I am not allowed to add or change environment variables on my office machine.
However I did as you suggested on a free test machine:

1. I added http_proxy and https_proxy as system variables
2. I tested (System/getenv "http_proxy") on a column => it works, I see the value of the variable

But I don't know how to set these as proxies to be used by OpenRefine.

[wetneb]

This is likely to have changed in the development version as we have switched to a different underlying HTTP library. Is it an option for you to try with a development snapshot? https://github.com/OpenRefine/OpenRefine-snapshot-releases/releases
Otherwise we will soon release 3.5-beta1.

[ToVie]

@thadguidry Your comment got me thinking and I did some further tests on OpenRefine reconciliation.

1. Windows proxy and -Djava.net.useSystemProxies=true with valid proxy
I tested with a working openly available test proxy (85.214.250.48:3128) set in windows system settings (manual/custom proxy configuration) and openrefine.l4j.ini with -Djava.net.useSystemProxies= set to true.
=> Reconciliation service works as expected

2. Windows proxy and -Djava.net.useSystemProxies=true with invalid proxy
I tested with a invalid proxy (127.0.0.1:1234) set in windows system settings (manual/custom proxy configuration) and openrefine.l4j.ini with -Djava.net.useSystemProxies= set to true.
=> Reconciliation service does not work as expected

3. Commented out -Djava.net.useSystemProxies=true and instead define invalid proxy in -Dhttp.proxyHost / -Dhttp.proxyPort and -Dhttps.proxyHost / -Dhttps.proxyPort
I added 127.0.0.1 and 1234 to the proxy settings, restarted openrefine.exe and ran reconciliation.
=> Expected behaviour: Reconciliation fails
=> Actual behaviour: Reconciliation succeeds

[ToVie]

@wetneb I try the development snapshot.

[ToVie]

@wetneb I tested the latest development snapshot as described above. The result is the same:

1. -Djava.net.useSystemProxies=true and define invalid proxy => reconciliation does not work - as it is expected
2. Commented out -Djava.net.useSystemProxies=true and instead define invalid proxy in -Dhttp.proxyHost / -Dhttp.proxyPort and -Dhttps.proxyHost / -Dhttps.proxyPort => reconciliation works although it should not

Unrelated to this issue there seems to be a problem with log4j in the development snapshot.

[thadguidry]

@ToVie I think the system properties are referring to classic Java 1.4 and not OS system (Windows). Much has changed with Java and defaults handling over the past 11 years since OpenRefine was first created.
This explains why system properties for proxy handling are not the best and why it changed after Java 1.5

3. Proxy class
   As we have seen, the system properties are powerful, but not flexible. The "all or nothing" behavior was justly deemed too severe a limitation by most developers. That's why it was decided to introduce a new, more flexible, API in Java SE 5.0 so that it would be possible to have connection based proxy settings.

https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html

---

@wetneb Looks like we might need to add the getProxy() to our RequestConfig.custom on
https://github.com/OpenRefine/OpenRefine/blob/master/main/src/com/google/refine/util/HttpClient.java#L69 ?

What's interesting also is the note on the DefaultRoutePlanner, which I didn't check on if it is indeed utilized through the call stack (you might check), which says:

Default implementation of an HttpRoutePlanner. It will not make use of any Java system properties, nor of system or browser proxy settings.

Then there's also the SystemDefaultRoutePlanner which says:

HttpRoutePlanner implementation based on ProxySelector. By default, this class will pick up the proxy settings of the JVM, either from system properties or from the browser running the application.

[ToVie]

@thadguidry I see your point. In our case, that is working in a restricted environment, it would be helpful to define a proxy that differs from the general setting on OS level. The general setting enforces a proxy with kerberos authentication based on user and token, which breaks a lot of java applications. Therefore an alternate proxy was defined that allows or disallows access based on the machine sending the request.

[wetneb]

Thanks for the thorough analysis!

Then this issue is pretty clear to me: one needs to make sure that our HttpClient class respects the custom proxy settings that we advertise (such as http.proxyPort and http.proxyHost).

I am marking this as a good first issue because @ToVie's diagnosis in #3919 (comment) should be easy to follow, to test a fix.