Implement support for HTTP/HTTPS/SOCKS proxy usage

[ronaldtse]

From a user.

Curl allows this:

curl --proxy "https://user:pwd@x.x.x.x:8080" "https://httpbin.org/ip"

We can execute it with a proper user id, a password and a proxy address with Socks proxy.
Or, if HTTP_PROXY and HTTPS_PROXY environmental variables are set via .bashrc, curl works.

From @CAMOBAP :

Per https://ruby-doc.org/stdlib-2.6.3/libdoc/open-uri/rdoc/OpenURI.html
OpenURI use those variables

There are two known locations in the Metanorma/Relaton stack that accesses the internet.

metanorma-ietf:
https://github.com/metanorma/metanorma-ietf/blob/fce3ae41d3bb3e61573d233d5b9815226835cb59/lib/asciidoctor/ietf/validate.rb#L55-L78

Relaton accesses the Internet for references.

This work is to support HTTP/HTTPS/SOCKS proxy in this stack.

[CAMOBAP]

Well, I need to simulate the environment, I will prepare a docker image which:

• block all traffic and allow only a specific SOCKS proxy IP
• as far as I understand this can be done only via iptables manipulation

In progress

[CAMOBAP]

Steps:

• docker run -it --privileged metanorma/mn /bin/bash
• apt-get update -y && apt-get install iptables iputils-ping vim -y
• export SOCKS5_PROXY_HOST=88.99.14.224
• export SOCKS5_PROXY_PORT=1080
• export SOCKS_PROXY=socks5h://${SOCKS5_PROXY_HOST}:${SOCKS5_PROXY_PORT}
• iptables -A INPUT -s $SOCKS5_PROXY_HOST -j ACCEPT
• iptables -A OUTPUT -d $SOCKS5_PROXY_HOST -j ACCEPT
• iptables -P INPUT DROP
• iptables -P OUTPUT DROP
• env SOCKS_PROXY metanorma new -d standard -t csd csd-foo-standard

Trying to find reliable proxy right now, because

fatal: unable to access 'https://github.com/metanorma/mn-templates-csd/': Proxy CONNECT aborted

[CAMOBAP]

Right command is env HTTP_PROXY=socks5://$SOCKS5_PROXY_HOST:$SOCKS5_PROXY_PORT HTTPS_PROXY=socks5://$SOCKS5_PROXY_HOST:$SOCKS5_PROXY_PORT metanorma new -d standard -t csd csd-foo-standard-2

But once I drop all other traffic with iptables -P INPUT DROP && iptables -P OUTPUT DROP I get:

fatal: unable to access 'https://github.com/metanorma/mn-templates-csd/': Failed to connect to 212.129.28.76 port 44921: Connection timed out

It feels like I missing some settings in iptables, @ronaldtse maybe we have someone iptables/linux expert in our team?

[ronaldtse]

@CAMOBAP Checking if we can find someone to help here.

[CAMOBAP]

Asked on https://serverfault.com/questions/1069448/prove-that-software-works-via-socks

[CAMOBAP]

Per answer, ruby will not work such way this is why socksify gem was developed with it support looks much easier

http://rpm.repo.onapp.com/sources/rubygem-socksify-1.7.1/socksify-1.7.1/doc/

gem install socksify

Testing:

socksify_ruby $SOCKS5_PROXY_HOST $SOCKS5_PROXY_PORT /usr/local/bundle/bin/metanorma new -d standard -t csd csd-foo-standard

require 'socksify'
require 'open-uri'

Socksify::resolve("google.com")

[ronaldtse]

As you've seen the rename of the title, I'm not sure if the user really needs SOCKS. He said the proxy is SOCKS, but he can access by setting HTTPS_PROXY.

Let's try making HTTP_PROXY and HTTPS_PROXY work first by ensuring all network calls use URI.open?

The StackOverflow comments do not suggest that we cannot simulate the condition though.

[CAMOBAP]

He said the proxy is SOCKS, but he can access by setting HTTPS_PROXY.

@ronaldtse he can do this for curl for metanorma too?

According to my information socks proxy will not work for HTTP[S]_PROXY in case of ruby

The StackOverflow comments do not suggest that we cannot simulate the condition though.

Finally I was able to simulate test conditions, it fails on git calls, I'm trying a workaround right now

[ronaldtse]

@CAMOBAP That's why I suspect that his corporate proxy provides both HTTP/HTTPS proxy and SOCKS access (he mentioned SOCKS specifically).

If we only use URI.open with those env vars perhaps it would work...

[CAMOBAP]

Well to many assumptions, but still let me reply to some of your thoughts

As you've seen the rename of the title, I'm not sure if the user really needs SOCKS. He said the proxy is SOCKS, but he can access by setting HTTPS_PROXY.

According to the original curl --proxy "https://user:pwd@x.x.x.x:8080" "https://httpbin.org/ip" form user he clearly uses HTTPS proxy. I sure about this because for SOCKS proxy URL schema must be different any of:

• socks:// - for SOCKS4
• socks5:// - for SOCKS5
• socks5h:// - for SOCKS5 with host resolving

Let's try making HTTP_PROXY and HTTPS_PROXY work first by ensuring all network calls use URI.open?

For HTTP/HTTPS proxy there is nothing that needs to be done, it just works out-of-the-box. Both Net::HTTP and open-uri supports HTTP[S]_PROXY environment variables

That's why I suspect that his corporate proxy provides both HTTP/HTTPS proxy and SOCKS access (he mentioned SOCKS specifically).

It possible to keep all proxies on a single host but I'm not sure that they able to share a single port, and as I noted above user uses HTTPS proxy for curl

Ruby doesn't support SOCKS proxy passed in HTTP[S]_PROXY, example:

require 'open-uri'

URI.open('http://www.example.org/').read

and env HTTP_PROXY=socks5://$SOCKS5_PROXY_HOST:$SOCKS5_PROXY_PORT ruby test_open_simple.rb finishes with:

/usr/local/bundle/gems/open-uri-0.1.0/lib/open-uri.rb:258:in `open_http': Non-HTTP proxy URI: socks5://212.129.25.57:44921 (RuntimeError)
	from /usr/local/bundle/gems/open-uri-0.1.0/lib/open-uri.rb:741:in `buffer_open'
	from /usr/local/bundle/gems/open-uri-0.1.0/lib/open-uri.rb:212:in `block in open_loop'
	from /usr/local/bundle/gems/open-uri-0.1.0/lib/open-uri.rb:210:in `catch'
	from /usr/local/bundle/gems/open-uri-0.1.0/lib/open-uri.rb:210:in `open_loop'
	from /usr/local/bundle/gems/open-uri-0.1.0/lib/open-uri.rb:151:in `open_uri'
	from /usr/local/bundle/gems/open-uri-0.1.0/lib/open-uri.rb:721:in `open'
	from /usr/local/bundle/gems/open-uri-0.1.0/lib/open-uri.rb:29:in `open'
	from test_open_simple.rb:3:in `<main>'

What needs to be done from our side

• we use git for templates download, and for font download and need to configure proxy for it explicitly Clone through proxy ruby-git/ruby-git#512
• for SOCKS we need to use socksify gem and introduce handling of SOCKS_PROXY variable
• maybe anything else to be continued...

[CAMOBAP]

Also related to this ruby-git/ruby-git#531 (not blocking but may fails for non master branches

[CAMOBAP]

Sometimes socks_authenticate': Server doesn't reply authentication (SOCKSError) happens researching why

[CAMOBAP]

PRs prepared, once we will agreed on open questions, I will prepare post on metanorma.com

[CAMOBAP]

Done, and released in 1.4.12