HTTP Proxy for outbound requests

[JonTheNiceGuy]

Further to the email discussion with @kentonv:

Currently there are several places in Sandstorm's code where we use Node's "http" and "https" modules to make outgoing HTTP requests.

It looks like the following files use these modules:

shell/server/sandcats.js: To talk to the Sandcats server for dynamic DNS updates, if you use sandcats.
shell/server/installer.js: To download packages.
shell/server/hack-session.js: To implement the httpGet() API method, which e.g. TinyTinyRSS uses to fetch RSS feeds.
shell/server/drivers/external-ui-view.js: To implement a more general mechanism for apps to make outgoing HTTP requests (not yet used by any apps).

And this file uses Meteor's HTTP API:

shell/packages/sandstorm-db/user.js: To fetch profile pictures from Google and Github.

Meteor's HTTP is a wrapper around the request npm package, which in turn wrap's node's HTTP client. It looks like request explicitly supports proxies (and Meteor's HTTP supports passing arbitrary options to request):

https://www.npmjs.com/package/request#proxies

It's also possible to get Node's raw HTTP client to use proxies per this StackOverflow question:

http://stackoverflow.com/questions/3862813/how-can-i-use-an-http-proxy-with-node-js-http-client

So we need to make sure that all the places where we make HTTP requests use the proxy.

I think the right way to do this would be to introduce a new Sandstorm-internal HTTP API that either wrap's Meteor's HTTP or the npm request package (probably the latter) and allows us to force all requests globally through a porxy, configurable through the admin interface. Then we should update all the above code to use it.

Additionally, run-bundle.c++ shells out to curl to fetch Sandstorm updates. I believe you can make curl use a proxy by setting an environment variable, so if you set that variable in the initscript that calls sandstorm start it should apply. (Maybe it makes sense for the Sandstorm shell to pick up the http_proxy environment variable automatically as well, rather than require setting it in the admin settings?)

[A-Picogna]

Hi,

I also have a problem with proxy, there is a way to add it to sandstorm for the moment ? to force the plate-forme going through it ?

[paulproteus]

Hi @A-Picogna ,

It's your lucky week! I'm testing this right now and one of my goals for the week is to make sure that this works.

I'm going to leave a few comments here about how I test this, so that I can test it again more easily in the future.

[paulproteus]

Goal: Run a Sandstorm process on my laptop with no access to the actual Internet, except via a HTTP proxy

Strategy:

• Install Sandstorm and make sure it uses the username sandstorm

grep SERVER_USER= /opt/sandstorm/sandstorm.conf

• Disable wifi connectivity for that user

iptables -A OUTPUT -o wlp6s0 -m owner --uid-owner sandstorm -j DROP
sudo su - sandstorm -c 'cd /tmp ; wget http://www.google.com/'  # make sure this fails

• Install a HTTP proxy

sudo apt install squid

• Test the HTTP proxy

sudo su - sandstorm -c 'cd /tmp ; https_proxy=http://localhost:3128/ http_proxy=http://localhost:3128/ wget https://www.google.com/'  # this should succeed now

Great. Next up, adding it to Sandstorm.

(Note: wget requires lowercase http_proxy and doesn't use http_proxy for https:// URLs so requires both environment variables set.)

[paulproteus]

• Added it to Sandstorm by editing /etc/systemd/system/sandstorm.service to say:

[Unit]
Description=Sandstorm server
After=local-fs.target remote-fs.target network.target
Requires=local-fs.target remote-fs.target network.target

[Service]
Type=forking
ExecStart=/opt/sandstorm/sandstorm start
ExecStop=/opt/sandstorm/sandstorm stop
Environment=http_proxy=http://127.0.0.1:3128/
Environment=https_proxy=http://127.0.0.1:3128/

[Install]
WantedBy=multi-user.target

[paulproteus]

• Restarted Sandstorm by doing:

sudo systemctl daemon-reload  # to get the new sandstorm.service into systemd's mind
sudo service sandstorm restart

[paulproteus]

• Install an app
  • Visit http://local.sandstorm.io:6080/
  • (Note: I'm currently logged in as Alice Admin)
  • Click Install ➡️ https://apps.sandstorm.io/?host=http://local.sandstorm.io:6080 (success) ➡️ http://local.sandstorm.io:6080/install/a33e26d3b0fa40c658e2235dab438f1a?url=https://app-index.sandstorm.io/packages/a33e26d3b0fa40c658e2235dab438f1a (success) ➡️ see message that says "Failed to install app getaddrinfo ESRCH" (failure)

[paulproteus]

I'm working on this right now, so self-assigning.

[A-Picogna]

Hi @paulproteus

I'll be very interested by your results ^^. I have a distant machine on an internal network which need to get through a proxy to acces internet.

the command : systemctl daemon-reload dosen't exist on ubuntu 14.04 apparently :)

[paulproteus]

Per #1438 I have app install working with a proxy now. Not everything is ready though, and this isn't in a released Sandstorm build yet.

[kentonv]

Github auto-closed this because it saw the sentence "does not fully close #693" and only paid attention to the "close #693" part. But this is not actually done yet, so re-opening.

[paulproteus]

Thanks @kentonv for fixing the metadata fail!

[paulproteus]

Next up - test/fix the following:

• shell/server/sandcats.js: To talk to the Sandcats server for dynamic DNS updates, if you use sandcats.
• shell/server/installer.js: To download packages.
• shell/server/hack-session.js: To implement the httpGet() API method, which e.g. TinyTinyRSS uses to fetch RSS feeds.
• shell/server/drivers/external-ui-view.js: To implement a more general mechanism for apps to make outgoing HTTP requests (not yet used by any apps).
• And this file uses Meteor's HTTP API: shell/packages/sandstorm-db/user.js: To fetch profile pictures from Google and Github.
• sudo sandstorm update

[mitar]

Could we also allow HTTP POST requests? This would allow apps to do bot POST requests in the name of the user. For me Sandstorm is really about making personal apps and such apps often also want to automatize some tasks. Sending an outbound HTTP POST is pretty common.

[kentonv]

@mitar This issue is not about grains doing HTTP requests, it's about making the Sandstorm shell's HTTP requests go through a proxy (as is required on e.g. many corp networks).

[mitar]

Oh, sorry. I misread. I read "To implement the httpGet() API method, which e.g. TinyTinyRSS uses to fetch RSS feeds." Is there a more appropriate ticket? Or should I open a new one?

[kentonv]

@mitar Dunno if there is a ticket. Secretly you can do posts today using HackSessionContext::getUiViewForEndpoint() and passing a regular old URL (rather than a webkey) and then requesting a WebSession. However, all these APIs will be replaced soon with Powerbox APIs (which will definitely support POST). Feel free to open a ticket or don't, either way it'll get fixed just as fast.

[mitar]

Opened #1924.

[paulproteus]

I haven't done any work on this lately, so I'm de-assigning from myself for now so that no one thinks I am.