Add behaviors to enable setting strictSSL option for file downloads #2212

[mluisbrown]

Implements #2212.

@kenny-evitt as I didn't hear anything more from you on the issue, I went ahead and implemented my suggestion. Let me know what you think.

[kenny-evitt]

@mluisbrown Thanks for the PR! The changes look fine.

Does this work with the current version of the request package?

One problem the rest of the core team might have with these changes is not having a way for us to personally QA them. We've run into the same problem with the change to bump the version of the request package (#2208). If you could help us, or anyone, configure a proxy on our computers that simulates or is equivalent to your proxy setup, then we could QA the changes ourselves and merge the changes right away.

[kenny-evitt]

@cldwalker @rundis Thoughts?

[mluisbrown]

Does this work with the current version of the request package?

@kenny-evitt unfortunately no. I had to update the request package to 2.72.0 to get this to work. Proxy support generally seems to be broken in the current version (2.14.0).

If you're running OS X you can use Charles Proxy as a local proxy and also configure it to use a MITM SSL certificate to intercept and decrypt SSL traffic, which accurately simulates the environment this PR is trying to address.

Charles proxy automatically sets itself up running on http://127.0.0.1:8888. Because of the hoops you have to jump through to get environment variables picked up by GUI apps launched the normal way in OS X, it's easier if you set them in the shell and then launch LT from the shell:

• export http_proxy=http://127.0.0.1:8888
• export https_proxy=http://127.0.0.1:8888
• open -a <path to LightTable.app>

Also, I found that I had to disabled the "Menu -> Proxy -> Mac OS X Proxy" option in Charles. This makes Charles run as the OS X system proxy for all network requests, but we want to only proxy requests explicitly sent via 127.0.0.1:8888.

Once you've configured Charles to intercept SSL traffic to api.github.com:443 if you launch LT as above and force a refresh of the plugin repo, you will see a request to api.github.com in Charles which is unsuccessful. If you add the [:app :lt.objs.deploy/disable-strict-ssl] behavior to user.behaviors and try the same thing again, you should see the request succeed, and see it decrypted in the Charles window.

If you're running on Windows you should be able to user Fiddler instead of Charles to achieve something similar (it also can do SSL intercepts).

[kenny-evitt]

@mluisbrown Thanks. I'll try to give setting-up a proxy another go sometime soonish.

[mluisbrown]

@kenny-evitt let me know if you need any help setting up a proxy. With Charles or Fiddler (on OS X or Windows respectively) it's really just a question of downloading and running the proxy. They both auto-configure to 127.0.0.1:8888 and as soon as you quit the app your system is back to normal.
As well as QAing this PR I was also able to reproduce the problem and validate the fix in #2208.

[kenny-evitt]

@mluisbrown Help would be extremely appreciated. I think I can figure out how to download and install Charles or Fiddler, but detailed instructions on setting-up a proxy would be great.

We'd also need to do the same on Linux, but just being able to reliably QA this (and similar) issues would be a huge step forward.

[cldwalker]

@mluisbrown Thanks for the PR. Seems reasonable given your use case. As @kenny-evitt mentioned detailed, instructions to setting up a proxy would be great as it enables us to QA and support this feature going forward

@kenny-evitt unfortunately no. I had to update the request package to 2.72.0 to get this to work. Proxy support generally seems to be broken in the current version (2.14.0).

If proxying functionality requires 2.72.0 we should pull that into this PR. Alternatively, since there is only one place where we use request, we could switch to the standard http lib

[mluisbrown]

@kenny-evitt and @cldwalker: I will post detailed instructions for setting up and configuring a proxy here as soon as I can.

If proxying functionality requires 2.72.0 we should pull that into this PR. Alternatively, since there is only one place where we use request, we could switch to the standard http lib

I will see how easy it is to configure proxy support (for both http and https) with the standard http lib. It would appear that it's not totally straightforward.

[mluisbrown]

Here are the steps required to setup a proxy on OS X:

Setup Proxy

1. Download and install Charles Proxy. It has a free trial and becomes nag-ware after that, so should be fine for this purpose.
2. Launch Charles.
3. There is no step 3 😄 (no really, you now have a proxy running on 127.0.0.1:8888).
4. Ok, there are some more small details:
5. From the Charles menu: Proxy -> Mac OS X Proxy to ensure that Charles doesn't insert itself into the OS X network stack. This way only apps that explicitly set a proxy to 127.0.0.1:8888 will use it:
   

Launch Light Table with proxying enabled

As it's a little fiddly to set environment variables in OS X for GUI apps, it's easier for the purpose of testing set the proxy environment variables on the command line and launch Light Table from the command line (so that it picks them up):

$ export http_proxy=http://127.0.0.1:8888
$ export https_proxy=http://127.0.0.1:8888
$ open -a <path to LightTable.app>

Now you can try updating the Plugin Metadata (Plugins: Show plugin manager or Plugins: Refresh plugin list). If it works and you see the traffic (well, the CONNECT requests at least) in the Charles window, you know that the proxying is working.

Setup SSL proxying to simulate MITM certificate

1. In the Charles menu Proxy -> SSL Proxy Settings:
   
2. Enable SSL proxying and add api.github.com:443 as a host / port you want to monitor / decrypt:
   

And click OK.

1. Now if you launch LT as above and try to download the Plugin Metadata it won't work. You will see SSL failures in the Charles log.
2. Now add [:app :lt.objs.deploy/disable-strict-ssl] to the User Behaviours file to disable certificate checking.
3. Restart Light Table.
4. Now, with the SSL proxying still enabled, try downloading the Plugin Metadata. It should now work, and you should see the decrypted traffic to https://api.github.com in the Charles log.

@kenny-evitt let me know if this is detailed enough or if you need any more info.

@cldwalker I looked into the feasibility of using the standard http lib instead of the request package and I could not get it to work for HTTPS via a proxy, even without a bogus certificate chain. I think we will have to upgrade the request package to 2.72.0. #2208 was already created to do this. My GitHub fu is not advanced enough to know how to merge two PRs into one...

[kenny-evitt]

@mluisbrown I'll try this out and let you know if it works for me.

As for merging PRs, we can just leave them separate for now. But if we wanted to merge, just fetch the relevant branches (i.e. you fetch the branch from my fork, or vice versa), merge (locally), and then push the new version of the branch of your fork (or mine) to GitHub (and optionally force the push if you had to git rebase or otherwise rewrite history or replace commits).

[sbauer322]

@mluisbrown I seem to be running into a bit of trouble when trying to get Light Table to play nicely with the proxy environment variables needed for Charles to pick up its activity. On Ubuntu, when I set everything up as directed, an error is thrown about unescaped characters when forcing LT to retrieve the plugin metadata... It does not appear that LT successfully makes a connection before the error since Charles shows no activity. However, if I was to unset the proxy environment variables then it will function normally (and not go through Charles).

Did you encounter anything like this?

[mluisbrown]

@sbauer322 that is the symptom of the broken proxy support in v2.14.0 of the request npm package that is used by LT for making HTTP requests. See the comments in #1984.

Getting proxy support to work at all requires updating the version of the request package to at least 2.72.0. There was a separate PR for this (#2208) but it seems to have been closed. The problem is that updating one package brings in a whole bunch of other npm package changes.

This PR is specifically about adding support for disabling strict SSL support when using a proxy, but that is only meaningful if proxy support is working.

[sbauer322]

Ah, yes - looks like I had checked out your branch rather than merging it into my branch with the updated request package. Sorry for the confusion! 😨

Will give this another try tonight.

[sbauer322]

Yep, works perfectly with the updated request package, @mluisbrown. As far as I can tell, this PR is ready to be merged once the request package is updated.

I have opened a PR for updating the request package, PR #2331.

[mluisbrown]

Awesome! And thanks for opening #2231 @sbauer322 👍

[sbauer322]

@LightTable/maintainers - I will be merging this after 2017-04-29 unless there is concerns or feedback.

[mluisbrown]

🎉 Thanks Scott 🎉