Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NVDA and add-ons are not update when a http proxy is used by the operating system #8948

Closed
sonor3000 opened this issue Nov 15, 2018 · 26 comments

Comments

@sonor3000
Copy link

I am using the latest stable NVDA release on Win7 in an environment where a http proxy has to be used to access the internet. Unfortunately with this system config no automated updates for NVDA or installed add-ons can be downloaded.

Is this a known issue?
Can it be resolved somehow?
Can the update mechanism of NVDA be reimplemented to also take care of http proxy settings of the operating system, the default browser e.g., or can the proxy settings be made configurable?

@josephsl
Copy link
Collaborator

josephsl commented Nov 15, 2018 via email

@jscholes
Copy link

This isn't much of a mystery. urllib2 doesn't automatically fetch proxy information from the OS, and NVDA doesn't add it. It's not a technical failure, it's a feature which just doesn't exist. The WinHttpGetIEProxyConfigForCurrentUser function can be used to get the information. Other decisions should also be made, such as:

  1. What sort of proxies should NVDA support? Just HTTP? HTTp and SOX5?
  2. In addition to fetching the proxy config from the OS, should a panel be added to the NVDA settings dialog to turn this off or enter proxy information for NVDA specifically?

@jscholes
Copy link

Should we also advise add-on authors to use any configured proxy when making HTTP requests? Or provide a preconfigured HTTP client of some sort that add-ons can import and use, knowing that the correct network config will be in place?

@jcsteh
Copy link
Contributor

jcsteh commented Nov 16, 2018 via email

@josephsl
Copy link
Collaborator

josephsl commented Nov 16, 2018 via email

@mohdshara
Copy link

hi. @jcsteh you're right, we discussed this before #7893
we need to mark one of those as duplicate. Thanks.

@marlon-sousa
Copy link
Contributor

Hello,

I would like to take this issue.

I have no such configuration here but by using fiddler I have been able to emulate what I believe to be the same situation. I can take a look at it and see if I can fix it.

Can I go ahead?

@sonor3000
Copy link
Author

Because I created this issue, I'd be glad to help if I can. If you have no environment with an http proxy, I can test it in my companies network, thats where I have the described problem.

@josephsl
Copy link
Collaborator

josephsl commented Nov 24, 2018 via email

@marlon-sousa
Copy link
Contributor

Ok.

I have a working copy with the fix.

The problem is really with the urllib library. Interestingly it is discovering proxies correctly on Windows (which is the only OS we are interested in) but the urlopen method is failing even when the proxies resolved by urllib itself are issued in the proxies parameter.

For reference, the urlib.getproxies() method returned no proxy when I was behind a proxy and the correct list of proxies (one for http and one for https) when I had fiddler set up as a system proxy.

If this is a bug on urllib or not I don't know. There was no sense to investigate because python 2.7 and 3 already have a new library, called urllib2.

In its documentation, it is specified that the urlopen method of urllib2 auto discovers proxies and setup the connection accordingly. My tests confirm this, as importing the newer library made the process work seamlessly with and without a proxy.

Will this solve all possible problems we might ever have with proxies? Hardly. There might be proxies that urllib2 can not handle, the users operating system might or might not be configured to detect proxies automatically and so and so ... but this will possibly solve the vast majority of problems encountered inn the most common scenarios.

@sonor3000 I will try to generate a working copy of NVDA and make it available for you to test.

Be aware that this copy will pretend it is a release version of NVDA but it is not. It will pretend it is a release version because test versions do not offer the update option in the help menu and thus it would be impossible to test the feature.

Be also aware that I can not be held responsible for any problems caused by the run of this copy and be aware that this is a strictly based test version that should be deleted as soon as your tests finish.

If you can contact me via e-mail I can provide you and anyone else wanting to test the thing with a link.

If tests in real environments are successful I will submit the pr.

@sonor3000
Copy link
Author

sonor3000 commented Nov 28, 2018 via email

@marlon-sousa
Copy link
Contributor

Hello,

Here is what I carefully have found after some investigation.

I ask @josephsl and @LeonarddeR and who else you think is needed to read this carefully as it will be the bases for a proposal soon to be written.

To begin wit I must say that portability between operating systems are not being taken in account, since NVDA is and will for a foreseeable future be a project running only on Windows.

When using fiddler as a proxy, urllib.getproxies method consistently failed to detect I was behind a proxy. Even with me issuing the correct proxy address in the proxy parameter of urlopen the connection was not being established.
When using urllib2, the urlopen method was able to auto discover that I was behind a system proxy and could establish connections correctly.
I then took the copy of NVDA using urllib2 and tried it behind a real corporate proxy. This one has a .pac file set up via an administrative script. This time, urllib2.urlopen failed to detect the proxy and the connection was not established.
Python 3 urllib implementation is based on python2 urllib2 implementation, so I have good reasons to believe that its algorithm is not prepared to auto discover given kinds of proxy setup on Windows.
By researching, it was clear that python has no built-in, universal method to consistently discover and deal with proxies on Windows. This is the reason why the pypac project https://github.com/carsonyl/pypac was created.

So, it was very clear for me that if we really want to make NVDA HTTP connections more globally available, including for machines behind multiple kinds of proxies we would have to change the current implementation strategy.
The pipac project is great, but I started to think that including this dependency in NVDA was a little over kill. The mean reasons are:

1- The urllib and urllib2 libraries of python 2 and the urllib library of python 3 are low level pieces of code that are making HTTP connections available for python on multiple operating systems. However, because of this interoperability, it is clear that those libraries are not dealing with more specific maters, proxy being one of them.
2- Pypac would bring yet another dependency, making the project somewhat harder to maintain.
3- but, in the other hand, Windows is clearly able to resolve its own HTTP questions, including proxies by its on.
4- And Microsoft offers HTTP services for its client applications via com automation.
5- NVDA Will run only on Windows, so nothing offering portability is really an advantage in this case. In the other hand, using the high level windows services will separate the concerns of NVDA from the concerns of HTTP connectivity, making the code cleaner, providing correct connectivity management and making updates on connectivity and security maters bound to the current operating system.

With this in mind, there was no reason, at least in my perspective, to keep trying to deal with urllib or with external packages. I needed to check what services were being exposed and how those services were exposed.

By investigating more, I have found two candidates:
1- Microsoft xml (https://docs.microsoft.com/en-us/previous-versions/windows/desktop/ms759148(v%3Dvs.85).
2- WinHTTPRequest (https://docs.microsoft.com/en-us/windows/desktop/winhttp/winhttprequest).

I have chosen winHTTPRequest. Reasons are several, the main one being that msxml has several versions and some of them tend to be made available with more modern versions of office. The guids of the com object change for new versions. In the other hand, WinHTTPRequest 5.1 is supported since xp to 10 and nothing points towards it being unsupported. In fact, if you search for things such as proxy and Windows, WinHTTPRequest will often be listed as the solution adopted.
The comtypes library was queen on helping me to test a quick prototype without even needing to generate yet another nvda copy for testing. In fact, when I was behind that corporate proxy, issuing those lines validated the solution:

url = "https://www.google.com"
from comtypes import client
client.gen_dir = None # needed otherwise contypes fail when using the console
httpClient = client.CreateObject("WinHTTP.WinHTTPRequest.5.1")
httpClient.open("get", url)
httpClient.send()

After that, typing
httpClient.status

printed 200

and typing

httpClient.ResponseText

printed a nice HTML with google's page.

I then went home and coded the updater to use WinHTTPRequest instead of urllib. Another dist copy of NVDA was generated and taken to the machine behind the proxy and everything worked great.

I am also almost certain that the use of an operating system HTTP service will solve #4803 but I am not sure.

What next?

I am fully convinced that this is by far the best strategy we can use. It separates concerns, it makes NVDA code more clear, it adds no hurdles and it guarantees that Microsoft new requirements and complex situations on HTTP will be solved automatically, all with a cost of being tied to a specific operating system implementation that does not exist at all because NVDA itself is tied to Windows anyways.

Because of that, I will create a tiny wrapper to WinHTTPRequest that will provide NVDA code with a high level api to:
1- Make HTTP connections.
2- Get results synchronously or asynchronously.
3- provide specific call backs with partial data if the connection is asynchronous.

This is needed so that neither NVDA modules nor other addons get their code cluttered with http specific maters and also to provide a service with logging and other facilities added to make connectivity easier.

Finally, one last benefit is that urllib, urlib2 of python 2 and urllib of python 3 present syntactical differences, which make the code implementing the use of all these libraries at the same time to support python 2 and python 3 a really mess dealing with import as and handling import errors ...

Well, I can provide a branch with all this in place if there are no objections.

Independently of all this, I will send @sonor3000 a link with the WinHTTPRequest implementation to see if he / she can also at least get the update dialog to work.

Thanks,
Marlon

@josephsl
Copy link
Collaborator

josephsl commented Dec 1, 2018 via email

@mohdshara
Copy link

hi. since I too am behind a broxy at work, and I have reported this issue initially, I can certainly test your copy if that helps. my email address is: mohammed@atexplorer.com

@Adriani90
Copy link
Collaborator

cc: @derekriemer, @michaelDCurran

@sonor3000
Copy link
Author

sonor3000 commented Dec 17, 2018 via email

@marlon-sousa
Copy link
Contributor

marlon-sousa commented Dec 17, 2018 via email

@sonor3000
Copy link
Author

sonor3000 commented Dec 19, 2018 via email

@marlon-sousa
Copy link
Contributor

Hello @sonor3000 and @mohdshara ,

I have just sent you both an e-mail with link and further instructions.

I generated a dist copy of NVDA from a branch somewhat outdated, little after 2018.3, and implemented http connection via winHTTPRequests in the update checking process. As a consequence, now at least the update checking process, leading to a dialog asking you to update, should show whereas the same dialog, activated when going to NVDA menu then help then search for updates should present an error in the official copies of NVDA that still use urlib to manage HTTP connections.

Should the tests succeed on this patched version, I can code it into the main NVDA repo and open a pull request.

Please do replay my e-mail reporting if testing is successful and also update this issue here with your findings so everyone is tracked.

Please do not use this copy for anything else than testing.

Please do not try to update this copy (e.e do not reply yes in the dialog when asked to update) because the downloader is not yet with WinHTTPRequest, it is using urllib. The goal here is not to test the update process but is ratter to check the new WinHTTPRequest implementation which is, again, only implemented in the update checking process.

Please do delete this copy as soon as tests are performed. There is nothing worth in this version for day to day usage.

Thanks,
Marlon

@sonor3000
Copy link
Author

sonor3000 commented Dec 27, 2018 via email

@Adriani90
Copy link
Collaborator

@marlon-sousa is this work still in progress?

@LeonarddeR
Copy link
Collaborator

@marlon-sousa Could be I missed this, but... have you actually tested all scenarios with urllib on Python 3? I recall you only mention urllib and urllib2 on python 2.
As we have threshold_py3_staging snapshots, it should be possible to test NVDA update checking without too much hassle.

@Adriani90
Copy link
Collaborator

@marlon-sousa could you please give an update on this? See last comment posted by @LeonarddeR above.

@marlon-sousa
Copy link
Contributor

Hello,

Sorry for the response delay.

I have left the office where I had similar conditions in terms of proxy to test and have no longer how to do it.

Trials to simulate the exact proxy used and settings out of that office failed.

My intention was to use Microsoft http client via com plus. At that time, I could bypass my own simulated proxy but trial builds still failed on @sonor3000 scenarios.

Other than that, we are now on python 3, using urllib 3 and may be things changed.

It would be nice to know from @sonor3000 if this is the case after 2019.3.

@sonor3000
Copy link
Author

sonor3000 commented Nov 30, 2020 via email

@seanbudd
Copy link
Member

Closing as stale

@seanbudd seanbudd closed this as not planned Won't fix, can't repro, duplicate, stale Aug 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants