Perform Proxy logic more like cURL

[Steve-Mcl]

closes #3904

Types of changes

Introduces a helper function getProxyForUrl that computes the proxy to use based on Env Vars and options provided in accordance with the discussion and linked articles in the issue

Proxy Variables Handling:

• No proxy variables: The function correctly returns an empty string for various protocols when no proxy variables are set.
• Invalid URLs: The helper function returns an empty string for invalid URLs.
• Proxy options:
  • allowUpperCase:false: Prevents *_PROXY values being returned when set to false.
  • favourUpperCase:false: Causes *_PROXY to be used before *_proxy when set to false. The expectation is that lowercase are preferred and get priority
  • includeNpm:false: Does not return npm_config_*_proxy env vars when set to false.
  • legacyMode:false: Processes URLs proxy in node-red <= v3.1 compatibility mode.

Proxy Environment Variables:

• Handles environment variables like http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY, ftp_proxy, ws_proxy, mqtt_proxy, all_proxy, no_proxy, etc.
• Properly handles different configurations of these variables, including lowercase has priority over uppercase, scheme handling, with/without port, default port numbers (e.g. http:80), IP addresses (v4/v6), host suffix, wildcard matching, and case sensitivity.
• Takes precedence of some variables over others based on the configurations.

NPM Proxy Configuration:

• Works with npm_config_http_proxy, npm_config_https_proxy, npm_config_proxy, npm_config_no_proxy.

• Handles precedence of NPM proxy configurations over environment variables like HTTP_PROXY, HTTPS_PROXY, NO_PROXY.

• Overall, the "Proxy Helper" function effectively handles various proxy configurations and environment variables, providing consistent and expected results according to the specified configurations.

• Bugfix (non-breaking change which fixes an issue)

• New feature (non-breaking change which adds functionality)

Proposed changes

• HTTP, ws and MQTT nodes will all determine the proxy by calling to this new helper function
• HTTP node will compute the proxy for dynamic URLs (currently only computed against the nodes edit panel URL field)
• support overriding with system env var NR_PROXY_MODE (setting to legacy reverts functionality)

TODO

• Tests & Docs for NR_PROXY_MODE

Checklist

• I have read the contribution guidelines
• For non-bugfix PRs, I have discussed this change on the forum/slack team.
• I have run npm run test to verify the unit tests pass
• I have added suitable unit tests to cover the new/changed functionality

[knolleary]

When are the npm config options used? I can't see any reference to them in the original issue other than when you bring them into the discussion and google searches don't really reveal anything about how they expected to be used beyond npm's own use.

[Steve-Mcl]

When are the npm config options used? I can't see any reference to them in the original issue other than when you bring them into the discussion and google searches don't really reveal anything about how they expected to be used beyond npm's own use.

If node-red is launched via NPM and the npmrc (or CLI) has specified proxy options, I believe the correct thing to do is respect them. In the interest of getting this in, I could disable the npm proxy parts (there is an option excludeNpm we could default to true

There are a number issues and inconsistencies this covers including:

• In the http-request node, the proxy URL is only determined when the node is instantiated - meaning when using a dynamic URL, the proxy is NOT re-evaluated per request
• no_proxy should not be case-sensitive
• no_proxy with protocols that have a default port are not considered
• Does not consider NPM proxy configuration at all (we could ignore this for now as discussed per above)
• Doesn't consider https_proxy or HTTPS_PROXY
• Incorrectly prioritises HTTP_PROXY over http_proxy. HTTP_PROXY is not always supported or recommended
• doesn't consider all_proxy or ALL_PROXY

So it would be great if we can get it in.

[Steve-Mcl]

Here are the places to change to exclude npm proxy options

[knolleary]

One concern I have is how many settings this is added to the settings file - for a topic that has very rarely caused any questions or issues.

Why can we not just have a standard behaviour in terms of upper-case and lower-case? Document which we will use in preference to the other and not expect users to have to understand all of these flags that have been added to the settings file.

[Steve-Mcl]

We could be opinionated and expose the finer control at a later if that is an issue.

I mean, we dont really have to add anything at all to the settings file (since it has defaults) but we may wish to keep the mode: "legacy" flag listed out as a formative option.

e.g. settings.js

    // proxyOptions: {
    //     mode: "legacy", // legacy mode is for non-strict previous proxy determination logic (node-red < v4 compatible)
    // },

That way, the user can be least change back to legacy (if needed) and the unlisted options are considered "unsupported"/"subject to change"

[knolleary]

Okay - lets go with what you've got - just remove the entries from the settings file to reduce noise.