PWX-32011: added https proxy support

[ezhang-px]

What this PR does / why we need it:
Enable HTTPS proxy for telemetry (diags phonehome and callhome)

Which issue(s) this PR fixes (optional)
Closes #

Special notes for your reviewer:

[zoxpx]

Does this code have a support for NO_PROXY environment variable?

• note that we have a few customers using and relying on NO_PROXY functionality, which is not trivial to "calculate" as it supports subnets and DNS-host wildcards

To address proxy-support in porx -- we have introduced util.ProxyFromPxEnvironment() (@ref) -- so setting up the proxy-connection should be as simple as this:

   client := &http.Client{
      Transport: &http.Transport{
         Proxy: ProxyFromPxEnvironment,
      },
   }

• note, this is similar to "golang native" http.ProxyFromEnvironment -- except it'll react to PX_HTTP_PROXY, NO_PROXY, etc.. so, PX-specific proxy-vars)

So.. does Envoy/Telemetry react to the environment variables (i.e. env HTTP_PROXY=xxx NO_PROXY=xxx), or do we always have to "package" the http-proxy into special config-files?

• I think that ideally PX-Operator should react on HTTP_PROXY and PX_HTTP_PROXY env variables (generic and PX-specific versions of the proxy-vars), and
• forward the proxy-setting environment vars to the components

[ezhang-px]

Does this code have a support for NO_PROXY environment variable?

• note that we have a few customers using and relying on NO_PROXY functionality, which is not trivial to "calculate" as it supports subnets and DNS-host wildcards

To address proxy-support in porx -- we have introduced util.ProxyFromPxEnvironment() (@ref) -- so setting up the proxy-connection should be as simple as this:

   client := &http.Client{
      Transport: &http.Transport{
         Proxy: ProxyFromPxEnvironment,
      },
   }

• note, this is similar to "golang native" http.ProxyFromEnvironment -- except it'll react to PX_HTTP_PROXY, NO_PROXY, etc.. so, PX-specific proxy-vars)

So.. does Envoy/Telemetry react to the environment variables (i.e. env HTTP_PROXY=xxx NO_PROXY=xxx), or do we always have to "package" the http-proxy into special config-files?

• I think that ideally PX-Operator should react on HTTP_PROXY and PX_HTTP_PROXY env variables (generic and PX-specific versions of the proxy-vars), and
• forward the proxy-setting environment vars to the components

That's a good point @zoxpx , thanks for bring it up. Operator currently handles the case where no_proxy is required using this usePxProxy flag like (https://github.com/libopenstorage/operator/blob/master/drivers/storage/portworx/component/telemetry.go#L609) which basically tells which template of envoy config yaml to use.
If no proxy, use config like this (https://github.com/libopenstorage/operator/blob/master/deploy/ccm/envoy-config-register.yaml), otherwise use the one which requires either http or https proxy (https://github.com/libopenstorage/operator/blob/master/deploy/ccm/envoy-config-rest-custom-proxy.yaml)

[codecov]

Codecov Report

Patch coverage: 78.33% and no project coverage change.

Comparison is base (4ebe494) 75.60% compared to head (cffc25b) 75.61%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1120   +/-   ##
=======================================
  Coverage   75.60%   75.61%           
=======================================
  Files          64       64           
  Lines       17871    17902   +31     
=======================================
+ Hits        13512    13536   +24     
- Misses       3400     3406    +6     
- Partials      959      960    +1     

Impacted Files	Coverage Δ
drivers/storage/portworx/component/telemetry.go	69.33% <65.62%> (+0.03%)	⬆️
drivers/storage/portworx/util/util.go	71.42% <91.66%> (+0.35%)	⬆️
drivers/storage/portworx/component/grafana.go	77.68% <100.00%> (+0.09%)	⬆️
drivers/storage/portworx/portworx.go	84.98% <100.00%> (-0.03%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[github-actions]

This PR is stale because it has been in review for 3 days with no activity.

[jrivera-px]

Did we ever get the proxy that Lap setup to work with https?

[ezhang-px]

Did we ever get the proxy that Lap setup to work with https?

No, actually used the squid https proxy set up as instructed here

[zoxpx]

This is shocking -- you needed 1'500 lines of configuration files to set up proxy, where most Linux tools can do in one line (env http_proxy=xxx curl ...)

But, I suppose it is what it is.
I did log in a few questions/remarks, leaving it up to you if you want to clean it up before the commit.

[zoxpx]

i'm surprised you had to parse the URL like this --

• did https://pkg.go.dev/net/url#Parse not work for your specific use-case?

[ezhang-px]

This is mainly for retrieving the basic auth (user, password) from the url that customer puts in their STC's env variable and convert into header for envoy to pick up

[zoxpx]

Hmm.. that should have worked OK -- the https://pkg.go.dev/net/url#URL does have an optional Userinfo parameter, that should be parsing the user/passwd in URL correctly.

[ezhang-px]

Thanks for bring it up. Yes, I should have be using that. Fixed in the latest commit