(Linux) Add Proxy configuration support

[SwonVIP]

Expected Behaviour

When using the aio-cli behind a corporate proxy the proxy should be somehow configurable or it should be able to use the environment variables

Actual Behaviour

aio-cli cannot connect to the internet for example to load the project list from AEMaaCS account.

Reproduce Scenario (including but not limited to)

Be behind a corporate proxy like proxy.example:8080 which supports NTLM authentication.

Steps to Reproduce

Launch aio and try to access your AEMaaCS information during the init steps.

Platform and Version

Linux Ubuntu (18.04)
aio-cli:Latest
node/npm: LTS

Sample Code that illustrates the problem

Cant priovide this only had temporary access to a AEMaaCS test account during AdaptTo() 2020

Logs taken while reproducing problem

Connection failed error message

[meryllblanchet]

Hi @SwonVIP and thanks for reaching out!

Could we ask you a bit more of details?

• The output of the aio info command on your machine
• Which was the use-case you were working on?

This is the first time we have a user facing proxy issues and would love to hear more.

Note: this should not be the reason for your issue, but fyi aio-cli currently supports nodeJS 10 and 12 LTS, as per our documentation.

cc @purplecabbage @sarahxxu

[sarahxxu]

@SwonVIP

our teammate @shazron found this -- https://www.serverlab.ca/tutorials/linux/administration-linux/how-to-configure-proxy-on-ubuntu-18-04/ would you like to give it a try?

[SwonVIP]

Hi @meryllblanchet

1. Output of aio info:

aio info
System:
OS: Linux 5.4 Ubuntu 18.04.5 LTS (Bionic Beaver)
CPU: (4) x64 Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
Memory: 23.24 GB / 31.12 GB
Shell: 4.4.20 - /bin/bash
Binaries:
Node: 12.14.1 - ~/.nvm/versions/node/v12.14.1/bin/node
Yarn: Not Found
npm: 6.13.4 - ~/.nvm/versions/node/v12.14.1/bin/npm
Virtualization:
Docker: 19.03.13 - /usr/bin/docker
npmGlobalPackages:
@adobe/aio-cli: 3.7.0
CLI plugins:
@adobe/aio-cli 3.7.0 (core)
@adobe/aio-cli-plugin-app 3.5.2 (core)
@adobe/aio-cli-plugin-auth 2.4.0 (core)
@adobe/aio-cli-plugin-certificate 0.2.0 (core)
@adobe/aio-cli-plugin-config 2.3.0 (core)
@adobe/aio-cli-plugin-console 3.2.0 (core)
@adobe/aio-cli-plugin-events 1.1.0 (core)
@adobe/aio-cli-plugin-info 1.1.1 (core)
@adobe/aio-cli-plugin-runtime 1.10.2 (core)
@oclif/plugin-autocomplete 0.2.0 (core)
@oclif/plugin-help 2.2.3 (core)
@oclif/plugin-not-found 1.2.4 (core)
@oclif/plugin-plugins 1.9.0 (core)
@oclif/plugin-warn-if-update-available 1.7.0 (core)

2. So my use-case was to create or init an app/project for an adaptTo conference workshop I was participating:

aio app init adaptTo
Retrieving information from Adobe I/O Console..
[CoreConsoleAPISDK:ERROR_GET_ORGANIZATIONS] FetchError: request to https://developers.adobe.io/console/organizations failed, reason: connect ETIMEDOUT 20.41.43.227:443
at ClientRequest. (/home/XXX/XXXXX/.nvm/versions/node/v12.14.1/lib/node_modules/@adobe/aio-cli/node_modules/swagger-client/node_modules/node-fetch/index.js:133:11)
at ClientRequest.emit (events.js:223:5)
at TLSSocket.socketErrorListener (_http_client.js:406:9)
at TLSSocket.emit (events.js:223:5)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:81:21) {
name: 'FetchError',
message: 'request to https://developers.adobe.io/console/organizations failed, reason: connect ETIMEDOUT 20.41.43.227:443',
type: 'system',
errno: 'ETIMEDOUT',
code: 'ETIMEDOUT'
}

You are about to initialize the project 'adaptTo'

@sarahxxu
Unfortunately the global proxy environment variables are already set. However it is a proxy with NTLM authentication - maybe that could be the problem in this case.

I dont know if it is possible but it would be really nice if the aio would have an available flag to set or configure the proxy somehow. Maybe to even be more system independant - could be useful in certain situations.

[shazron]

For NTLM, we need to wrap all our service calls with an NTLM handshake. This will have to be a new feature @sarahxxu
This is some sample code: https://gist.github.com/JoeStanton/64697ebcd865d6d1145d020475d7a0f4

[sarahxxu]

@shazron Thanks for the investigation. Let's put this as an improvement ticket for later! :)

[meryllblanchet]

@sarahxxu , I flagged this one as enhancement request accordingly.

[aiojbot]

JIRA issue created: https://jira.corp.adobe.com/browse/ACNA-950

[meryllblanchet]

Hi @justinedelson , it seems that adobe/aio-cli-plugin-cloudmanager#177 relates to this one.
What about joining forces to handle the proxy support at CLI level?

cc @sarahxxu

[justinedelson]

@meryllblanchet I absolutely think that the mechanism to set proxy configuration should be consistent across plugins. I honestly don't know enough to say whether the mechanism to use that configuration can be shared easily. For example, aio-lib-ims (which the Cloud Manager plugin still doesn't use due to adobe/aio-lib-ims#52, but that's a different topic) uses request-promise-native but aio-lib-cloudmanager uses cross-fetch.

So... I hate to say this, but I think what we need is aio-lib-fetch which would then be adopted by all of the libs (probably starting with aio-lib-ims).

[meryllblanchet]

@shazron any reason why we opted for request-promise-native in the context of the aio-lib-ims? AFAIK we usually leverage cross-fetch.

@justinedelson adobe/aio-lib-ims#52 hasn't been prioritized yet on our side.
https://github.com/adobe/aio-lib-core-networking is the starting point you're thinking about.

cc @sarahxxu

[shazron]

@meryllblanchet I believe we can move off request-promise since it has been deprecated. Since this was implemented by another developer, there was no cohesiveness of library usage, which we can do now. We are using node-fetch in most of the libraries, and sometimes using cross-fetch (which uses node-fetch behind the scenes).

[shazron]

Related: adobe/aio-lib-core-networking#13

[arobsco]

Is proxy support still something that is being worked on?

[pahupe]

@arobsco : this has been implemented for aio. Not sure why this ticket is still open

[arobsco]

Thanks wasn't sure as having issues with the aio-cli app not connecting and only reason I can come up with is it's not honouring proxy settings but I'll do some more digging

[pahupe]

Please reach out to @shazron for implementation details, if needed

[arobsco]

@shazron it looks like this issue isn't fully resolved in that although https://github.com/adobe/aio-lib-core-networking is proxy aware and supports proxies, not all the plugins for AIO CLI use the networking package. For example https://github.com/adobe/aio-lib-console/blob/7ec79a21e83d8d4db6642b8ad6bb84a386a1fb14/src/index.js#L185-L190 uses Swagger API which is not configured to use networking package and therefore have proxy awareness

Changing the code to require the networking package and use custom fetch solves the issue at least for the aio-lib-console package