Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

serenity-bdd update ignores proxy settings #356

Closed
JoeChester opened this issue Oct 11, 2019 · 15 comments
Closed

serenity-bdd update ignores proxy settings #356

JoeChester opened this issue Oct 11, 2019 · 15 comments

Comments

@JoeChester
Copy link

Hello everyone, I am having troubles with using the serenity-bdd update command behind a corporate http proxy.

As far as I understand, @serenity-js/serenity-bdd package has deprecated the serenity-cli package, right?.
I have configured my corporate proxy settings as environment variable for node. With the deprecated serenity update command, the proxy settings are accepted and the serenity .jar file is downloaded and put into the node_modules/serenity-js/.cache folder. However, with the newer serenity-bdd update command, I either get the error message

I'm terribly sorry, but something didn't go according to plan.
The API call has failed

when I use the HTTP_PROXY environment variable, or

I'm terribly sorry, but something didn't go according to plan.
Looks like there was an issue with Axios configuration

when I use the HTTPS_PROXY environment variable.
I'd really like to use the newer @serenity-js/serenity-bdd package, as it lets me create reports with Java 11+ as well.

The old cli fails on the Java warning

Error: WARNING: An illegal reflective access operation has occurred
at C:\DATA\flores\flores-dlm\dlm-e2e\node_modules\serenity-cli\lib\actions\logging.js:9:35
at process._tickCallback (internal/process/next_tick.js:68:7)

while the newer cli still produces that warning, but ignores it and produces the report regardless.

My current workaround is to download the .jar file with the old cli-downloader and reference it with the new cli via the --cacheDir command line option. But this means I can not remove the deprecated serenity-cli dependency in favor of the newer one right now.

Is there a chance that the newer update cli can respect the system http proxy settings?

#135 might be related to that, but I think it used the old cli.

@abhinaba-ghosh
Copy link

Hi,
You don't need the older CLI to download the necessary serenity jars.

You simply put AXIOS dependency in your package.json ( npm i --save axios) and run serenity-bdd update. It should work.

Lets try and let us know.

@jan-molak
Copy link
Member

jan-molak commented Oct 13, 2019

Hi @JoeChester,

Could you please have a look at the Serenity/JS template project and see if you can use it to download the Serenity BDD jar behind the proxy? The project is configured to do it postinstall, so as soon as you've run npm install.

As per @abhinaba1080's comment above, Serenity/JS version 2 uses the axios http client, which should be installed as one of your (dev) dependencies (npm install axios --save-dev --save-exact). However, even though axios detects the proxy setting under most circumstances, it might struggle to connect to a HTTPs url over a HTTP proxy (as opposed to a HTTPs proxy). Could you please provide some more details on the type of proxy your company uses? (HTTP vs HTTPs).


The Error: WARNING: An illegal reflective access operation has occurred looks like a Java compatibility issue in Serenity BDD CLI. I've reported this at serenity-bdd/serenity-cli#6

@okey-right
Copy link

hey, @jan-molak ,

Even I am facing issues with the newer serenity-bdd update command behind a corporate proxy and got the below message:

I'm terribly sorry, but something didn't go according to plan.
Looks like there was an issue with Axios configuration

With the deprecated serenity update there aren't any issues and the jar file is downloaded.

@jan-molak
Copy link
Member

Hey @JoeChester, @okey-right - would you be able to share the (anonymised) proxy configuration you're using? I'd be curious to understand if the issues you're seeing are caused by using a http proxy with a https repo url.

@okey-right
Copy link

hey @jan-molak , - It's a (PAC) Proxy auto-configuration. Goes like

http://pac.xxxxxxxxx.com.xx/wpad.dat
sorry, but I am not sure if this is what you were expecting.

@okey-right
Copy link

@jan-molak , Were you able to check this out ? Thanks.

@habmukandu
Copy link

@jan-molak same problem here. We have a HTTP proxy without authentification for HTTP/HTTPS requests. Same error on linux and windows clients. Let me know if I can provide more details for solving the issue.

@jan-molak
Copy link
Member

jan-molak commented Nov 12, 2019

@habmukandu - do you use a pac file as well?

It doesn't seem like axios, the HTTP client that @serenity-js/serenity-bdd module uses, supported pac files out of the box. However, we could get around that with something like pac-proxy-agent perhaps.

@jan-molak jan-molak added this to the 2.0 milestone Nov 12, 2019
@habmukandu
Copy link

@habmukandu - do you use a pac file as well?

no, I defined the proxy in environment with 'https_proxy' (http://proxy.corproot.net:8079)

However, it seems to be a general problem with Axios and Squid proxies. I left a comment on Axios issues to reproduce it: link

When using a modified version of Axios using 'https-proxy-agent' it works with our proxy.

@jan-molak
Copy link
Member

jan-molak commented Nov 12, 2019

@habmukandu - I see, so the solution seems to be to intelligently choose the appropriate proxy agent based on the config the developer has provided.

Thinking out loud

Could we have something along those lines:

  • serenity-bdd update - use any default HTTP_PROXY or HTTPS_PROXY settings and rely on axios to sort it out itself (seems to work in most environments)
  • serenity-bdd update --proxy=https://proxy.corproot.net:8079 - a HTTPS proxy specified to fetch content from a HTTPS server; don't use any external agents, pass the config as is to axios
  • serenity-bdd update --proxy=http://proxy.corproot.net:8079 - a HTTP proxy specified to fetch content from a HTTPS server where Serenity BDD is hosted; use https-proxy-agent
  • serenity-bdd update --proxy=http://pac.example.com:1234/wpad.dat - the url points to a file wpad.dat, therefore it has to be a PAC file; use pac-proxy-agent

The --proxy switch would override any proxy-related env variable settings.

Does this logic sound about right? Or am I missing something?

@habmukandu
Copy link

@jan-molak - sure this would solve the issue. The drawback is the need of specifying a proxy config before running npm install in some cases.

I would recommend to use the same proxy behaviour as npm uses:

  1. Get proxy config from env or .npmrc file
  2. If proxy is set, use axios with https-proxy-agent
const axiosConfig = {
    proxy: false
};
const HttpsProxyAgent = require("https-proxy-agent");
const proxyConfig = process.env.HTTPS_PROXY || process.env.https_proxy || process.env.npm_config_https_proxy || process.env.HTTP_PROXY || process.env.http_proxy || process.env.npm_config_proxy || false;
const httpsAgent = new HttpsProxyAgent(proxyConfig);
Object.assign(axiosConfig, proxyConfig ? {httpsAgent: httpsAgent} : null);
const axios = require("axios").create(axiosConfig);

This would work for users having proxy configured in env or npm. I think PAC support is not needed as npm is also not supporting it.

@jan-molak
Copy link
Member

jan-molak commented Nov 13, 2019

That's a good point about NPM not supporting the pac files, @habmukandu.

@okey-right - how do you install the node modules in your setup? How does npm learn about the pac file config?

@okey-right
Copy link

@jan-molak - The pac file comes pre-configured on the windows machine for everyone on a system level. Everything network action on the system goes through the configuration on the pac file by default.

I am able to download all other dependencies including other serenity js dependencies. But only face problems during fetching the cli when running the serenity-bdd update command.

@habmukandu
Copy link

@okey-right when a PAC file is configured for a windows machine applications are not automatically forced to use the PAC settings. Applications like Chrome or Edge will always use the system proxy configuration while others like NPM completely ignore it.

When npm install works for you behind a proxy then I would imagine that you have configured a proxy in environment (https_proxy) or in your .npmrc file in your home directory.

@jan-molak jan-molak added this to To do in Serenity/JS Board Mar 10, 2020
@jan-molak jan-molak moved this from To do to Awaiting feedback in Serenity/JS Board Mar 14, 2020
@jan-molak jan-molak moved this from Awaiting feedback to In progress in Serenity/JS Board Jun 4, 2020
@jan-molak jan-molak moved this from In progress to To do in Serenity/JS Board Jul 8, 2020
Serenity/JS Board automation moved this from To do to Done Oct 27, 2020
@jan-molak
Copy link
Member

As of 2.17.4, serenity-bdd update supports both npm config as well as env variables. Please take it for a spin and let me know what you think @JoeChester @okey-right @habmukandu !

It turned out to be a bit more involved than I had hoped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

No branches or pull requests

5 participants