serenity-bdd update ignores proxy settings

[JoeChester]

Hello everyone, I am having troubles with using the serenity-bdd update command behind a corporate http proxy.

As far as I understand, @serenity-js/serenity-bdd package has deprecated the serenity-cli package, right?.
I have configured my corporate proxy settings as environment variable for node. With the deprecated serenity update command, the proxy settings are accepted and the serenity .jar file is downloaded and put into the node_modules/serenity-js/.cache folder. However, with the newer serenity-bdd update command, I either get the error message

I'm terribly sorry, but something didn't go according to plan.
The API call has failed

when I use the HTTP_PROXY environment variable, or

I'm terribly sorry, but something didn't go according to plan.
Looks like there was an issue with Axios configuration

when I use the HTTPS_PROXY environment variable.
I'd really like to use the newer @serenity-js/serenity-bdd package, as it lets me create reports with Java 11+ as well.

The old cli fails on the Java warning

Error: WARNING: An illegal reflective access operation has occurred
at C:\DATA\flores\flores-dlm\dlm-e2e\node_modules\serenity-cli\lib\actions\logging.js:9:35
at process._tickCallback (internal/process/next_tick.js:68:7)

while the newer cli still produces that warning, but ignores it and produces the report regardless.

My current workaround is to download the .jar file with the old cli-downloader and reference it with the new cli via the --cacheDir command line option. But this means I can not remove the deprecated serenity-cli dependency in favor of the newer one right now.

Is there a chance that the newer update cli can respect the system http proxy settings?

#135 might be related to that, but I think it used the old cli.

[abhinaba-ghosh]

Hi,
You don't need the older CLI to download the necessary serenity jars.

You simply put AXIOS dependency in your package.json ( npm i --save axios) and run serenity-bdd update. It should work.

Lets try and let us know.

[jan-molak]

Hi @JoeChester,

Could you please have a look at the Serenity/JS template project and see if you can use it to download the Serenity BDD jar behind the proxy? The project is configured to do it postinstall, so as soon as you've run npm install.

As per @abhinaba1080's comment above, Serenity/JS version 2 uses the axios http client, which should be installed as one of your (dev) dependencies (npm install axios --save-dev --save-exact). However, even though axios detects the proxy setting under most circumstances, it might struggle to connect to a HTTPs url over a HTTP proxy (as opposed to a HTTPs proxy). Could you please provide some more details on the type of proxy your company uses? (HTTP vs HTTPs).

---

The Error: WARNING: An illegal reflective access operation has occurred looks like a Java compatibility issue in Serenity BDD CLI. I've reported this at serenity-bdd/serenity-cli#6

[okey-right]

hey, @jan-molak ,

Even I am facing issues with the newer serenity-bdd update command behind a corporate proxy and got the below message:

I'm terribly sorry, but something didn't go according to plan.
Looks like there was an issue with Axios configuration

With the deprecated serenity update there aren't any issues and the jar file is downloaded.

[jan-molak]

Hey @JoeChester, @okey-right - would you be able to share the (anonymised) proxy configuration you're using? I'd be curious to understand if the issues you're seeing are caused by using a http proxy with a https repo url.

[okey-right]

hey @jan-molak , - It's a (PAC) Proxy auto-configuration. Goes like

http://pac.xxxxxxxxx.com.xx/wpad.dat
sorry, but I am not sure if this is what you were expecting.

[okey-right]

@jan-molak , Were you able to check this out ? Thanks.

[habmukandu]

@jan-molak same problem here. We have a HTTP proxy without authentification for HTTP/HTTPS requests. Same error on linux and windows clients. Let me know if I can provide more details for solving the issue.

[jan-molak]

@habmukandu - do you use a pac file as well?

It doesn't seem like axios, the HTTP client that @serenity-js/serenity-bdd module uses, supported pac files out of the box. However, we could get around that with something like pac-proxy-agent perhaps.

[habmukandu]

@habmukandu - do you use a pac file as well?

no, I defined the proxy in environment with 'https_proxy' (http://proxy.corproot.net:8079)

However, it seems to be a general problem with Axios and Squid proxies. I left a comment on Axios issues to reproduce it: link

When using a modified version of Axios using 'https-proxy-agent' it works with our proxy.

[jan-molak]

@habmukandu - I see, so the solution seems to be to intelligently choose the appropriate proxy agent based on the config the developer has provided.

Thinking out loud

Could we have something along those lines:

• serenity-bdd update - use any default HTTP_PROXY or HTTPS_PROXY settings and rely on axios to sort it out itself (seems to work in most environments)
• serenity-bdd update --proxy=https://proxy.corproot.net:8079 - a HTTPS proxy specified to fetch content from a HTTPS server; don't use any external agents, pass the config as is to axios
• serenity-bdd update --proxy=http://proxy.corproot.net:8079 - a HTTP proxy specified to fetch content from a HTTPS server where Serenity BDD is hosted; use https-proxy-agent
• serenity-bdd update --proxy=http://pac.example.com:1234/wpad.dat - the url points to a file wpad.dat, therefore it has to be a PAC file; use pac-proxy-agent

The --proxy switch would override any proxy-related env variable settings.

Does this logic sound about right? Or am I missing something?

[habmukandu]

@jan-molak - sure this would solve the issue. The drawback is the need of specifying a proxy config before running npm install in some cases.

I would recommend to use the same proxy behaviour as npm uses:

1. Get proxy config from env or .npmrc file
2. If proxy is set, use axios with https-proxy-agent

const axiosConfig = {
    proxy: false
};
const HttpsProxyAgent = require("https-proxy-agent");
const proxyConfig = process.env.HTTPS_PROXY || process.env.https_proxy || process.env.npm_config_https_proxy || process.env.HTTP_PROXY || process.env.http_proxy || process.env.npm_config_proxy || false;
const httpsAgent = new HttpsProxyAgent(proxyConfig);
Object.assign(axiosConfig, proxyConfig ? {httpsAgent: httpsAgent} : null);
const axios = require("axios").create(axiosConfig);

This would work for users having proxy configured in env or npm. I think PAC support is not needed as npm is also not supporting it.

[jan-molak]

That's a good point about NPM not supporting the pac files, @habmukandu.

@okey-right - how do you install the node modules in your setup? How does npm learn about the pac file config?

[okey-right]

@jan-molak - The pac file comes pre-configured on the windows machine for everyone on a system level. Everything network action on the system goes through the configuration on the pac file by default.

I am able to download all other dependencies including other serenity js dependencies. But only face problems during fetching the cli when running the serenity-bdd update command.

[habmukandu]

@okey-right when a PAC file is configured for a windows machine applications are not automatically forced to use the PAC settings. Applications like Chrome or Edge will always use the system proxy configuration while others like NPM completely ignore it.

When npm install works for you behind a proxy then I would imagine that you have configured a proxy in environment (https_proxy) or in your .npmrc file in your home directory.

[jan-molak]

As of 2.17.4, serenity-bdd update supports both npm config as well as env variables. Please take it for a spin and let me know what you think @JoeChester @okey-right @habmukandu !

It turned out to be a bit more involved than I had hoped.