global-agent breaks WebdriverIO keep-alive support. Options?

[th0car]

After a couple of weeks of troubleshooting we have come to these conclusions:

1. To run automated tests in Browserstack Automate using keep-alive is required to get good performance. Without sessions being kept-alive it is more or less unusable.
2. WebdriverIO supports keep-alive.
3. Since we need to use a corporate proxy to run WebdriverIO tests we need to use global-agent.
4. Our corporate proxy supports keep-alive (we know this because of some other test frameworks being successfully used with keep-alive and Browserstack using the same proxy)
5. global-agent does not support keep-alive...

There is an open issue to add keep-alive for global-agent but it has not changed status since 2019 and I want to discuss the options here first.

What do you think? Are my conclusions correct or did I miss something relevant?
Are there options to global-agent that could actually be used with the current version of WDIO with none or small modifciations to WDIO?
Is helping gajus/global-agent implementing keep-alive support the only way forward?

[christian-bromann]

What do you think? Are my conclusions correct or did I miss something relevant?

This seems like a correct conclusion.

Are there options to global-agent that could actually be used with the current version of WDIO with none or small modifciations to WDIO?

WDIO could implement a different proxy agent that might support keep alive.

Is helping gajus/global-agent implementing keep-alive support the only way forward?

I would say so, yes!

[th0car]

Thank you! I am pretty sure that neither me or anyone in my team has the required knowledge on the protocol level needed to implement keep-alive in global-agent, but i will definitely try to find out.

Were there any other specific proxy agents that were considered for WDIO that I should have a look at?

Would you say that the WDIO proxy implementation has to be adapted for a specific proxy agent / package like now or could it be possible to make this configurable in, lets say, the same way as we can freely chose from different reporting services?

[christian-bromann]

Were there any other specific proxy agents that were considered for WDIO that I should have a look at?

WebdriverIO is using got for making requests. Any proxy agent that is compatible with got works with WebdriverIO.

Would you say that the WDIO proxy implementation has to be adapted for a specific proxy agent / package like now or could it be possible to make this configurable in, lets say, the same way as we can freely chose from different reporting services?

I don't see a need so far to implement a proxy agent natively into WebdriverIO. It should be handled by got. Feel free to make suggestions how this can be improved so it would fit your needs.

[th0car]

Let me see if I understand this correctly: If I install and configure another proxy agent that is got compatible this proxy agent will be used without anyone having to modify any code in WebdriverIO?

[christian-bromann]

If I install and configure another proxy agent that is got compatible this proxy agent will be used without anyone having to modify any code in WebdriverIO?

That depends how the proxy agent is integrated with got, but I am happy to make necessary changes to make this possible.

[th0car]

Good news! I tried installing hpagent and used it to configure the agent: section I added to the conf. Now WDIO successfully uses it as a proxy agent and requests to browserstack does not generate any keep-alive errors.
So - at least for this agent no modifications whatsoever to the WebdriverIO code was required. If I can find any other agents that supports keep-alive and got I will try out those as well. Thanks again Christian!

[christian-bromann]

@th0car awesome! Mind adding some docs how to use hpagent with WebdriverIO somewhere here: https://webdriver.io/docs/proxy ?

[th0car]

Sure I can do that. Just give me some time to first double check the values in the hpagent config so that the example is suitable for WebdriverIO.

[th0car]

So - using hpagent instead of global-agent as proxy agent will make your WebdriverIO + Browserstack Sessions use keep-alive. This is good!

The bad news is a side-effect that I didn't notice at first: Using hpagent instead of global-agent makes the @wdio/browserstack-service unable to get request through to api.browserstack.com and collector-observability.browserstack.com which heavily reduces the information sent to the browserstack dashboard. The error is
@wdio/utils:shim: RequestError: getaddrinfo ENOTFOUND api.browserstack.com which is the same error I used to get for hub.browserstack.com before using a proxy-agent.

As far as I can workout this is likely to be a problem in @wdio/browserstack-service but the only thing that I am sure of is that these requests works when global-agent is used as the proxy-agent.

Should I file this as a bug even if hpagent isn't "officially" supported by WDIO?
Any other thoughts on this?

[christian-bromann]

I think in order to make this work in the BrowserStack service, we only need to ensure that the proxy is passed in whenever a request is made. The service uses got as request lib, similar to what the webdriver package does. Feel free to file a feature request but a pull request would be much appreciated as well.

Should I file this as a bug even if hpagent isn't "officially" supported by WDIO?

Given the amount of different proxy packages out there we can't make any official statements on what is supported and what is not.