[🚀 Feature]: Improve success experience with network request transformer

[RenderMichael]

Feature and motivation

The Selenium C# bindings have network interception capabilities which let users add various network interception knobs. For example, one may define a Func<HttpRequestData, HttpRequestData> RequestTransformer to apply some modification to outgoing requests in the browser.

However, it only works if you return the same HttpRequestData which was passed in (or somehow return a Selenium-provided HttpRequestData instead of creating your own).

If we modify the Selenium-provided example to return a new instance instead of the same one:

public async Task TransformNetworkRequest()
{
    var handler = new NetworkRequestHandler
    {
        RequestMatcher = request => request.Url.Contains("one.js"),
        RequestTransformer = request =>
        {
            return new HttpRequestData
            {
                Headers = request.Headers,
                Method = request.Method,
                PostData = request.PostData,
                Url = request.Url.Replace("one", "two"),
            };
        }
    };
    INetwork networkInterceptor = driver.Manage().Network;
    networkInterceptor.AddRequestHandler(handler);
    await networkInterceptor.StartMonitoring();

    driver.Url = "https://www.selenium.dev/selenium/web/devToolsRequestInterceptionTest.html";
    driver.FindElement(By.TagName("button")).Click();
    await networkInterceptor.StopMonitoring();

    var resultElement = driver.FindElement(By.Id("result"));
    Assert.That(resultElement.Text, Is.EqualTo("two"));
}

We get an CDP exceptions stating that the ContinueRequestCommandSettings.RequestId value is missing:

Unhandled error occurred in event handler of 'Fetch.requestPaused' method. OpenQA.Selenium.DevTools.CommandResponseException: Fetch.continueRequest: Invalid parameters - Failed to deserialize params.requestId - BINDINGS: string value expected at position 18
   at OpenQA.Selenium.DevTools.DevToolsSession.SendCommand(String commandName, String sessionId, JsonNode commandParameters, CancellationToken cancellationToken, Nullable`1 millisecondsTimeout, Boolean throwExceptionIfResponseNotReceived) in D:\Code\GitHub\RenderMichael\selenium\dotnet\src\webdriver\DevTools\DevToolsSession.cs:line 315
   at OpenQA.Selenium.DevTools.DevToolsSession.SendCommand(String commandName, JsonNode commandParameters, CancellationToken cancellationToken, Nullable`1 millisecondsTimeout, Boolean throwExceptionIfResponseNotReceived) in D:\Code\GitHub\RenderMichael\selenium\dotnet\src\webdriver\DevTools\DevToolsSession.cs:line 261
   at OpenQA.Selenium.DevTools.DevToolsSession.SendCommand[TCommand,TCommandResponse](TCommand command, CancellationToken cancellationToken, Nullable`1 millisecondsTimeout, Boolean throwExceptionIfResponseNotReceived) in D:\Code\GitHub\RenderMichael\selenium\dotnet\src\webdriver\DevTools\DevToolsSession.cs:line 233
   at OpenQA.Selenium.DevTools.V133.V133Network.ContinueRequest(HttpRequestData requestData) in D:\Code\GitHub\RenderMichael\selenium\dotnet\src\webdriver\DevTools\v133\V133Network.cs:line 170
   at OpenQA.Selenium.NetworkManager.OnRequestPaused(Object sender, RequestPausedEventArgs e) in D:\Code\GitHub\RenderMichael\selenium\dotnet\src\webdriver\NetworkManager.cs:line 235
   at OpenQA.Selenium.DevTools.Network.<>c__DisplayClass25_1.<<OnRequestPaused>b__0>d.MoveNext() in D:\Code\GitHub\RenderMichael\selenium\dotnet\src\webdriver\DevTools\Network.cs:line 186

This exception is logged and not surfaced to the user; instead, the user sees a timeout exception:

OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:50073/session/782cfe4ef4d1547d6a85b5d9511e5195/url timed out after 60 seconds.
  ----> System.Threading.Tasks.TaskCanceledException : The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
  ----> System.TimeoutException : The operation was canceled.
  ----> System.Threading.Tasks.TaskCanceledException : The operation was canceled.
  ----> System.IO.IOException : Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
  ----> System.Net.Sockets.SocketException : The I/O operation has been aborted because of either a thread exit or an application request.

This problem is insurmountable from the user's side because the HttpRequestData.RequestId has an internal setter. However, even if it wasn't, the experience is unclear without specific CDP knowledge.

Solutions

Option 1

Expose HttpRequestData.RequestId setter publicly. This is the most "let users fix it" option. It does not address the exception we get, but it does give users a mechanism to work around it. I'm not sure if there are any unwanted consequences of this.

Option 2

Change the RequestTransformer from Func<HttpRequestData, HttpRequestData> to Action<HttpRequestData>.

This will modify the existing instance instead of asking users to return another one. (side benefit that return null cannot happen).

Today:

var handler = new NetworkRequestHandler
{
    RequestMatcher = request => request.Url.Contains("one.js"),
    RequestTransformer = request =>
    {
        request.Url = request.Url.Replace("one", "two");
 
        return request;
    }
};

With this change:

var handler = new NetworkRequestHandler
{
    RequestMatcher = request => request.Url.Contains("one.js"),
    RequestTransformer = request =>
    {
        request.Url = request.Url.Replace("one", "two");
    }
};

The potential drawbacks:

• Yet another obsoletion process (though this is an advanced feature)
• Users cannot return another instance (if that was ever considered useful)

Option 3

Set the value on the returned instance internally.

This addresses the exception, but it has the same drawback that users cannot return another instance of the request data (if that was ever considered useful)

Usage example

See examples above

[github-actions]

@RenderMichael, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!