Proposal: Add a built-in HTTP request/response interception API to the Page type

[mobilezht]

Proposal: Add a built-in HTTP request/response interception API to the Page type

Target repository: obscura
Topic: Feature request / API design discussion

---

Motivation

We are building a web crawler and security analysis tool on top of obscura, and a critical requirement is the ability to intercept, inspect, and optionally modify all HTTP requests and responses initiated by page-level JavaScript — including XMLHttpRequest and the fetch() API.

This capability is essential for:

• Web crawling / scraping: Capturing API payloads (often returned as XHR/Fetch responses) that the SPA renders asynchronously, without reverse-engineering every front-end bundle.
• Automated security scanning: Intercepting requests to inject test payloads, block malicious resources, or audit CORS/SSRF behavior.
• Observability & debugging: Logging all network activity from a page, including resource type (Xhr, Fetch, Document, Script, etc.) and timing.
• Ad-tech / tracker analysis: Blocking or rewriting requests based on URL patterns.

Chrome DevTools Protocol (CDP) provides this via Fetch.enable and Network.requestIntercepted. obscura, being a standalone headless browser engine (not Chromium-based), needs its own first-class interception API.

---

Current workaround and its limitations

Since obscura does not yet expose a public interception API, we have been using two workarounds:

1. JavaScript injection via page.evaluate() after navigation
We inject a script that wraps XMLHttpRequest.prototype and window.fetch to collect request/response data into window.__xhr_records:

(function() {
    if (window.__xhr_records) return;
    const records = [];
    // Wrap XMLHttpRequest...
    const origOpen = XMLHttpRequest.prototype.open;
    XMLHttpRequest.prototype.open = function(method, url) { ... };
    // Wrap fetch...
    const origFetch = window.fetch;
    window.fetch = function(input, init) { ... };
    window.__xhr_records = records;
})();

Problem: Scripts injected via page.evaluate() run after the page's own scripts have already executed. Any XHR/Fetch calls made during page initialization (e.g., SPA bootstrap requests) are missed entirely.

2. Using preload_scripts (via Cargo [patch])
We discovered that obscura_browser::Page has a preload_scripts: Vec<String> field that executes scripts before any of the page's own <script> tags — exactly matching the CDP Page.addScriptToEvaluateOnNewDocument contract. This is the ideal injection point for an XHR interceptor.

However, this field is private (declared without pub), so external crates cannot push to it. To work around this, we currently use [patch.crates-io] in Cargo.toml to fork the dependency:

[patch.crates-io]
obscura = { git = "https://github.com/..., branch = "..." }

This approach has three major drawbacks:

• Maintenance burden: Every obscura upstream change requires rebasing our patch.
• Not a stable API: Private fields are not subject to semver guarantees; they can be renamed, removed, or refactored at any time.
• Unsafe pointer casts: To access the private field we resort to unsafe pointer transmutation, which is fragile and could break with compiler optimizations or struct layout changes.

3. Existing interception infrastructure is internal-only
We found that obscura-js already contains an intercept_tx / intercept_enabled mechanism in op_fetch_url (ops.rs L536-L630) that can intercept JS-level fetch/XHR calls and route them through a tokio::sync::mpsc channel. However, intercept_tx is also a private field on Page, and the interception is only accessible from within the crate.

Similarly, ObscuraHttpClient already has on_request and on_response callback registries (client.rs L264-L266), but these callbacks are not invoked by op_fetch_url — they only fire for navigation-level requests issued via ObscuraHttpClient.fetch_with_method(). The JS runtime uses its own reqwest Client via cached_request_client() (ops.rs L632), completely bypassing these callbacks.

---

Suggested API design

We propose adding a public, stable interception API to the Page type (or to the obscura crate's public surface). The design should cover three dimensions:

Option A: Expose preload_scripts as a public method

The simplest change that would enable the most use cases:

// On `Page`
impl Page {
    /// Register a script to execute before any page-defined scripts run.
    /// Equivalent to CDP's `Page.addScriptToEvaluateOnNewDocument`.
    /// Must be called before `Page::navigate()` / `Page::goto()`.
    pub fn add_preload_script(&mut self, script: &str);
}

This is a minimal, non-breaking change — the internal preload_scripts: Vec<String> already exists and is functional; only the visibility needs to change. It would allow external crates to inject XHR interceptors (or any initialization logic) without unsafe code or patching.

Option B: Add on_request / on_response callbacks to Page

A higher-level, more ergonomic API that avoids the need for JavaScript injection entirely:

/// Information about a JS-initiated request (XHR or Fetch).
pub struct FetchRequestInfo {
    pub request_id: String,
    pub url: String,
    pub method: String,
    pub headers: HashMap<String, String>,
    pub resource_type: ResourceType,  // Xhr or Fetch
}

/// Actions the interceptor can take on a request.
pub enum InterceptAction {
    /// Let the request proceed unchanged.
    Continue,
    /// Block the request entirely.
    Block,
    /// Fulfill with a custom response (CDP `Fetch.fulfillRequest` equivalent).
    Fulfill { status: u16, headers: HashMap<String, String>, body: String },
    /// Modify request headers before sending.
    ModifyHeaders(HashMap<String, String>),
}

impl Page {
    /// Register a callback that fires for every XHR/Fetch *request*.
    /// The callback can inspect, block, modify, or mock the request.
    pub fn on_request(&mut self, cb: Arc<dyn Fn(&FetchRequestInfo) -> InterceptAction + Send + Sync>);

    /// Register a callback that fires for every XHR/Fetch *response*.
    pub fn on_response(&mut self, cb: Arc<dyn Fn(&FetchRequestInfo, &[u8]) + Send + Sync>);

    /// Enable the built-in interception channel. 
    /// Returns a receiver for `InterceptedRequest` events.
    pub fn enable_interception(&mut self) -> tokio::sync::mpsc::UnboundedReceiver<InterceptedRequest>;
}

This design leverages the existing InterceptedRequest / InterceptResolution types already defined in obscura_js::ops (ops.rs L16-L39). The key change needed is:

1. Make op_fetch_url (the Deno op that backs JS fetch/XHR) invoke ObscuraHttpClient.on_request / on_response callbacks before/after the request, or route through the existing intercept_tx channel.
2. Expose the intercept_tx setup from Page to external consumers.

The ResourceType::Xhr and ResourceType::Fetch enum variants already exist in obscura_net::client (client.rs L70-L71), which shows this was anticipated — they just need to be plumbed through op_fetch_url.

Option C: Combine both — public preload scripts + interception callbacks

For maximum flexibility, we suggest implementing both Option A and Option B:

• Option A (low effort, high value) — can be shipped very quickly by simply making preload_scripts accessible.
• Option B (more thorough, Rust-native) — provides a type-safe, callback-driven API that does not require any JavaScript knowledge from the consumer and can support request modification and full response mocking (useful for testing).

---

Why this matters for the obscura ecosystem

obscura is positioned as a modern, embeddable headless browser engine. Its key differentiators include being Rust-native (no Chrome/Chromium binary dependency) and providing a library-first API. A first-class request interception API is a cornerstone capability for any headless browser, and its absence forces downstream projects to either:

• Write unsafe code to access private internals (fragile and unmaintainable), or
• Fork the dependency (high maintenance cost), or
• Fall back to less capable tools (Puppeteer/Playwright with full Chrome, defeating obscura's purpose).

Adding this API would significantly increase obscura's adoption in the crawling, automation, and security tooling communities.

---

We are happy to help

We have already done extensive analysis of the codebase and can confirm that the infrastructure for interception already exists but is not yet exposed publicly. Specifically:

• preload_scripts execution is already in place (page.rs L574-L581)
• intercept_tx/InterceptedRequest/InterceptResolution types are defined (ops.rs)
• on_request/on_response callback slots are on ObscuraHttpClient (client.rs)
• ResourceType::Xhr / ResourceType::Fetch are already defined (client.rs)

We would be glad to:

• Submit a PR that makes preload_scripts publicly accessible (with appropriate documentation and safety guarantees).
• Contribute to the design and implementation of a higher-level on_request/on_response callback API, including wiring op_fetch_url to invoke these callbacks.
• Help test the new API against real-world SPAs to ensure XHR/Fetch calls are captured comprehensively.

---

Discussion questions

1. Do you see any concerns with exposing preload_scripts as a public method?
2. Would an on_request/on_response callback API (Option B) be something you'd accept a PR for?
3. Are there existing plans or a roadmap item for this feature that we could help accelerate?

Looking forward to the community's thoughts!

[SGavrl]

Thanks for the detailed writeup and the codebase analysis, that made this easy to scope.

We went with Option C and implemented the full interception API on the embeddable obscura crate's Page. What's included:

• add_preload_script(&str) runs a script before any of the page's own <script> tags (the Page.addScriptToEvaluateOnNewDocument contract). No more unsafe field access or [patch] forking.
• enable_interception() returns a receiver of every JS fetch()/XHR request. Resolve each through its resolver with InterceptResolution::{Continue, Fulfill, Fail} to pass, mock, or block it, and Continue can rewrite the url, method, headers, or body before the request goes out. A rewritten URL is re-checked against the SSRF/private-network gate, same as a redirect, so an interceptor can't accidentally open a hole.
• on_request / on_response passive callbacks. The key fix is that op_fetch_url now invokes these for JS fetch()/XHR (previously they only fired for navigation), so SPA API payloads come through on on_response without reverse-engineering anything. Works in stealth mode too.

To your questions:

1. No concerns exposing preload scripts, it's public now via add_preload_script.
2. Yes, the on_request/on_response API is in, wired through op_fetch_url as you suggested.
3. No prior roadmap item, this is it.

One current limitation: resource_type reports Fetch for JS-initiated requests and doesn't yet split Xhr from Fetch, since the op backs both through the same path. Splitting them cleanly needs an extra op parameter and a snapshot rebuild, so I left it as a follow-up. Everything else from the proposal is covered. There's an end-to-end test (crates/obscura/tests/interception.rs) covering preload scripts, the channel, the passive callbacks, and a Continue URL rewrite.