Problem
The current HTTP client uses reqwest with rustls-tls-native-roots. This emits a rustls TLS fingerprint (JA3), not a Chrome fingerprint. Any WAF with TLS fingerprinting (Cloudflare, DataDome, Akamai) detects the scraper immediately.
Current state (v1.0.6): Capa 1 implemented — WAF challenge detection in HTTP 200 responses via signature matching. This catches ~70% of false-200 pages but does NOT prevent detection.
Proposed Solution
Migrate from reqwest to wreq (731K downloads, 718 stars, hard fork of reqwest). Uses BoringSSL for real TLS/JA3/JA4 fingerprint impersonation.
use wreq::Client;
use wreq_util::Emulation;
let client = Client::builder().emulation(Emulation::Chrome131).build()?;
Scope
- Replace
reqwest with wreq + wreq-util
- Rewrite reqwest_middleware → wreq native tower middleware
- Replace reqwest_retry → manual backoff or wreq middleware
- Verify BoringSSL doesn't conflict with openssl-sys
- Update all 267 tests
Risks
- BoringSSL vs openssl-sys symbol conflict
- reqwest-middleware/retry tightly coupled to reqwest
- MSRV 1.85+ (project is 1.88+, OK)
Acceptance Criteria
References
Problem
The current HTTP client uses
reqwestwithrustls-tls-native-roots. This emits a rustls TLS fingerprint (JA3), not a Chrome fingerprint. Any WAF with TLS fingerprinting (Cloudflare, DataDome, Akamai) detects the scraper immediately.Current state (v1.0.6): Capa 1 implemented — WAF challenge detection in HTTP 200 responses via signature matching. This catches ~70% of false-200 pages but does NOT prevent detection.
Proposed Solution
Migrate from
reqwesttowreq(731K downloads, 718 stars, hard fork of reqwest). Uses BoringSSL for real TLS/JA3/JA4 fingerprint impersonation.Scope
reqwestwithwreq+wreq-utilRisks
Acceptance Criteria
References