Skip to content

Conversation

@thewebscraping
Copy link
Owner

@thewebscraping thewebscraping commented Oct 10, 2025

This pull request introduces two major enhancements that significantly improve the library's anti-detection capabilities and overall robustness:

  1. A Smart Rotator System: For automatically rotating proxies, headers, and TLS identifiers to effectively mimic authentic traffic.
  2. Robust Library Management: A dependency-free, self-managing mechanism for handling the lifecycle of the core tls-client C library.

Detailed Changes:

1. The Rotator System

  • Introduced Three New Rotator Classes: ProxyRotator, HeaderRotator, and TLSIdentifierRotator. These classes provide a unified sync/async API (next/anext methods) for managing and cycling through resources.
  • Seamless Client Integration:
    • Smart Defaults: The Client and AsyncClient now enable header and TLS identifier rotation by default, using a curated set of built-in, realistic templates.
    • Unified Parameters: The proxy, headers, and client_identifier parameters now accept multiple types: a single value (for static use), a list (to automatically enable rotation), or a pre-configured Rotator instance.
    • Proxy Feedback Loop: The Client now automatically calls mark_result/amark_result on the ProxyRotator, providing performance feedback (success/failure, latency) to optimize the weighted rotation strategy.

2. TLSLibrary Refactor

  • Dependency Elimination: The requests and tqdm dependencies have been completely removed. Library downloading now uses Python's built-in urllib and json modules, making the library lighter and more portable.
  • Intelligent Version Management:
    • The TLSLibrary class is now version-aware. It compares the version of the local C library file against a required version constant (LATEST_VERSION_TAG_NAME) hardcoded in the package.
    • Automatic Upgrade/Installation: If no local library is found, or if the existing version is outdated, TLSLibrary.load() will automatically download the correct version from GitHub.
    • Automatic Cleanup: After a successful download or validation of a new version, old library files in the bin directory are automatically deleted to prevent conflicts and keep the directory clean.

These changes transform the library from a powerful tool into a more intelligent, self-managing, and reliable solution for automation and scraping tasks.

@thewebscraping thewebscraping force-pushed the feat/rotator branch 9 times, most recently from 2a64dd5 to 2f75a01 Compare October 10, 2025 09:58
@thewebscraping thewebscraping changed the title feat: rotator supported Feature: Smart Rotators & Robust Library Management Oct 10, 2025
@thewebscraping thewebscraping force-pushed the feat/rotator branch 3 times, most recently from 30efe90 to 7eac6c5 Compare October 10, 2025 11:13
@thewebscraping thewebscraping merged commit d8d98c5 into main Oct 14, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant