v2.16.0 — The Crawler That Refuses to Die

The free-proxy rotation chain: harvest from public lists → validate → rotate
per request → feed every outcome back → refill on exhaustion. The engine
v2 and the self-healing freshness layer already existed; this change wires
them into the crawl path, fixes their deployment blockers, and adds the
first tests they ever had.

Fixed

• proxy_engine_v2: cache path was a hardcoded absolute path from another
  machine (/home/yu/...) — now project-relative .crawl_cache (env
  OMK_CACHE_DIR still wins). IPv6 probe no longer runs on the constructor's
  critical path (lazy, once). Burned proxies are now persisted (in-memory
  burn died on restart and the cache resurrected known-bad endpoints).
• Protocol detection read the URL scheme, so every https-served text list
  (all of GitHub raw) was labeled https. Detection now uses filename/query
  markers (http.txt, protocol=socks5, type=http, HTTPS_RAW,
  /socks5_list/) with the scheme excluded.
• aiohttp/aiohttp-socks were lazy-imported but declared nowhere — new
  proxy extra (pip install 'omk-crawl[proxy]'), included in all.

Added

• Freshness ("newest"): ProxyNode.harvested_at/source; a source
  re-serving a proxy touches its freshness instead of duplicating it;
  STALE_SECONDS (default 1h, OMK_PROXY_STALE_SECONDS) demotes stale
  entries in scoring and next(fresh_only=True) excludes them.
• rotate_crawl.py — RotationCrawler: one pool proxy per attempt over
  curl_cffi with the node's assigned TLS profile; outcome feedback
  (ok→score up, 429→per-proxy backoff, 407→burn, blocked/error→failure
  streaks); bounded synchronous refill when the pool empties mid-crawl;
  crawl_many() bounded concurrency. pool_cycles() is the bounded
  무한의 굴레 loop (cycles=0 → until cancelled).
• proxy:// scheme + ProxyPoolTool: proxy://stats (offline),
  proxy://refresh, proxy://loop?cycles=N&interval=S. Bad query params
  fail closed as results, never raise (BaseTool contract).
• CLI --proxy-pool: crawl a URL through the rotation pool.
• 30 offline tests (tests/test_proxy_rotation.py); suite now 714 passing.

Verification boundaries

• Live harvesting/validation against public proxy sources and the Firebase
  validation target are network paths exercised only by the live lane — the
  offline suite covers parsing, scoring, selection, feedback, persistence
  and CLI plumbing with fakes.
• Same-domain-style caveat: free proxies are untrusted third parties —
  credentials or personal data must not transit them; the pool is for
  public-content crawling only.

v2.13.0

Added

• Session-based X search (XSearchTool, x_search) and guest-token X trends
  (get_trends, trend_to_tweets, trending_with_content). These reuse the caller's
  own warmed session cookies; no API keys or OAuth secrets are stored.
• Shared target routing for CLI, sync/async public APIs, and dry-run plans.
  Native URIs and local files bypass the web chain; service hosts use hostname matching.
• Per-crawl execution traces, capability filtering, adapter exception isolation,
  max_fetches, and cooperative total_timeout (default 120 seconds). The default
  adapter cap is now 8. Browser disabling includes optional internal rendering;
  LLM adapters require explicit opt-in.
• Offline HAR inspection through analyze_har(), --tool har, and .har CLI routing.
  Metadata-only defaults omit headers, cookies, request bodies, and URL credentials,
  query strings, and fragments. JSON response bodies require explicit opt-in.
  File, entry, and decoded-body limits bound processing; malformed entries have indexed diagnostics.
• Router HTTP recovery for 408/429/502/503/504, bounded backoff, and Retry-After
  seconds/date support. Explicit terminal reasons and per-crawl attempt counts.

Fixed

• InsaneSearch shares its timeout across profiles and browser work, preserves HTTP
  status/headers, and stops on authentication or server backpressure. SDK absence is
  reported instead of escaping as an import error; browser close runs in finally.
• Invalid HTTP SDK impersonation profiles are rejected before creating a session.
  CI explicitly installs the existing curl extra required by transport tests.
• Async rate-limit waits no longer block the event loop or hold a global lock while
  sleeping. Robots checks run off-loop, and async convenience arguments configure the router.
• Auth/rate-limit/unsafe-method stops return the terminal response rather than a
  larger earlier failure. Pending retries remain cancellable; unsafe methods are not retried.
• CLI version output uses the package's __version__ instead of a stale literal.

v2.11.0

Added — one-keypress GitHub star button

• After the 3rd successful interactive run, omk-crawl asks once whether you want to
  star the repo. [enter] stars it in place through an authenticated gh CLI,
  [b] opens the browser, [n] never asks again.
• omk-crawl --star — star on demand, no waiting.
• Never touches stdout (> out.md / --json | jq stay byte-clean), never speaks on a
  non-TTY, never in CI or under pytest, never writes state unless the session is interactive.
• Opt out permanently: OMK_CRAWL_NO_STAR=1 or the vendor-neutral DO_NOT_TRACK=1.
• State: $XDG_STATE_HOME/omk-crawl/star.json (%LOCALAPPDATA% on Windows).
• 27 tests in tests/test_star.py, plus a real-PTY verification of the prompt path.

v2.10.0 — mobile · baemin · appstore · reddit

Ship train since 2.6.0: mobile layer, Baemin live shops, iOS App Store meta, Reddit JSON bypass.

Added — Reddit target client

• RedditClient: old.reddit session warmup → .json listings
• Bypasses www "Please wait for verification" challenge (no browser JS)
• Methods: subreddit, search, user, comments
• CLI: reddit://r/programming, reddit://search?q=…, plain reddit.com URLs
• Tool registry: reddit
• Docs: references/tools/reddit.md
• Live verified: r/programming, r/korea JSON 200

v2.5.0 — detection-aware routing · adapter contract · benchmark

[2.5.0] — 2026-07-24

Quality release driven by external code review — shifts weight from docs/demo
to routing quality, adapter contract, and a real benchmark (the three levers
the review identified for moving 78 → 90).

Added — detection-aware routing (Phase 2)

• routing.py — ROUTE_TABLE maps each BlockType to a preferred tool
  order (TLS→curl_cffi, JS→crawl4ai, CF/WAF→scrapling). preferred_order()
  reorders the remaining chain after a detected block; reorder_tools() is
  stable for duplicate names. AUTH_REQUIRED returns [] — we never escalate
  into an auth bypass.
• SmartRouter.crawl() / crawl_async() now reroute mid-escalation based on
  the live detection (once, above a 0.5 confidence threshold), skip auth
  blocks, and record rerouted_to in result metadata.
• diagnose() now emits the routing table so callers see why a block type
  would reorder the chain.
• 17 routing tests (table, permutation, stability, router reroute integration).

Added — unified adapter contract (Phase 1)

• BaseTool.capabilities (frozenset) + COMMON_KWARGS (timeout/proxy/headers/
  cookies/session). supports(), unsupported_features(), contract_metadata()
  report requested-but-unsupported features explicitly instead of silent no-ops.
• curl_cffi: proxy normalization + cookies; crawl4ai: headers; scrapling: proxy.
  All four core adapters declare capabilities.
• 15 contract tests.

Added — browser-use cost guards (Phase 5)

• max_steps, max_cost_usd, deadline_s caps; excluded from the chain when
  no LLM key is configured; failure taxonomy (nav/login/timeout/model/unknown);
  dry_run mode reports guardrails without spending.

Added — benchmark harness (Phase 3)

• benchmarks/sites.yaml — 20 sites across static / JS / soft-wall / hard-wall.
• scripts/bench.py — mock (CI) + live modes; success@1/final, p50/p95 latency,
  bytes, tool path, cost proxy → benchmarks/latest.json + Markdown table.
• README benchmark table from a polite live run (7/7 success@1).

Changed — consistency (Phase 0)

• Tool-count framing unified: 6 runtime adapters (4 core escalation + 2 aux),
  skill catalog references 10. Version aligned across pyproject/__init__/tag.
• GitHub topics added (web-scraping, crawler, anti-bot, markdown, python, …).

v2.4.0 — resilience · adaptive fetcher · Naver/Baemin clients

[2.4.0] — 2026-07-24

Large capability + stability release. All new modules keep the zero-dependency
core: heavy deps (curl_cffi, beautifulsoup4, playwright, charset-normalizer) are
imported lazily inside functions, so import omk_crawl still works with no
extras installed. Install omk-crawl[targets] for the Naver/Baemin clients.

Added

• resilience.py — TokenBucket (sync + acquire_async), RetryPolicy/
  retry with exponential backoff, ResponseCache, HeaderStore,
  ImpersonateRotator (8 TLS fingerprints, auto-excludes failed profiles),
  EndpointChain fallback, ensure_playwright() auto-installer.
• stability.py — CircuitBreaker (per-host, closed/open/half-open),
  BreakerRegistry, SessionManager (connection reuse + cookies),
  TimeoutBudget, structured get_logger.
• adaptive.py — AdaptiveFetcher with a 3-tier strategy ladder:
  DIRECT (curl_cffi TLS impersonation) → SESSION → RENDER (Playwright with
  automatic XHR/fetch interception). Discovers a site's real APIs without
  prior knowledge. CapturedCall.decoded_text() auto-detects legacy Korean
  encodings (EUC-KR/CP949) via Hangul-scored candidate selection.
• async_batch.py — AsyncBatchFetcher: concurrent crawling bounded by a
  shared token bucket + per-host circuit breakers (~5× throughput on multi-page
  fetches).
• baemin.py — BaeminClient built on an APK-verified endpoint registry
  (DEX string extraction of com.sampleapp v16.15.0), replacing guessed URLs.
• naver.py — NaverLandClient (public real-estate markers) and a fully
  rewritten NaverCafeClient: no-login REST article list / notices / popular /
  menus / multi-page crawl, plus full article body via browser-decoded frame
  text (sidesteps the legacy EUC-KR yortapaper iframe). Session methods
  check_login() / can_access() validate YOUR OWN session for content you
  are authorized to see.
• cookies.py — CookieManager: import your own browser session from
  JSON (EditThisCookie/Cookie-Editor), Netscape cookies.txt, or a Cookie:
  header; filters expired cookies; injects into curl_cffi + Playwright.
• examples/ — baemin_reviews.py, baemin_mitm_capture.py,
  naver_cafe_public.py, naver_land_public.py,
  naver_private_cafe_own_session.py.
• scripts/verify_endpoints.py — 17-check endpoint + component suite.

Changed

• __init__.py exports expanded to 38 symbols; __version__ → 2.4.0.
• pyproject.toml: new targets optional-extra; all updated; description
  refreshed.

Legal scope

This release does not bypass, forge, or defeat authentication. Accessing
private/login-gated content is supported only via the user's own legitimate
session (CookieManager), i.e. automating access the account is already
authorized for. Unauthorized access is out of scope (정보통신망법 §48).