4277332

LXManga: fetch through the WebView when Cloudflare rejects OkHttp

The device log settled it:

    [LxManga] GET https://lxmanga.space/danh-sach?... | jar=AndroidCookieJar cookies=[_ga,_ga_XZQ22BMRNK,cf_clearance,cf_chl_rc_ni]

and then nothing — no "-> HTTP <code>" line, i.e. the request never came
back because the app's CloudFlareInterceptor turned the response into a
CloudFlareProtectedException. So: the jar is the WebView-backed
AndroidCookieJar, the HttpOnly cf_clearance *is* in it, OkHttp *does*
send it — and Cloudflare challenges the request anyway.

That is the known behaviour for a clearance bound to the client it was
issued to (Cloudflare ties cf_clearance to the TLS/JA4 fingerprint and
UA as well as the cookie when the site is in "Under Attack" mode). A
Java/OkHttp client is not the Chromium that solved the Turnstile, so
presenting the cookie cannot be enough — which is why the whole
"CloudFlareActivity + shared cookie jar" loop never converges on this
site and kept asking the user to solve the challenge again.

fetchDocument now tries two transports:

  1. plain OkHttp, as before — used whenever Cloudflare accepts it. A
     Cloudflare rejection (CloudFlareProtectedException /
     CloudFlareBlockedException — both `IOException` subclasses from the
     app, detected by class name so the parser keeps compiling against
     the plain tsuki library) is caught instead of propagated, and a
     challenge body that no interceptor caught is detected explicitly.

  2. the app's WebView via `evaluateJs` + `fetch`: `evaluateJs(url, …)`
     loads a stub document whose origin is the page URL, so the fetch is
     same-origin and goes through the WebView network stack — the
     clearance *and* the fingerprint it was issued to. Same HTML,
     parsed with Jsoup, so every selector is untouched. If the WebView
     is challenged too, AuthRequiredException points the user at the
     existing "Đăng nhập" row, whose WebView is the one that can clear
     it.

Result handling got fixed along the way:

  * `WebView.evaluateJavascript` hands string results back JSON-encoded
    (`"[{\"url\":…}]"`), while the parser assumed raw text — so
    getPages could never have parsed a chapter's URL array. Results are
    now unwrapped via JSONTokener.
  * Older WebView builds do not resolve the promise an async IIFE
    returns and hand back `{}`; fetchDocument then retries with a
    synchronous XHR script that cannot be mishandled.
  * getListPage returned all 120 `div.relative` nodes for the 60 mangas
    the site renders (every card exists twice, mobile + desktop), which
    skews the paginator. Now collapsed by URL.
  * Diagnostics kept (jar class, cookie names, clearance *age* parsed
    from the token, which transport served the request).

Tests: 40/40 in the JVM harness, incl. the new fallback paths (rejection
→ WebView HTML, quoted vs unquoted results, sync-XHR retry, challenge
shell, AuthRequired message).

fd7d16c

LXManga: drop lxmanga.xyz (522) + trigger requestBrowserAction on CF

Two things broke the user-facing flow of the previous commit:

1. lxmanga.xyz (the only domain in the fallback slot) was 522 from
   the sandbox. As soon as the app's catalog probe on .space 403'd
   on Cloudflare's 'Just a moment...' page, the host-fallback logic
   the user runs in the reader silently flipped the saved setting
   from .space to .xyz and the parser never recovered. Drop .xyz
   entirely so the user stays on the only working host.

2. Even on lxmanga.space, the very first catalog request hits the
   'Under Attack' Cloudflare challenge, so OkHttp got 403 'Just a
   moment...' and there was no path forward. Now every catalog GET
   goes through a fetchDocument() helper that runs
   CloudFlareHelper.checkResponseForProtection on the response and
   hands the URL to context.requestBrowserAction() if the page is a
   challenge — same pattern CuuTruyenParser uses for the same CF
   front. After the user clears the challenge in the custom tab the
   OkHttp jar ends up with cf_clearance and the rest of the parser
   works without further interaction. MangaParserAuthProvider.authUrl
   is kept as a safety net for the WebView path (PAGES_SCRIPT's
   same-origin fetch needs cf_clearance in the WebView's
   CookieManager too).

The fetchDocument helper also covers getDetails() and the tag
listing (the previous loadAvailableTags), so the CF guard is
applied uniformly. The KDoc at the top of the class is rewritten
to call this out and to explain why cf_clearance is HttpOnly and
therefore cannot be read from WebView.document.cookie to copy into
OkHttp the way Hv2tComics does for the age-gate cookie.

bae4d55

LXManga: present one browser identity — UA + matching client hints

Answering "why not just use CloudFlareInterceptor": we do — it is
inherited from `context.httpClient` on every parser and the device log
proves it fires (`okhttp rejected by Cloudflare (...):
CloudFlareProtectedException`). What the interceptor cannot do is make
Cloudflare *accept* a clearance that was issued to a different client.

The client it recorded was the app's WebView. So send exactly that
identity instead of half of it:

  * the WebView user agent (userAgentKey is left at the app default);
  * `Sec-Ch-Ua`, `Sec-Ch-Ua-Mobile`, `Sec-Ch-Ua-Platform` derived from
    that user agent — these were missing entirely, and a request that
    claims to be Chrome on Android but sends no client hints at all is
    a contradiction Cloudflare can see;
  * `Accept-Language` from the device locales and
    `Upgrade-Insecure-Requests: 1`, both of which the WebView sends.

`Sec-Fetch-*` is deliberately absent: those values differ per request
type and a wrong one is worse than none.

Tests: 35/35, including the new header assertions.

ba30345

LXManga: drop MangaParserAuthProvider so the 'Đăng nhập' row goes away

The user pointed out that login and 'solve captcha' are two different
features, and that showing a 'Đăng nhập' (Sign in) row for a site that
has no account system at all is actively misleading. lxmanga.space
only has the Cloudflare Turnstile captcha — there is no username,
no password, no 'Đã đăng nhập' state to ever reach.

The fix: drop the [MangaParserAuthProvider] interface entirely so
the host app's source-settings screen has nothing to render for
this parser. Captcha solving is the only auth flow, and the user
hits it automatically the first time the catalog request returns
the 'Just a moment…' page — [fetchDocument] detects the response,
hands the URL to [MangaLoaderContext.requestBrowserAction], and
the host app surfaces its own captcha prompt (custom tab, dialog,
whatever it normally does for 'solve this captcha').

Compared to the previous design (78ee241):
  * No more 'Đăng nhập' row in source settings — the user is not
    asked to 'sign in' anywhere.
  * No more [authUrl] in the in-app WebView — the only thing the
    app does is run [requestBrowserAction] when the catalog 403s.
  * No more [isAuthorized] / [getUsername] / [MangaParserAuthProvider]
    at all. The OkHttp cookie jar check that used to live in
    [isAuthorized] is gone too: with no auth provider the app
    has no way to invoke us, so there is no need to expose state.
    The captcha prompt is the *only* entry point.

The chapter reader (PAGES_SCRIPT + evaluateJs) is unchanged — the
obfuscated image-URL builder still has to run inside a browser
context.

Tests: 24/24 pass. The harness covers the auth state, the static
request headers, and all of [getPages]'s JSON / error paths.
The live catalog flow is exercised by the CI runner.

af2d2df

LXManga: widen the webview retry window, time each attempt, ping the …

94dfc19

LXManga: re-add MangaParserAuthProvider so the captcha has a home

Two things were missing on the device:

  1. No 'Đăng nhập' row in the source settings. That row is the only
     on-demand entry point for the Cloudflare challenge, and the app
     renders it only when the parser implements MangaParserAuthProvider
     with a non-empty authUrl (SourceSettingsFragment:
     `authProvider != null && authUrl.isNotEmpty()`), so dropping the
     interface in ba30345 made the captcha reachable only by first
     tripping a 403 while browsing.

  2. The clearance obtained from the challenge was not usable by the
     plain-HTTP client, so catalog/details/tags kept failing over
     OkHttp after the captcha was solved.

Both are now handled the way the host app is actually built:

  * authUrl = https://<domain>/ — the homepage is Cloudflare-gated, so
    opening the row renders the Turnstile challenge in the app's in-app
    browser (SourceAuthActivity, which uses the parser's own
    User-Agent via getRequestHeaders()).

  * isAuthorized() = "is cf_clearance in the jar?". Usagi's cookie jar
    is AndroidCookieJar, i.e. android.webkit.CookieManager, so the
    HttpOnly cookie the browser writes is visible here even though
    document.cookie would not show it. The row is therefore disabled
    while a clearance is valid and SourceAuthActivity closes itself
    with the 'Authorized' toast once the user clears the challenge.

  * No username: getUsername() returns null, the site has no accounts.

The catalog path itself is unchanged (plain OkHttp via fetchDocument):
the app installs CloudFlareInterceptor on the HTTP client every parser
inherits, it throws CloudFlareProtectedException on a 'Just a moment…'
body, and the resolver opens the dedicated CloudFlareActivity, which
watches the same jar for a new cf_clearance and retries the request.
The manual checkResponseForProtection() + requestBrowserAction()
fallback stays for hosts without that interceptor.

Chapter error messages now point at the 'Đăng nhập' row instead of
'the source page in a webview', which is what actually fixes a stale
clearance.

Tests: 29/29 pass in the JVM harness (auth state incl. cf_clearance vs
non-CF cookie, static headers, all getPages JSON/error paths). Live
catalog is exercised by the CI runner.

78ee241

LXManga: use OkHttp + requestBrowserAction (CuuTruyen pattern) — show…

6ff1bff

LXManga: add on-device diagnostics (jar class, cookie names, HTTP codes)

The captcha still loops on the device after the auth row landed, and the
app produces no parser-level logging, so there is nothing in logcat to
reason from. Add a few `println` lines (logcat tag System.out) that pin
down *where* the chain breaks:

* `jar=<class>` — which CookieJar the parser got. Usagi normally binds
  AndroidCookieJar (reads android.webkit.CookieManager, so the HttpOnly
  cf_clearance written by the challenge WebView is visible); if the
  factory fell back to PreferencesCookieJar the WebView cookies are
  invisible to every plain-HTTP request and the challenge can never
  converge. This one word decides which of the two worlds we are in.
* `cookies=[...]` — the cookie *names* the jar reports for the domain
  (no values), i.e. whether cf_clearance/XSRF-TOKEN/lxmanga_session are
  actually there when the request is built.
* `-> HTTP <code>` — the response code of each catalog request, so we
  can tell "no clearance" (403) from "clearance sent but rejected" from
  "request never retried".
* getPages logs the same, plus the first 120 chars of what the WebView
  script returned.

Temporary: remove once the failure point is known.

682409d

LXManga: route every HTTP request through the WebView (no more 're-au…

5e358f6

LXManga: chapters — download with OkHttp, never prompt for a login

Two problems reported from the device: the catalogue works now, but
opening a chapter popped the "Đăng nhập" screen — on a site that has no
accounts at all, where solving the Cloudflare challenge is the entire
gate.

  1. The prompt was ours. getPages threw AuthRequiredException for
     *any* failure, and that exception is the app's "ask the user to
     sign in" signal, so it opened SourceAuthActivity — which then
     closed itself immediately (isAuthorized() sees the clearance) and
     reappeared on the retry. `fail()` now only asks for
     authentication when there is genuinely no clearance; otherwise it
     raises a plain IOException with the reason.

  2. The chapter fetch did not fit the WebView's budget. A single
     `evaluateJs` call is capped at 4 s by the app, and the script was
     expected to download a ~900 kB page *and* run the page's
     obfuscated decryptor inside it. So the download now happens over
     OkHttp first (no such limit, and Cloudflare already accepts this
     client — the catalogue proves it), and the WebView only has to
     `document.write` the markup we hand it and wait up to 3 s for the
     decrypted `window["_0x…"]` array.

The render step still cannot be avoided: the page ships no image URLs,
it ships a script that decrypts them.

Tests: 36/36, including "challenge on the chapter fetch propagates to
the app", "failure with a clearance → IOException, never the login
screen", "failure without one → AuthRequiredException".