Skip to content

v1.13.6

Choose a tag to compare

View all tags
@github-actions github-actions released this 03 Oct 09:09
v1.13.6
4d7e98a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Changes

Some sites answer a fetch with little more than "please use a modern browser", and until now that politely went into the cache as if it were the page. This PR treats those app shells as a failed fetch so the next backend gets a turn, and adds an AniList reader that gets anime and manga data from the site's public API instead of its client-side pages.

Highlights

  • Pages that only render with JavaScript no longer come back as a useless "requires a modern browser" result; the next fetch backend is tried and nothing empty gets cached
  • AniList title and character links now return the cast: each character's role, gender, age, description and voice actors
  • search_anilist finds an anime or manga by title so its cast can be read in one more step, including titles typed with accents or long-vowel marks
Technical changes

App shell detection

  • New is_js_shell_page() in prompture/capabilities/challenge.py, exported from prompture.capabilities. It looks for "needs JavaScript / a newer browser" markers in the first 1,000 characters only, so an article that mentions JavaScript further down still counts as content
  • JinaReaderBackend and DirectBackend raise RequestRejectedError on a shell. The fetch chain moves to the next backend, and since failures are never cached, a shell can't be stored for a page's full lifetime

AniList reader

  • New prompture/tools/web/readers/anilist.py with AniListReader, registered among the built-in readers after Wikipedia. It handles anilist.co anime, manga and character URLs by ID and queries the public GraphQL API at graphql.anilist.co. Requests always go to that fixed host; the URL only supplies the ID
  • Character entries include role, gender, age, a cleaned description (spoilers removed by default) and voice actors. API errors map to RequestRejectedError / BackendUnavailableError so they fail over like the other readers
  • search_anilist() is exported from prompture.tools.web to go from a title to its page. AniList's search matches romanized titles without macrons but returns unrelated entries with them, so fold_title() strips accents from Latin letters first ("Shippūden" → "Shippuden") and the query as given is tried only when that finds nothing; marks on other scripts, such as kana dakuten, are kept
  • AniList results use the 1-day cache lifetime, the same as Wikipedia
  • README reader lists and the cache lifetime table mention AniList

Details

Contributors

jhd3197
Assets 2
Loading