v4.14.0

Added

• 51 new sources, across English, Spanish, Chinese, Russian and Turkish. Most cost almost nothing to add: two new templates read any site built on WordPress categories or Blogger labels, which is what a large share of translation blogs are.
• A link from an unsupported site can now be read by guessing the page structure — off by default, via Guess Unsupported Sites. Chapter text comes out reliably; the chapter list is inferred, so every result says how much of the list it could account for. Read that before trusting the download: a site that hides part of its list behind a button produces a book quietly missing chapters.
• A Proxies screen, and proxies can now say what kind of address they are. Only ISP, residential and mobile addresses get past a site that blocks on reputation, and every proxy used to be read as datacenter — so a residential proxy bought for exactly that was never used for it. Existing configuration imports automatically.
• Six crawler settings you could not reach before, including how many requests a site gets at once, and whether pages may be read from the Web Archive. Archive reading can recover a novel from a site that is gone for good; it is off by default because it sends the first visit to every site to a snapshot.

Changed

• A rewritten HTTP layer, built around how sites detect crawlers. Across 150 real source hosts it retrieves more of them, and a challenge is answered by a real browser whose clearance is then reused.
  • Firefox solves by default, Chrome as the fallback, with a Challenge Solver Browser setting to name one. Whichever browser solves decides what every later request has to look like, and Firefox reaches the most sites.
  • The browser stays hidden, and shows itself only when hiding fails. A hidden browser gets past every site a visible one does, so the window is worth opening only for the challenge nothing can answer alone — and once it is open it waits five minutes instead of ninety seconds, because somebody is there to finish it. A new Challenge Solver Window setting pins it to always hidden or always visible; on a server it stays hidden whatever you pick, since nobody could see it. Replaces the old headless switch.
  • The Docker image ships Firefox on arm64 as well, so an arm64 image can solve for the first time. Set TZ to the timezone your address looks like it is in: with the container clock left wrong it cleared one challenged site of six, and with it right, all six.
  • New impersonate setting; selenium_grid is gone.
• 50 source domains are flagged as rejected — parked, redirecting into an ad network, or resold. None reported an error: a page full of adverts answers 200, so the crawl succeeded and produced an empty book.
• wordexcerpt and webnovelonline rebuilt against the APIs their sites now use, after both became single-page apps their old selectors read as empty. webnovelonline now walks its whole listing — 1305 chapters where the page shows 50.
• Forty sources were fetching fewer chapters at once than they could. Ten go from one worker to three, thirty from two.
• The README is rewritten and the site list moved to SOURCES.md, so the front page is about installing and using the app rather than 600 lines of table.

Fixed

• An empty chapter is no longer saved and marked finished — the most common way a source breaks, with no signal for it anywhere. It is retried now, and chapters already stored empty are reopened on their own. A challenge page served as 200 is no longer parsed as chapter content either.
• A failed job says why, instead of showing a stack trace — which defence is blocking, and whether any setting could help.
• allow_fallback_on_proxy_miss said the opposite of what it does. The direct address joins the proxy list ranked ahead of Tor and datacenter ones, so with a tor-pool configured at the default, requests were leaving from the machine's own address.
• lncrawl search now ends when it says it will. A 10 second timeout measured 74; the same search finishes in 16 and returns more.
• The server no longer leaks memory on every failed request — 66 MB per 800 requests, now flat over 6,400.
• Smaller: a page that never finished rendering is a diagnosis rather than a crash; a source's chosen HTML parser is finally used; a source's main address no longer varies between runs; what a crawl learned survives the command exiting; chireads search results have titles again.

Full Changelog: v4.13.1...v4.14.0

v4.13.1

Fixed

• Translator Dashboard — Fixes dependency import in PyInstaller build

Full Changelog: v4.13.0...v4.13.1

v4.13.0

Added

• In-process translation engine — the built-in translator backends (Google/Bing/Baidu/Microsoft/Lingva) are replaced by the external lncrawl-translator package run in-process, giving multi-engine routing and failover managed from an admin-gated dashboard mounted at /api/translator. New config: a translator.enabled master switch to turn the feature off, plus translator.request_timeout and translator.config_file
• Novel glossary — a per-novel, per-language translation glossary (new add_novel_glossary_table migration) is sent with every request and merged back from the engine so names stay consistent across chapters; manageable via the API and kept when a translation is deleted
• Single-volume artifacts — POST /api/jobs/create/make-artifacts accepts an optional volume (volume serial number) to build an e-book covering just that volume; artifacts record the volume in a new volume column (new artifact_volume_column migration) and can be filtered by it via GET /api/artifacts. Whole-novel listings (list_latest/novel page/download emails) still show only full-novel artifacts
• Per-translation delete — remove a single language's translation of a novel without touching the others (the glossary is preserved)
• Translated titles & volume filtering — list_latest/get_latest support a volume filter, and the chapter/volume detail endpoints accept a language query param to return translated titles
• Library favorites — favorite a library and list favorites (new library_favorites migration)
• Reading history page — new endpoints and response models surface reading stats and continue-reading
• Basic-tier translation — single-novel translation jobs are available to the basic tier
• Automatic novel language — a novel's language is detected and set while crawling

Changed

• Per-source request rate limiting — request_concurrency and the rate knob are merged into a single per-source request_rate_limit, now enforced across every concurrent job hitting a source; FetchService uses one scraper instance per thread
• Resilient desktop webview startup — startup is reworked for a smoother experience: readiness is gated on /health, with a fail-fast path when the server exits early and a system-browser fallback
• Faster CLI startup — the FastAPI import is deferred off the crawler/source import path
• Sliding sessions — the current session is kept alive via a refresh token on /me, now with an absolute cap (new server.session_max_lifetime, default 30 days) and role/tier re-derived from the live user on each refresh
• Translator dashboard proxying — proxied via base-href injection instead of URL rewriting; only the translator API is admin-gated

Fixed

• SMTP — reconnect stale connections before sending mail
• Glossary merge race — concurrent translation jobs of the same novel no longer collide on the glossary unique constraint or drop each other's terms
• Invite reply loop — the incoming Subject/Message-ID reused for threaded invite replies are sanitized, so a crafted email can no longer wedge the invite handler into an endless reprocess loop
• Artifact listing count — the paginated total now applies the same filters as the results, fixing page counts
• Source crawlers — migrated many sources to the declarative request_rate_limit attribute

v4.12.0

Added

• Novel list filtering — GET /api/novels gains sort modes (popular, updated, created, chapters, title_asc, title_desc) via a new NovelSort enum, backed by a novel_popularity migration and activity-derived popularity scoring
• Per-novel tags — tag attachments now live in a dedicated NovelTag table (new novel_tags migration) instead of being embedded, with supporting DAO/service changes across tags.py, novels.py, and crawler.py
• CLI resume & rate limit (#3105) — lncrawl crawl gains --resume/--missing to download only not-yet-crawled chapters and --rate-limit to throttle requests; --resume and --refresh are mutually exclusive
• novelarrow.com — new source crawler

Changed

• Enums stored as plain scalars — DAO models no longer use native DB enum types; enum columns are stored as plain scalars, removing the need for Postgres enum-sync migrations (new drop_native_enums migration)
• Dialect-split schema evolution — services/db.py schema evolution is now split by dialect, and the SQLite DB is rebuilt from the current models while preserving data
• Sources list not cached — GET /api/sources no longer caches its response

Fixed

• novelfull — stop downloading duplicate "half" chapters, and drop an unnecessary soup.decompose call in chapter-body parsing
• Source language generation — corrected language derivation in the sources helper

New Contributors

• @HaProxyMage made their first contribution in #3104
• @shagunchandel made their first contribution in #3103

Full Changelog: v4.11.0...v4.12.0

v4.11.0

Generate source index

v4.10.0

Added

• Activity heatmap — hourly usage heatmap bucketing events by day-of-week × hour-of-day using portable integer epoch math (identical on SQLite and PostgreSQL) and shifted by a tz_offset query param to reflect the viewer's local time, plus additional activity-dashboard metrics — useful for spotting low-traffic deploy/maintenance windows
• One runner per domain — the job scheduler now guarantees a single runner processes a given domain at a time; new job_domain migration with supporting DAO/service logic
• GitHub feedback — feedback is submitted directly as GitHub issues via utils/github.py; the local feedback table is dropped (new migration)

Changed

• DB schema validation reworked — schema validation now runs through services/db.py on startup, with a dev migrate command and a lint-workflow hook
• Job runner hardening — added a drain loop and failure safety net, consolidated queue claims, only cancels stuck jobs on runner reset, rests between iterations, and runs gc during scheduler reset
• openai moved to a dev dependency; CI workflows optimized
• Updated lncrawl-scraper

Fixed

• freewebnovel (#3060) — source corrected
• Assorted cleanup and minor fixes across the crawler, binder, mail, and static-file middleware

Full Changelog: v4.9.0...v4.10.0

v4.9.0

Added

• IMAP inbox listener — MailService gains a start/close lifecycle and an IMAP IDLE loop (via imap-tools, defaults to ProtonMail Bridge on localhost:1143). Incoming mail from an unregistered sender automatically triggers an invite through the admin referral flow; known users are left untouched. Gated behind a new imap_enabled config flag (off by default); outbound mail is likewise gated behind smtp_enabled
• Fetch-missing / fetch-latest jobs — two new JobType values, FETCH_MISSING (60) and FETCH_LATEST (61), with dedicated handlers and POST /api/job/create/fetch-missing / fetch-latest endpoints for filling chapter gaps and pulling newly released chapters. New sync_jobtype Alembic migration for PostgreSQL compatibility
• Admin activity dashboard — GET /api/admin/activity?type=<kind>&days=<n> backed by four focused service methods (summary, daily active users, per-type trend, top users); new response models in server/models/activity.py
• Artifact download tracking — new ActivityType.ARTIFACT (12) recorded when a file under /artifacts/ is served; other static downloads remain DOWNLOAD
• Browser navigation middleware — a /browse/* proxy (BrowserNavigation) fetches remote URLs through the scraper engine and rewrites links so navigation stays within the local server, injecting a JS interceptor that routes fetch/XHR/anchor clicks through the same prefix. Adds a browse_helper utility and an assets/scripts package
• Proxy toggle — new config option to enable/disable proxy usage in the crawler

Changed

• Multiple proxies supported — proxy config reworked to accept a list of proxies; lncrawl-scraper bumped accordingly
• Activity worker count — user-activity tracking now uses runner_concurrency as the worker count

Fixed

• Active-user count — get_admin_summary counted activity types instead of distinct users; replaced with a COUNT(DISTINCT user_id) subquery, fixing PostgreSQL compatibility
• Sources — updated aquareader.net (#3035) and lightnovelpub.org; regenerated source index

Full Changelog: v4.8.0...v4.9.0

v4.8.0

Added

• Job notifications — JobNotificationService dispatches email on job state changes (pending → running → success/failure) via a background TaskManager; triggered from handler helpers (_set_running, _set_success, etc.)
• Docker healthcheck — server container now exposes a /health probe

Changed

• Job runner refactored into typed handlers — JobRunner now dispatches via a _HANDLER_REGISTRY of BaseHandler/BatchHandler subclasses; each job type has its own module under scheduler/handlers/
• Web app synced before Docker build — lncrawl-web artifacts are pulled in as part of the Docker build step
• crawler_version stamped on novel/chapter updates — upserts now use a merge strategy to preserve existing data

Fixed

• Server hangup — root cause of hang addressed (event lock contention / crawler resource leak)
• Server crash — crawler resource leak on shutdown fixed; Docker healthcheck added
• crawl.py (#3030) — regression in CLI crawl flow corrected
• Torproxy — re-enabled after an unintended regression

Full Changelog: v4.7.0...v4.8.0

v4.7.0

Added

• Background search jobs — novel search is now a proper background job with two new JobType values:
  • SEARCH_SOURCE — searches a single crawlable source; trigger via POST /api/job/create/search-sources?domain=…
  • SEARCH_ALL_SOURCES — fans out across every searchable source, spawning one SEARCH_SOURCE child per source; idempotent on retry
  • JobRunner handles execution: results stored in job.extra, matched URLs create a NOVEL_BATCH child job
  • New Alembic migration (add_jobtype) for PostgreSQL compatibility
• PAUSED job status — new JobStatus.PAUSED enum value for finer job lifecycle control
• Per-tier search-job rate limiting — BASIC users are capped at 1 concurrent search while the general active-job quota remains independent; search query length validated (2–50 chars); results sorted by match ratio
• NovelFire search — SEARCH capability added to NovelFireCrawler (#3009)

Changed

• Removed vendored lncrawl/cloudscraper — the embedded Cloudflare-bypass fork (v1/v2/v3 handlers, captcha integrations, JS interpreters, 7 913-line browsers.json) has been removed; HTTP scraping is now delegated to the lncrawl-scraper package
• JavaScript engine replaced: PyExecJs → quickjs → exejs — lighter dependency, no Node.js or external runtime required
• Proxy support in scraper — the lncrawl-scraper integration now supports proxies; build-essentials added to Docker base image (#3014)
• BrowserTemplate merged into SoupTemplate — BrowserTemplate is integrated directly into the soup template hierarchy rather than being a standalone class; all browser-based sources refactored accordingly
• Job service hardening — event locking and improved update logic in JobService; request timeouts in the scraper adjusted
• Docker improvements — faster image builds; updated compose.yml and server-compose files; fixed unintended root access in server-compose
• truyenfull: updated domain and search URL; base_url changed to a list to support multiple domains (#3010)

Fixed

• Security — path-traversal / static-file exposure vulnerability fixed in app.py and staticfiles.py (#3005)
• katreadingcafe — chapter link validation logic corrected to filter out non-chapter URLs (#3026)
• Race condition — parallel search result aggregation could yield inconsistent data under concurrent writes; fixed with proper locking
• EPUB + NovelFire (#2993):
  • Duplicate chapter title and serial number removed from chapter body content
  • download_chapter_body header extraction improved (regex + text normalisation)
  • EPUB serial heading logic refactored
• Source loading on restart — a failure loading one source no longer aborts the full reload cycle
• Cover download — full error stack trace suppressed for non-critical cover fetch failures
• PyInstaller packaging (setup_pyi) — fixed a regression in frozen-binary builds

New Contributors

• @GabrielCWT made their first contribution in #3009
• @augustanational made their first contribution in #3010
• @templeofshadow made their first contribution in #3026

Full Changelog: v4.6.0...v4.7.0

v4.6.0

What's Changed

New Features

• Novel Recommendations — the server now suggests related novels based on what you're reading
• Machine Translation — full translation service with multiple backends (Bing, Google, Lingva, Baidu) with automatic failover; translates chapter content, chapter titles, and artifacts (EPUB/etc.)
• Granular translation job types — translation tasks are now split per-resource (chapter, volume, title) instead of one monolithic TRANSLATION job, giving finer progress tracking
• Referral / invite system — users can invite others via email with a referral link
• Expanded browser detection — Brave, Vivaldi, Yandex, and Whale are now recognized for app-mode launching alongside Chrome/Edge
• More supported translation languages

Improvements

• Browser automation migrated from Selenium to nodriver for more reliable JS-rendered site scraping
• Switched to a custom caching layer instead of cachetools for better control
• Announcement banners improved in the web UI
• Chapter body cleaning improved when downloading
• User activity tracking added (page visits, static file downloads)
• Webview fallback now shows just the terminal when no app-mode browser is found
• Tightened API access control; auth guards now use Security() instead of Depends()
• Removed initial content when a language is pre-defined
• Invitation email subject line updated

Bug Fixes

• Fixed SQLite compatibility issue with migrations (batch_alter_table)
• Fixed Calibre-based artifact generation when using translations
• Fixed searching regression
• Fixed chapter fetch/translate functions not passing user ID correctly
• Fixed select_descendants typo in security module (#2966)
• Fixed invalid URL exceptions crashing fetch_chapter and fetch_image
• Fixed ensure_load crashing when sync thread was already cleaned up
• Fixed app startup issues

Source Updates

• wtr-lab.com — multiple fixes and updates
• novelfire.py — several iterative fixes
• Chapter title tag removal extended to <h4> elements
• More sources flagged as rejected/inactive in the index

Internal / Infrastructure

• lncrawl-web is no longer a git submodule; web build artifacts are bundled directly
• Removed deprecated fetch-novel API endpoint (replaced by fetch-novels)
• Python 3.15 excluded from psycopg test matrix (not yet supported upstream)
• server-compose.yml updated

Full diff: v4.5.0...v4.6.0