Self-Host Overhaul - v1.5.0

[nickscamara]

Self-Host Fixes

• Reworked Guide: The SELF_HOST.md and docker-compose.yaml have been updated for clarity and compatibility
• Kubernetes Imporvements: Updated self-hosted Kubernetes deployment examples for compatibility and consistency (Update self-hosted Kubernetes deployments examples for compatibility and consistency #1177)
• Self-Host Fixes: Numerous fixes aimed at improving self-host performance and stability (SELFHOST FIXES (FIR-1105) #1207)
• Proxy Support: Added proxy support tailored for self-hosted environments (feat(self-host): proxy support (FIR-1111) #1212)
• Playwright Integration: Added fixes and continuous integration for the Playwright microservice (feat(ci/self-host): add playwright microservice tests #1210)
• Search Endpoint Upgrade: Added SearXNG support for the /search endpoint (Add searxng for search endpoint #1193)

Core Fixes & Enhancements

• Crawl Status Fixes: Fixed various race conditions in the crawl status endpoint (fix(crawl-status): consider concurrency limited jobs as prioritized (FIR-851) #1184)
• Timeout Enforcement: Added timeout for scrapeURL engines to prevent hanging requests (fix(scrapeURL/sb): enforce timeout (FIR-980) #1183)
• Query Parameter Retention: Map function now preserves query parameters in results (fix(map): do not remove query parameters from results (FIR-1015) #1191)
• Screenshot Action Order: Ensured screenshots execute after specified actions (fix(scrapeURL/fire-engine): perform format screenshot after specified actions (FIR-985) #1192)
• PDF Scraping: Improved handling for PDFs behind anti-bot measures (feat(scrapeURL): handle PDFs behind anti-bot (FIR-722) #1198)
• Map/scrapeURL Abort Control: Integrated AbortController to stop scraping when the request times out (feat(v1/map): stop mapping if timed out via AbortController (FIR-747) #1205)
• SDK Timeout Enforcement: Enforced request timeouts in the SDK (feat(sdk): enforce timeout on client-side if set (FIR-864) #1204)

New Features & Additions

• Proxy & Stealth Options: Introduced a proxy option and stealthProxy flag (feat(v1): proxy option / stealthProxy flag (FIR-1050) #1196)
• Deep Research (Alpha): Launched an alpha implementation of deep research ((feat/deep-research) Alpha implementation of deep research #1202)
• LLM Text Generator: Added a new endpoint for llms.txt generation (Add llmstxt generator endpoint #1201)

Docker & Containerization

• Production Ready Docker Image: A streamlined, production ready Docker image is now available to simplify self-hosted deployments.

For the complete details, check out the full changelog.

What's Changed

• fix(crawl-status): consider concurrency limited jobs as prioritized (FIR-851) by @mogery in fix(crawl-status): consider concurrency limited jobs as prioritized (FIR-851) #1184
• fix(scrapeURL/sb): enforce timeout (FIR-980) by @mogery in fix(scrapeURL/sb): enforce timeout (FIR-980) #1183
• fix(map): do not remove query parameters from results (FIR-1015) by @mogery in fix(map): do not remove query parameters from results (FIR-1015) #1191
• fix(scrapeURL/fire-engine): perform format screenshot after specified actions (FIR-985) by @mogery in fix(scrapeURL/fire-engine): perform format screenshot after specified actions (FIR-985) #1192
• Update self-hosted Kubernetes deployments examples for compatibility and consistency by @tetuyoko in Update self-hosted Kubernetes deployments examples for compatibility and consistency #1177
• fix(v1/types): fix extract -> json rename (FIR-1072) by @mogery in fix(v1/types): fix extract -> json rename (FIR-1072) #1195
• feat(v1): proxy option / stealthProxy flag (FIR-1050) by @mogery in feat(v1): proxy option / stealthProxy flag (FIR-1050) #1196
• fix(v1/types): fix extract -> json rename, ROUND II (FIR-1072) by @mogery in fix(v1/types): fix extract -> json rename, ROUND II (FIR-1072) #1199
• (feat/deep-research) Alpha implementation of deep research by @nickscamara in (feat/deep-research) Alpha implementation of deep research #1202
• Add llmstxt generator endpoint by @ericciarla in Add llmstxt generator endpoint #1201
• fix(concurrency-limit): move to renewing a lock on each active job instead of estimating time to complete (FIR-1075) by @mogery in fix(concurrency-limit): move to renewing a lock on each active job instead of estimating time to complete (FIR-1075) #1197
• SELFHOST FIXES (FIR-1105) by @mogery in SELFHOST FIXES (FIR-1105) #1207
• feat(v1/map): stop mapping if timed out via AbortController (FIR-747) by @mogery in feat(v1/map): stop mapping if timed out via AbortController (FIR-747) #1205
• Playwright page error schema by @makeiteasierapps in Playwright page error schema #1172
• feat(ci/self-host): add playwright microservice tests by @mogery in feat(ci/self-host): add playwright microservice tests #1210
• feat(scrapeURL): handle PDFs behind anti-bot (FIR-722) by @mogery in feat(scrapeURL): handle PDFs behind anti-bot (FIR-722) #1198
• Use correct list typing for py 3.8 support by @niazarak in Use correct list typing for py 3.8 support #931
• feat(map): mock support (FIR-1109) by @mogery in feat(map): mock support (FIR-1109) #1213
• Add searxng for search endpoint by @loorisr in Add searxng for search endpoint #1193
• feat(sdk): enforce timeout on client-side if set (FIR-864) by @mogery in feat(sdk): enforce timeout on client-side if set (FIR-864) #1204
• feat(self-host): proxy support (FIR-1111) by @mogery in feat(self-host): proxy support (FIR-1111) #1212
• temp by @mogery in temp #1218
• gemini extractor Implementation by @aparupganguly in gemini extractor Implementation #1206

New Contributors

• @tetuyoko made their first contribution in Update self-hosted Kubernetes deployments examples for compatibility and consistency #1177
• @makeiteasierapps made their first contribution in Playwright page error schema #1172
• @niazarak made their first contribution in Use correct list typing for py 3.8 support #931
• @loorisr made their first contribution in Add searxng for search endpoint #1193

Full Changelog: v1.4.4...v1.5.0

---

This discussion was created from the release Self-Host Overhaul - v1.5.0.