features engine implementations

Engine implementations

Active contributors: Magnus Hedemark

Purpose

SlopSearX ships with 12 built-in engine adapters. Each adapter lives in its own file under engines/, is registered via the @register_engine decorator, and subclasses either EngineAdapter (for API-based engines) or ScrapeAdapter (for HTML-scrape engines). Adding a new engine requires zero changes to the orchestrator.

Key abstractions

Every adapter follows the same common pattern. The search() method receives a query string and optional params dict, executes the search against the backend, and returns an AdapterResponse. Errors are never raised — they are classified in AdapterResponse.status using the EngineStatus enum (OK, RATE_LIMITED, BLOCKED, ERROR, TIMEOUT). Rate limits are enforced through a shared Valkey-backed rate limiter injected at startup.

Engine table

Engine	File	Type	Categories	Auth	Notable behaviors
Brave	engines/brave.py	api	general, news, science, images	API key (ENGINE_BRAVE_API_KEY)	Rich JSON response with thumbnails; requires X-Subscription-Token header
Wikipedia	engines/wikipedia.py	api	general, science, reference	None	Two-stage pipeline: opensearch for titles, then rich_query for extracts + pageimages
DuckDuckGo	engines/duckduckgo.py	scrape	general, news	None	HTML form POST; _is_challenge_page() detection; CSS selector .result parsing
Google	engines/google.py	scrape	general, news	None	GET request; challenge detection; div.g CSS selector parsing
GitHub	engines/github.py	api	general, reference, github:code, github:issues, github:prs	Token (ENGINE_GITHUB_TOKEN)	Three sub-modes (code, issues/issues, repositories) based on category routing
HuggingFace	engines/huggingface.py	api	general, science, huggingface:datasets, huggingface:papers	Optional token	Three sub-modes (models, datasets, papers) based on category routing
Internet Archive	engines/internetarchive.py	api	reference, web:archive, historical	None	Domain queries route to Wayback CDX API; general queries use advancedsearch
OpenAlex	engines/openalex.py	api	general, science, reference	None	Scholarly works search; inverted index abstract reconstruction via _reconstruct_abstract()
Stack Exchange	engines/stackexchange.py	api	general, reference, science, stackexchange:code, stackexchange:serverfault	Optional API key	Category-based site routing (stackoverflow, serverfault)
arXiv	engines/arxiv.py	api	general, science, reference	None	Atom XML feed parsing via xml.etree.ElementTree; rate-limited to 1 req/3s per arXiv ToS
Hacker News	engines/hackernews.py	api	general, news	None	Algolia API integration; filters to stories only via tags: "story", excludes comments
Semantic Scholar	engines/semanticscholar.py	api	general, science, reference	Optional API key	Paper search with citation data, author metadata, arXiv cross-referencing via externalIds

Class hierarchy

The adapter class hierarchy has two levels:

classDiagram
    class EngineAdapter {
        +name: str
        +display_name: str
        +env_prefix: str
        +engine_type: str
        +categories: list[str]
        +search(query, params)* AdapterResponse
        +health() EngineStatus
        +warmup() None
        +shutdown() None
    }
    class ScrapeAdapter {
        +engine_type: str
        +request_headers: dict
        +timeout_ms: int
        +health() EngineStatus
    }
    class BraveAdapter
    class WikipediaAdapter
    class GitHubAdapter
    class HuggingFaceAdapter
    class InternetArchiveAdapter
    class OpenAlexAdapter
    class StackExchangeAdapter
    class ArxivAdapter
    class HackerNewsAdapter
    class SemanticScholarAdapter
    class DuckDuckGoAdapter
    class GoogleAdapter

    EngineAdapter <|-- ScrapeAdapter
    EngineAdapter <|-- BraveAdapter
    EngineAdapter <|-- WikipediaAdapter
    EngineAdapter <|-- GitHubAdapter
    EngineAdapter <|-- HuggingFaceAdapter
    EngineAdapter <|-- InternetArchiveAdapter
    EngineAdapter <|-- OpenAlexAdapter
    EngineAdapter <|-- StackExchangeAdapter
    EngineAdapter <|-- ArxivAdapter
    EngineAdapter <|-- HackerNewsAdapter
    EngineAdapter <|-- SemanticScholarAdapter
    ScrapeAdapter <|-- DuckDuckGoAdapter
    ScrapeAdapter <|-- GoogleAdapter

Loading

How it works

Common adapter pattern

1. search() flow — read config (base_url, timeout, api_key), build request parameters, send HTTP request via httpx, measure latency, parse response into SearchResult list, return AdapterResponse.
2. Error handling — every adapter catches httpx.TimeoutException, HTTP status errors (429 for rate limiting, 403/503 for blocking), and generic exceptions. All are classified into EngineStatus and returned in AdapterResponse.
3. Rate limiting integration — the rate limiter is injected into each adapter at construction time. Adapters call self.rate_limiter.acquire() before making requests. The rate limiter is Valkey-backed and distributed across all replicas.

Unique patterns by engine

• BraveAdapter — requires an API key set via ENGINE_BRAVE_API_KEY. Uses X-Subscription-Token header for authentication. Parses web.results from the JSON response, extracting thumbnails from the nested thumbnail.src field.
• WikipediaAdapter — two-stage pipeline. Stage 1 calls the opensearch API action to resolve the query to page titles. Stage 2 calls the query API action with prop=extracts|pageimages to fetch rich content (summaries, thumbnails) for each resolved title.
• DuckDuckGoAdapter — a ScrapeAdapter subclass. Sends an HTML form POST to https://html.duckduckgo.com/html/. The _is_challenge_page() method detects CAPTCHA walls by checking for known indicators in the response body. Parses results using CSS selector .result with lxml.
• GoogleAdapter — a ScrapeAdapter subclass. Sends a GET request to https://www.google.com/search with stealth headers. Challenge detection checks for reCAPTCHA and "unusual traffic" indicators. Parses organic results using the div.g CSS selector.
• GitHubAdapter — selects its API endpoint based on category routing: github:code routes to /search/code, github:issues or github:prs routes to /search/issues, and everything else routes to /search/repositories. Requires ENGINE_GITHUB_TOKEN. Returns HTTP 422 for code searches without sufficient qualifiers, handled gracefully as empty results.
• HuggingFaceAdapter — routes to one of three HuggingFace API endpoints based on category: huggingface:datasets routes to /api/datasets, huggingface:papers routes to /api/papers, and the default routes to /api/models. Parses model metadata (pipeline_tag, library_name, downloads, likes) into rich result content.
• InternetArchiveAdapter — detects whether the query looks like a domain name (e.g., example.com) and routes domain queries to the Wayback CDX API for snapshot data. General queries use the advancedsearch.php endpoint for books, audio, video, and other archived media.
• OpenAlexAdapter — searches scholarly works via https://api.openalex.org/works. Reconstructs abstracts from OpenAlex's inverted index format ({"word": [positions], ...}) using the _reconstruct_abstract() helper. Sorts results by cited_by_count:desc.
• StackExchangeAdapter — maps sub-categories to Stack Exchange sites: stackexchange:code routes to stackoverflow, stackexchange:serverfault routes to serverfault, and the default is stackoverflow. Converts Unix timestamps to ISO 8601 for the published_date field.
• arXivAdapter — parses Atom XML feeds using xml.etree.ElementTree with namespace http://www.w3.org/2005/Atom. Rate-limited to 1 request per 3 seconds as required by arXiv's Terms of Service. Strips arXiv IDs from the atom <id> element and constructs paper URLs.
• HackerNewsAdapter — integrates with Algolia's HN search API at https://hn.algolia.com/api/v1/search. Filters to stories only by passing tags: "story", deliberately excluding comments.
• SemanticScholarAdapter — searches papers via the Semantic Scholar graph API. Requests rich fields (title, url, abstract, citationCount, publicationDate, externalIds, authors). Builds content from abstract, author names (truncated to 3 with "et al."), citation count, and arXiv cross-reference.

Key source files

• All files in engines/ directory
• slopsearx/adapter.py — base classes EngineAdapter and ScrapeAdapter
• slopsearx/ratelimit.py — distributed rate limiting shared by all engines

See also

• Output formatters — how engine results are serialized
• Search result types — the SearchResult dataclass
• System architecture — request flow through engines
• Glossary — project-specific terms