Skip to content

v0.7.2

Choose a tag to compare

View all tags
@karust karust released this 27 Apr 01:43
· 55 commits to main since this release
v0.7.2
7a0fb21

OpenSERP 0.7.2

Highlights

  • New API response envelope for search and image endpoints.
  • Unified normalized result model with stable IDs, domain enrichment, classification, metadata, and pagination.
  • New output formats: json, markdown, text, ndjson.
  • Better locale, language, region, and date-filter handling across engines.
  • Major browser runtime improvements: persistent browser connection, isolated contexts, profiles, smart waits, resource blocking.
  • Better proxy health handling and captcha isolation.
  • New OpenAPI spec, architecture docs, contributing guide, CI, and integration tests.

Added

API response model

  • Added JSON response envelopes with:

    • query
    • meta
    • results
    • pagination
    • engines_responded
    • engines_failed

  • Added normalized Result model with:

    • stable result IDs
    • result type
    • position metadata
    • engine metadata
    • domain enrichment

  • Added dedicated image response envelope and image result structure.

  • Added result clustering for /mega/search, grouping same URLs across engines.

  • Added Bing redirect URL unwrapping before result hashing.

Search features

  • Added locale propagation into browser profile and Accept-Language.
  • Added improved query handling with locale support and date filters.
  • Improved URL builders and added edge-case coverage.

Browser/runtime

  • Replaced generic stealth mode with custom browser profile functionality.
  • Added browser profiles and profile coherence tests.
  • Added persistent browser connection reuse.
  • Added isolated browser contexts per request.
  • Added configurable resource/tracker blocking.
  • Added smart waits for SERP loading.
  • Added fingerprint debug tests and endpoint.
  • Added graceful shutdown with drain signal and browser cleanup.

Error handling

  • Added typed 400 validation errors with stable reason codes.
  • Added limit validation range: 1..100.
  • Improved error handling across search engines.
  • Added safe HTTP response draining/closing.

Proxy and captcha handling

  • Isolated proxy health from captcha/parser errors.
  • Replaced proxy re-enable thrashing with quarantine behavior.
  • Improved captcha detection selectors.
  • Added single source of truth for captcha selectors.
  • Captcha solver misconfiguration now exits non-zero.

Docs and release assets

  • Added OpenAPI spec.
  • Added Swagger UI/docs embedding.
  • Added architecture documentation.
  • Added contributing guide.
  • Updated README to match new API format and features.
  • Clarified OpenSERP open-source status and third-party affiliation wording.

CI and tests

  • Added GitHub CI workflow.

  • Added integration workflow.

  • Added integration tests for:

    • Baidu
    • Bing
    • DuckDuckGo
    • Google
    • Yandex

  • Added sanitized real SERP HTML fixtures.

  • Added raw HTML parser tests.

  • Added server endpoint coverage.

  • Added browser, profile, captcha, middleware, retry, proxy, locale, and response tests.

Docker

  • Refactored Docker setup.
  • Switched to chromedp/headless.
  • Updated Dockerfile and docker-compose configuration.

Changed

  • Search internals now distinguish raw engine results from normalized API results.
  • Multi-engine search now returns responded/failed engine metadata.
  • Non-JSON output formats bypass JSON response cache to avoid cache pollution.
  • Logging is now more structured and includes more useful request/search context.
  • Request context now propagates through search, retry, limiter, and browser navigation.
  • README examples were updated for the new response shape.

Fixed

  • Fixed API response format issues.
  • Fixed captcha selector detection problems.
  • Fixed CI issues.
  • Fixed linter errors.
  • Fixed unsafe/unclear engine error handling.
  • Fixed proxy health behavior where captcha/parser failures could incorrectly affect proxy status.
  • Fixed HTTP response cleanup with safe drain-and-close logic.

Breaking / migration notes

  • Search endpoints no longer return bare arrays; they now return response envelopes.
  • Consumers should read results from results instead of treating the whole response as an array.
  • Error responses now include stable machine-readable reason values.
  • limit is now validated and must be between 1 and 100.
  • Clients relying on old response structs should update to the new v1 schema.

Full Changelog: v0.6.0...v0.7.2

Assets 10
Loading