Add minAge parameter to scrape (ENG-4073)

[amplitudesxd]

Summary by cubic

Adds a minAge option to v2 scrape to require a minimum cache age and avoid re-scraping. If cached data is younger than minAge, the API returns 404 (SCRAPE_NO_CACHED_DATA) without falling back to other engines. Fulfills ENG-4073.

• New Features

  • minAge added to scrape options (API, JS SDK, Python SDK).
  • Index RPC updated to honor minAge; no engine fallback when minAge is set.
  • New SCRAPE_NO_CACHED_DATA error with 404 handling in the controller.
  • Tests cover cache miss/hit behavior with minAge.

• Migration

  • Pass minAge in scrape options when you require older cached results.
  • Handle 404 with code SCRAPE_NO_CACHED_DATA in clients.
  • Update SDKs: JS v4.8.0, Python v4.10.0.

^{Written for commit 42f19c7. Summary will update automatically on new commits.}

[cubic-dev-ai]

1 issue found across 12 files

Prompt for AI agents (all 1 issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="apps/api/src/__tests__/snips/v2/scrape-cache.test.ts">

<violation number="1" location="apps/api/src/__tests__/snips/v2/scrape-cache.test.ts:145">
Use `scrapeRaw`/`scrapeWithFailure` instead of the success-only `scrape` helper when asserting error responses, otherwise the test throws a Jest assertion before it can inspect the 404 `SCRAPE_NO_CACHED_DATA` response.</violation>
</file>

_{Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR}

[amplitudesxd]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@amplitudesxd I've started the AI code review. It'll take a few minutes to complete.

[cubic-dev-ai]

No issues found across 12 files