feat: resolve redirects before mapping urls

[amplitudesxd]

Summary by cubic

Resolve and follow URL redirects before running map so sites like firecrawl.com (redirects to firecrawl.dev) are mapped correctly. Fixes ENG-4022 where map failed on redirected domains.

• Bug Fixes
  • Added resolveRedirects that follows redirects (HEAD first, GET fallback) with a secure dispatcher.
  • Map uses resolveRedirects; test verifies links resolve to the final domain.

^{Written for commit b825250. Summary will update automatically on new commits.}

[cubic-dev-ai]

2 issues found across 3 files

Prompt for AI agents (all 2 issues)

Understand the root cause of the following 2 issues and fix them.


<file name="apps/api/src/__tests__/snips/v2/map.test.ts">

<violation number="1" location="apps/api/src/__tests__/snips/v2/map.test.ts:128">
Wrap this redirect test in concurrentIf(ALLOW_TEST_SUITE_WEBSITE) (or use a mock) so it doesn’t make real network calls when external access is disabled.</violation>
</file>

<file name="apps/api/src/lib/validateUrl.ts">

<violation number="1" location="apps/api/src/lib/validateUrl.ts:213">
Propagate AbortError instead of swallowing it so cancellation via the provided AbortSignal actually rejects the promise.</violation>
</file>

_{Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR}

[amplitudesxd]

@cubic-dev-ai re-review

[cubic-dev-ai]

@cubic-dev-ai re-review

@amplitudesxd I've started the AI code review. It'll take a few minutes to complete.

[cubic-dev-ai]

1 issue found across 3 files

Prompt for AI agents (all 1 issues)

Understand the root cause of the following 1 issues and fix them.


<file name="apps/api/src/lib/validateUrl.ts">

<violation number="1" location="apps/api/src/lib/validateUrl.ts:213">
Abort signal is swallowed in resolveRedirects, so canceled map jobs keep running instead of stopping when the abort signal fires.</violation>
</file>

_{Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR}

[amplitudesxd]

@cubic-dev-ai re-review

[cubic-dev-ai]

@cubic-dev-ai re-review

@amplitudesxd I've started the AI code review. It'll take a few minutes to complete.

[cubic-dev-ai]

1 issue found across 3 files

Prompt for AI agents (all 1 issues)

Understand the root cause of the following 1 issues and fix them.


<file name="apps/api/src/lib/validateUrl.ts">

<violation number="1" location="apps/api/src/lib/validateUrl.ts:212">
HEAD responses are returned without checking for 405/501, so servers that disallow HEAD never hit the GET fallback and their redirects still are not resolved.</violation>
</file>

_{Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR}

[mogery]

LGTM