feat/better spam email domain checking

[forgetso]

• Better spam email domain checking
• docs(changeset): Better spam email domain checking
• Add client setting for spam email domain checking. Increase IP service timeout

[Copilot]

Pull request overview

Adds a configurable “spam email domain” validation flow that goes beyond a static denylist by performing DNS/redirect/IP reputation checks, and exposes it via a new provider endpoint with rate limits.

Changes:

• Introduces DNS utilities (runDnsChecks) and a new checkSpamEmail task with DNS + redirect + datacenter-IP checks.
• Adds client setting + DB schema field to gate spam email checking, wires it into captcha verification flows and a new /client/spam/email route.
• Updates rate limiting config and increases IP info lookup timeout.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
packages/util/src/index.ts	Exports new DNS utilities.
packages/util/src/dns.ts	Adds DNS + HTTPS redirect helpers used by spam-domain checking.
packages/types/src/provider/api.ts	Adds CheckSpamEmail path and default rate limit entry.
packages/types/src/client/settings.ts	Adds client setting spamEmailDomainCheckEnabled.
packages/types-database/src/types/client.ts	Persists spamEmailDomainCheckEnabled in DB schema.
packages/provider/src/tasks/spam/checkSpamEmail.ts	Implements enhanced spam email domain checking logic.
packages/provider/src/tasks/captchaManager.ts	Delegates spam checking to new task function.
packages/provider/src/tasks/powCaptcha/powTasks.ts	Gates spam email checking with new flag param.
packages/provider/src/tasks/imgCaptcha/imgCaptchaTasks.ts	Gates spam email checking with new flag param.
packages/provider/src/api/verify.ts	Passes client flag into verification flows.
packages/provider/src/api/captcha/checkSpamEmail.ts	Adds new client API handler for spam email checking.
packages/provider/src/api/captcha.ts	Registers the new spam email route.
packages/provider/src/api/startProviderApi.ts	Excludes spam route from “paths excluding verify” list.
packages/provider/src/services/ipInfo.ts	Increases request timeout and centralizes timeout constant.
packages/cli/src/RateLimiter.ts	Adds env-driven rate limit config for new route.
packages/provider/src/tests/unit/tasks/spam/checkSpamEmail.unit.test.ts	Adds unit tests for spam email parsing/DB behavior.
packages/provider/src/tests/unit/tasks/captchaManager.unit.test.ts	Removes older inlined spam email tests from CaptchaManager suite.
.changeset/fancy-icons-wish.md	Declares patch releases for multiple packages.
.changeset/better-apples-watch.md	Declares (overlapping) patch releases for multiple packages.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Pull request overview

This PR enhances spam email domain checking across the Provider by adding DNS-based heuristics (redirect/CNAME/MX/TLS-error) and improving DB lookups to match subdomains, alongside a new client-controlled toggle and a dedicated API endpoint for spam checks.

Changes:

• Add DNS utilities and a new Provider spam-email check flow (DB + DNS redirect/CNAME/MX + TLS error detection).
• Improve spam-domain DB lookup to match subdomains via generated domain-suffix candidates.
• Add configuration/supporting plumbing: client setting toggle, new client API route + rate limits, configurable DNS servers, and increased IP info timeout.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
packages/util/src/url.ts	Adds domain candidate builder for subdomain→base matching.
packages/util/src/tests/url.unit.test.ts	Adds unit tests for the new domain candidate builder.
packages/util/src/index.ts	Exports new DNS utilities module.
packages/util/src/dns.ts	New DNS + HTTPS redirect checking helpers for spam checks.
packages/types/src/provider/api.ts	Adds CheckSpamEmail path and default rate limit.
packages/types/src/config/config.ts	Adds dnsServers to config schema.
packages/types/src/client/settings.ts	Adds spamEmailDomainCheckEnabled client setting.
packages/types-database/src/types/client.ts	Persists spamEmailDomainCheckEnabled with default false.
packages/provider/src/tests/unit/tasks/spam/checkSpamEmail.unit.test.ts	Adds unit coverage for the new spam email checking task.
packages/provider/src/tests/unit/tasks/captchaManager.unit.test.ts	Removes old inline spam-email tests (moved to dedicated suite).
packages/provider/src/tasks/spam/checkSpamEmail.ts	New task implementing DB + DNS-based spam email checks.
packages/provider/src/tasks/powCaptcha/powTasks.ts	Gates spam-email checking behind the new boolean toggle parameter.
packages/provider/src/tasks/imgCaptcha/imgCaptchaTasks.ts	Gates spam-email checking behind the new boolean toggle parameter.
packages/provider/src/tasks/captchaManager.ts	Delegates spam-email checking to the new task module.
packages/provider/src/services/ipInfo.ts	Increases ipapi timeout and centralizes timeout constant.
packages/provider/src/api/verify.ts	Passes client toggle into captcha verification flows.
packages/provider/src/api/startProviderApi.ts	Adjusts header-check route selection to exclude spam routes.
packages/provider/src/api/captcha/checkSpamEmail.ts	New endpoint to check spam emails for enabled clients.
packages/provider/src/api/captcha.ts	Registers the new CheckSpamEmail client route.
packages/database/tsconfig.json	Adds util project reference.
packages/database/tsconfig.cjs.json	Adds util CJS project reference.
packages/database/src/databases/provider.ts	Updates spam-domain lookup to use suffix candidates.
packages/database/package.json	Adds @prosopo/util dependency for domain candidate helper.
packages/cli/src/prosopo.config.ts	Adds env-based configuration for custom DNS servers.
packages/cli/src/RateLimiter.ts	Adds env-based rate limit config for the new endpoint.
.changeset/fancy-icons-wish.md	Changeset describing spam email domain checking improvements.
.changeset/better-apples-watch.md	Changeset describing spam email domain checking improvements across packages.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Pull request overview

This PR improves spam email domain detection by introducing domain suffix candidate matching, adding an optional client setting + API endpoint to control spam email checks, and hardening outbound domain validation/DNS checks (plus a small IP info timeout bump).

Changes:

• Add utilities for extracting domains, building domain suffix candidates, DNS checks, and SSRF-safe domain validation.
• Add provider spam-email check task + new API route and client setting gate (spamEmailDomainCheckEnabled).
• Update database spam-domain lookup to match on subdomain suffixes (e.g., mail.fakemail.app → fakemail.app) and expand types/config to support custom DNS servers.

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
packages/util/src/url.ts	Adds domain suffix candidate generation for matching parent domains.
packages/util/src/tests/url.unit.test.ts	Adds tests for domain suffix candidate generation.
packages/util/src/domainValidation.ts	Adds SSRF-focused domain validation helpers.
packages/util/src/tests/domainValidation.unit.test.ts	Adds unit tests covering SSRF validation behavior.
packages/util/src/dns.ts	Adds DNS + HTTPS redirect checks with configurable resolver.
packages/util/src/email.ts	Adds email/domain parsing helper for spam checks.
packages/util/src/index.ts	Re-exports new util modules.
packages/database/src/databases/provider.ts	Updates spam domain DB lookup to use suffix candidates.
packages/database/package.json / tsconfig*	Adds @prosopo/util dependency and TS project refs.
packages/provider/src/tasks/spam/checkSpamEmail.ts	Introduces consolidated spam email checking flow (DB + DNS + SSRF guard).
packages/provider/src/api/captcha/checkSpamEmail.ts	Adds new API endpoint to check spam emails with client setting gating.
packages/provider/src/api/captcha.ts	Wires new endpoint into router.
packages/provider/src/tasks/Captcha.ts	Gates spam email checking behind a new boolean parameter.
packages/provider/src/tasks/captchaManager.ts	Moves spam check logic into shared task function.
packages/provider/src/services/ipInfo.ts	Increases timeout and centralizes the value.
packages/types/src/*	Adds new route path, rate limit, config option, and client setting.
packages/types-database/src/types/client.ts	Persists the new client setting with default false.
packages/cli/src/*	Adds env-driven DNS servers config and rate limit env vars for new route.
dev/config/src/webpack/webpack.config.ts	Refactors externals to auto-exclude Node builtins.
.changeset/*	Adds changeset entries for the release.

Comments suppressed due to low confidence (2)

packages/util/src/url.ts:1

• The function name says PrefixCandidates, but the JSDoc and behavior generate suffix candidates by stripping the leftmost label. This mismatch makes the API easy to misuse (and already conflicts with the doc comment). Consider renaming to something like buildDomainSuffixCandidates (or buildDomainSuffixCandidatesExcludingTld).
  packages/util/src/url.ts:1
• buildDomainPrefixCandidates() currently emits candidates that can be syntactically invalid domains (e.g., leading dot .example.com, trailing dot example.com., consecutive dots test..domain), and the tests explicitly encode those invalid outputs. Since this helper is used to form DB query conditions, keeping invalid candidates increases query noise and can create confusing matches.\nConsider normalizing/validating input first (trim, remove trailing dot from FQDN form, and reject/skip candidates with empty labels) so the candidate list contains only valid domain strings used consistently across the system.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 9 comments.

Comments suppressed due to low confidence (1)

packages/provider/src/api/startProviderApi.ts:157

• clientPathsExcludingVerify now holds the result of getClientApiPathsExpectingProsopoHeaders(), which is broader than just "excluding verify". Renaming this local variable would reduce confusion and make future middleware changes less error-prone.

	const clientPathsExcludingVerify = getClientApiPathsExpectingProsopoHeaders();

	env.logger.debug(() => ({
		msg: "Adding headerCheckMiddleware",
		paths: clientPathsExcludingVerify,

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 10 comments.

Comments suppressed due to low confidence (1)

packages/provider/src/api/startProviderApi.ts:158

• The variable name clientPathsExcludingVerify no longer matches the helper (getClientApiPathsExpectingProsopoHeaders) and the updated behavior (it excludes both verify and spam routes). Renaming the variable to reflect what it contains will reduce confusion when maintaining middleware ordering.

	const clientPathsExcludingVerify = getClientApiPathsExpectingProsopoHeaders();

	env.logger.debug(() => ({
		msg: "Adding headerCheckMiddleware",
		paths: clientPathsExcludingVerify,
	}));

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

done