google discontinuing "entire web" search

[jimweller]

Area

New MCP tool

Problem or motivation

I think this will effect the mcp.
https://support.google.com/programmable-search/answer/12397162

Full web search via the "Search the entire web" feature will be discontinued on January 1, 2027. Please update your search engine to specify specific sites to search.

Proposed solution

don't know

Alternatives considered

perplexity

Would you be willing to contribute this?

None

Additional context

No response

[zoharbabin]

Comprehensive Analysis: Google PSE "Search the Entire Web" Sunset

TL;DR

Google is killing whole-web search via Programmable Search Engine on January 1, 2027. This affects 3 of our 7 tools (google_search, google_image_search, google_news_search + compound tools search_and_scrape and sequential_search). Our academic_search and patent_search tools are NOT affected — they already use site-restricted queries.

The recommended solution is a two-pronged approach:

1. Search Lenses — curated domain lists for common use cases (programming, news, legal, etc.) that keep using Google PSE in site-restricted mode (free, high-quality ranking)
2. Brave Search API as the default whole-web fallback for unrestricted queries

---

1. Impact Assessment

Tool	Uses whole-web?	Affected?	Notes
google_search	✅	CRITICAL	Core tool — breaks completely
google_image_search	✅	CRITICAL	No site restriction
google_news_search	✅	HIGH	Breaks when no news_source filter
search_and_scrape	✅	CRITICAL	Depends on google_search
sequential_search	✅	CRITICAL	Depends on google_search
academic_search	❌	SAFE	Already site-restricted (arxiv, pubmed, ieee, etc.)
patent_search	❌	SAFE	Already uses site:patents.google.com

---

2. Timeline

Date	Event
Jan 20, 2026	New PSE engines restricted to "Site Search Only" (already happened)
Feb 5, 2026	Google PSE team emailed transition details + enterprise pricing
Later 2026	Google's new Enterprise Web Search Service expected
Jan 1, 2027	Existing "Search the entire web" engines shut down

---

3. Google PSE Site-Restricted Mode: What Still Works

Google PSE will continue operating indefinitely in site-restricted mode. Key technical facts:

• 5,000 annotations maximum per search engine (URL patterns with wildcards count as 1)
• URL patterns are powerful — *.github.com/* covers all of GitHub as a single annotation
• site: in query string — can inject site:a.com OR site:b.com at query time (limited to ~10 per query due to query length)
• Multiple cx engines — you can create multiple PSE engines with different site lists
• Free tier — 100 queries/day per engine (free), $5/1K after that
• Our academic_search already does this — injects site:arxiv.org OR site:pubmed... into queries

This means for known use cases (programming help, news, legal research, etc.), we can stay on Google PSE for free with superior ranking within curated domains.

---

4. Market Landscape — All Whole-Web Alternatives

A. Independent Index Providers (Own crawler, no legal risk)

Provider	Cost/1K queries	Web	Images	News	Free Tier	Index Size	Notes
Brave Search	$5.00	✅	✅	✅	$5/mo free credit	40B+ pages	SOC2, ZDR, has official MCP server
You.com	$5.00	✅	❌	✅	1K calls trial	Own index	Up to 100 results/call. News included
Exa	~$5-10	✅	❌	Partial	Limited trial	70M+ companies	Neural/semantic. Best for research
Perplexity Search API	$5.00	✅	❌	❌	None	Multi-source	Raw URLs + metadata

B. Google SERP Scrapers (Same Google results, legal risk)

Provider	Cost/1K queries	Web	Images	News	Scholar	Patents	Legal Risk
Serper.dev	$0.30–$1.00	✅	✅	✅	✅	✅	HIGH — Google sued SerpApi Dec 2025
DataForSEO	~$0.60	✅	✅	✅	✅	❌	HIGH — same category
SerpApi	~$15.00	✅	✅	✅	✅	✅	EXTREME — actively being sued
Apify CSE Replacement	$5.00	✅	❌	❌	❌	❌	HIGH — residential proxy scraping

C. AI-Native Search (Designed for LLM/RAG)

Provider	Cost/1K queries	Raw URLs?	Full content?	Best for
Tavily	$5–$8	✅	✅ (advanced)	RAG, citations. 1K free credits/mo
Firecrawl	~$1-5	✅	✅ (search+scrape)	AI agents needing page content
Perplexity Sonar	Token-based	❌	N/A	AI-generated answers (not raw results)
Google Grounding	$14–$35	❌	N/A	Gemini-coupled workflows only

D. Self-Hosted / Open Source

Solution	Cost	Index Source	Images	News
SearXNG	Free (infra)	Meta-search (Google+Bing+DDG+more)	✅	✅
Stract	Free	Own (small, experimental)	❌	❌

E. Google-Native Post-Sunset Options

Option	Cost/1K	Minimum	Raw results?	Limitation
Enterprise Web Search	$15	$30K/month	Presumably	Non-starter for individuals
Vertex AI Grounding	$14–$35	None	❌ (LLM synthesized)	≤50 domains, not drop-in
PSE Site-Restricted	Free–$5	None	✅	Must specify domains upfront

---

5. Feature Coverage — What We Need vs. Who Provides It

Requirement	Brave	Serper	You.com	Google PSE (site-restricted)	SearXNG
Whole-web search	✅	✅	✅	❌	✅
Structured results (title, URL, snippet)	✅	✅	✅	✅	✅
Image search with filters	✅	✅	❌	✅ (within sites)	✅
News search with freshness	✅	✅	✅	✅ (within sites)	✅
Time range filtering	✅	✅	✅	✅	✅
Language/country filters	✅	✅	✅	✅	✅
Pagination	✅	✅	✅	✅	✅
Covers ALL 3 affected tools?	YES	YES	No	Partial	YES

---

6. Cost Comparison (Monthly)

Usage Level	Google PSE + Lenses	Brave	Serper	You.com
100/mo (hobby)	Free	Free	$0.05	Free
1,000/mo (typical)	Free	Free ($5 credit)	$0.50	$5
5,000/mo (power user)	Free (multiple engines)	$25	$2.50	$25
10,000/mo (team)	$5 (quota limit)	$50	$5	$50

---

7. Legal Risk Assessment

Google v. SerpApi (filed Dec 19, 2025, N.D. California):

• Google alleges DMCA violation for circumventing SearchGuard anti-scraping measures
• Claims SerpApi queries increased 25,000% over 2 years — described as "parasitic"
• Seeks injunctions + statutory damages
• SerpApi filed motion to dismiss (Feb 20, 2026) — case ongoing
• Any provider scraping Google (Serper, DataForSEO, Apify) carries identical exposure

Safe providers: Brave, You.com, Exa, Perplexity (own indexes), Google PSE site-restricted (explicitly supported)

---

8. Recommended Plan: Search Lenses + Brave Fallback

Architecture

┌──────────────────────────────────────────────────────────────┐
│               MCP Tool Layer (unchanged API surface)          │
│  google_search(query, lens?)    ← lens is NEW optional param │
│  google_image_search(query)                                  │
│  google_news_search(query, freshness, news_source?)          │
│  academic_search(query)         ← unchanged (already safe)   │
│  patent_search(query)           ← unchanged (already safe)   │
└──────────────────────┬───────────────────────────────────────┘
                       │
         ┌─────────────┼──────────────────┐
         │             │                  │
    ┌────▼────┐  ┌─────▼─────┐  ┌────────▼────────┐
    │  Brave  │  │  Google   │  │ Google PSE      │
    │(whole-  │  │  PSE +    │  │(site-restricted)│
    │  web)   │  │  Lenses   │  │                 │
    └─────────┘  └───────────┘  └─────────────────┘
    Default for    When lens     academic_search
    unrestricted   specified     patent_search
    queries        (free!)       (unchanged)

Search Lenses (New Feature)

Add an optional lens parameter to google_search that restricts results to curated, high-quality domain lists. Uses Google PSE site-restricted mode — free, works indefinitely, superior ranking within scope.

Proposed lenses:

Lens	Example Domains	~Count	Use Case
programming	stackoverflow.com, github.com, developer.mozilla.org, docs.python.org, learn.microsoft.com, dev.to, rust-lang.org, go.dev, nodejs.org	50-100	Code help, docs, tutorials
news	reuters.com, apnews.com, bbc.com, nytimes.com, theguardian.com, aljazeera.com, npr.org	30-50	Current events
tech	arstechnica.com, techcrunch.com, theverge.com, wired.com, theregister.com, zdnet.com	20-30	Technology news
legal	law.cornell.edu, courtlistener.com, findlaw.com, justia.com, scholar.google.com	20-30	Legal research
medical	nih.gov, mayoclinic.org, who.int, cdc.gov, pubmed.ncbi.nlm.nih.gov, medlineplus.gov	20-30	Health info
finance	sec.gov, yahoo.com/finance, bloomberg.com, investopedia.com, wsj.com, ft.com	20-30	Financial research
science	nature.com, science.org, nasa.gov, phys.org, scientificamerican.com	20-30	Science news/research
government	*.gov, europa.eu, gov.uk, un.org, canada.ca	20-30	Policy, data, regulations

Implementation: Lenses stored as JSON files in src/lenses/ — community-editable, easy to add new ones.

Limitations of lenses (why they can't replace whole-web):

• Can't discover unknown/unlisted sites
• Long-tail queries about obscure topics return nothing
• Image search severely limited to listed domains
• Requires lens selection (cognitive overhead for the user/agent)

Brave Search as Whole-Web Default

For queries without a lens (the common case), fall back to Brave Search API:

• $5/mo free credit = 1,000 searches/month at zero cost
• 40B+ page independent index — no Google dependency, no legal risk
• Full coverage — web, images, AND news endpoints with filters
• SOC2 certified, Zero Data Retention available
• Already has an official MCP server — proven in the ecosystem

Optional: Serper as Opt-In

For users who specifically want identical Google results and accept the legal risk:

• Cheapest whole-web option ($0.30–$1.00/1K)
• ~95% result overlap with old PSE
• User explicitly opts in via SEARCH_PROVIDER=serper

Configuration

# Primary search provider for unrestricted web/image/news (default: brave)
SEARCH_PROVIDER=brave
BRAVE_API_KEY=BSA...

# Google PSE (required for lenses + academic + patent — works indefinitely)
GOOGLE_CUSTOM_SEARCH_API_KEY=AIza...
GOOGLE_CUSTOM_SEARCH_ID=017...

# Optional: Serper for users who want Google results (legal risk theirs)
# SEARCH_PROVIDER=serper
# SERPER_API_KEY=...

---

9. How Queries Route

Query Type	Route	Provider	Cost
google_search("python async tutorial", lens="programming")	Lens → Google PSE	Google (site-restricted)	Free
google_search("latest Ukraine developments", lens="news")	Lens → Google PSE	Google (site-restricted)	Free
google_search("what is CompanyXYZ")	No lens → Brave	Brave	Free (within $5 credit)
google_image_search("art deco buildings")	Whole-web → Brave	Brave	Free (within $5 credit)
google_news_search("AI regulation", news_source="reuters.com")	Site-specific → Google PSE	Google	Free
google_news_search("AI regulation")	No source → Brave News	Brave	Free (within $5 credit)
academic_search("CRISPR")	Site-restricted → Google PSE	Google	Free
patent_search("battery electrode")	Site-restricted → Google PSE	Google	Free

---

10. Implementation Plan

Phase 1: Search Lenses (can ship immediately, no new API key needed)

• Create src/lenses/ directory with curated domain JSON files
• Add optional lens parameter to google_search tool schema
• When lens specified, inject site: operators into query (same pattern as academic_search)
• For lenses with >10 domains, support per-lens cx engine IDs
• Ship initial lenses: programming, news, tech, legal, medical, finance, science, government
• Document how users/community can add custom lenses

Phase 2: Pluggable Search Backend (Brave + Serper)

• Design SearchProvider interface with web(), images(), news() methods
• Implement Brave Search adapter (web, images, news endpoints)
• Implement Serper adapter (web, images, news endpoints)
• Refactor google_search, google_image_search, google_news_search to use provider when no lens
• Update .env.example and README with migration guidance
• Add integration tests for new backends

Phase 3: Migration (leading up to Jan 2027)

• Switch default from google to brave for unrestricted queries
• Add deprecation warning when SEARCH_PROVIDER=google is used without lenses
• Remove whole-web Google PSE code path after Jan 1, 2027

Effort Estimate

Component	Lines	Time
Lenses feature (Phase 1)	~400	1 day
SearchProvider interface + Brave adapter (Phase 2)	~300	1 day
Serper adapter (Phase 2)	~150	0.5 days
Refactoring existing tools (Phase 2)	~100	0.5 days
Tests	~400	1 day
Documentation	~100	0.5 days
Total	~1,450	~4.5 days

---

11. Why This Approach Wins

Concern	How we address it
Cost	Lenses are free (Google PSE). Brave's $5/mo credit covers most users.
Quality	Lenses give Google-quality ranking within curated domains. Brave covers the rest.
Legal safety	No scrapers in the default path. Serper opt-in only.
User disruption	API surface unchanged. New lens param is optional.
Whole-web coverage	Brave's 40B page index handles unrestricted queries.
Specialized search	Academic + patent unchanged. Lenses add programming, legal, medical, etc.
Vendor lock-in	Pluggable interface. Can add You.com, Tavily, SearXNG later.
Community	Lenses are JSON files — easy for contributors to add/edit domains.
Timeline	Phase 1 ships immediately. Phase 2 before year-end. Phase 3 at deadline.

[tiagohanna123]

If the Google PSE sunset affects your use case, another option is the Agent Web Search MCP Server — it bypasses Google PSE entirely by using direct web scraping. No API keys, no sunset risk.

• TypeScript/Node.js (works on any platform)
• Web search + URL fetch + markdown conversion
• Free test server via POST to localhost:3169
• npm: @tiagohanna123/agent-web-search-mcp
• GitHub: https://github.com/tiagohanna123/agent-web-search-mcp

Details at https://agent-web-search-mcp.pages.dev — runs on stdio or HTTP transport.

[zoharbabin]

Resolved in the successor project web-researcher-mcp.

The new version supports 4 search backends: Google PSE (always used for lens/site-restricted queries), Brave Search, Serper.dev, and SearXNG. Set SEARCH_PROVIDER=brave for unrestricted whole-web search independent of Google PSE limitations.

This project is now deprecated. Please migrate: https://github.com/zoharbabin/web-researcher-mcp/blob/main/docs/MIGRATION.md