refactor: improve session management and concurrency in Google Search

[MODSetter]

Description

Motivation and Context

FIX #

Screenshots

API Changes

• This PR includes API changes

Change Type

• Bug fix
• New feature
• Performance improvement
• Refactoring
• Documentation
• Dependency/Build system
• Breaking change
• Other (specify):

Testing Performed

• Tested locally
• Manual/QA verification

Checklist

• Follows project coding standards and conventions
• Documentation updated as needed
• Dependencies updated as needed
• No lint/build errors or new warnings
• All relevant tests are passing

High-level PR Summary

This PR fixes a critical concurrency bug in the Google Search rendering system where concurrent SERP fetches would fail when one render encountered an error. The fix introduces a semaphore to limit concurrent page renders to 4, tracks in-flight renders per session, and defers browser session cleanup until all in-flight renders complete. This prevents the TargetClosedError cascade that previously occurred when a failing render closed the shared browser while sibling renders were still active. Additionally, the PR includes minor import ordering cleanup in unrelated files.

⏱️ Estimated Review Time: 30-90 minutes

💡 Review Order Suggestion

Order	File Path
1	surfsense_backend/tests/unit/platforms/google_search/test_fetch_concurrency.py
2	surfsense_backend/app/proprietary/platforms/google_search/fetch.py
3	surfsense_backend/alembic/env.py
4	surfsense_backend/scripts/check_migration_flow.py

⚠️ Inconsistent Changes Detected

File Path	Warning
surfsense_backend/alembic/env.py	Import reordering (moving ScriptDirectory import) is unrelated to Google Search session management and concurrency improvements
surfsense_backend/scripts/check_migration_flow.py	Whitespace cleanup in migration script is unrelated to Google Search session management and concurrency improvements

Summary by CodeRabbit

• Bug Fixes

  • Improved reliability of search results fetching when multiple pages are rendered at once.
  • Prevented browser sessions from closing too early during overlapping requests, reducing failed or interrupted fetches.
  • Added safer shutdown handling so active renders can finish before a session is cleaned up.

• Tests

  • Added coverage for concurrent render shutdown behavior and deferred session closing.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
surf-sense-frontend	Building	Preview, Comment	Jul 7, 2026 5:34am

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0c4ae602-8df1-4ece-a4c5-7f0d30c4be54

📥 Commits

Reviewing files that changed from the base of the PR and between 2afc321 and 72cdef0.

📒 Files selected for processing (4)

• surfsense_backend/alembic/env.py
• surfsense_backend/app/proprietary/platforms/google_search/fetch.py
• surfsense_backend/scripts/check_migration_flow.py
• surfsense_backend/tests/unit/platforms/google_search/test_fetch_concurrency.py

---

📝 Walkthrough

Walkthrough

Adds concurrency control to the Google Search browser fetch module: a semaphore-based render gate limits concurrent page renders, and session drop logic defers closure until in-flight renders complete, with new unit tests. Two unrelated files have minor import statement reordering.

Changes

Render Concurrency Fix

Layer / File(s)	Summary
Render gate and deferred session closure surfsense_backend/app/proprietary/platforms/google_search/fetch.py	Adds _MAX_CONCURRENT_PAGES semaphore gate, per-session _inflight counters, and _doomed flag; AsyncStealthySession now constructed with max_pages; _drop_session_on_loop defers close() until in-flight renders finish; _render_on_loop acquires the gate, tracks in-flight counts, and closes doomed sessions after the last render completes.
Concurrency regression tests surfsense_backend/tests/unit/platforms/google_search/test_fetch_concurrency.py	New test module with _FakeSession fixture and two tests verifying deferred close while renders are in-flight and immediate close when idle.

Unrelated Import Reordering

Layer / File(s)	Summary
Import statement reordering surfsense_backend/alembic/env.py, surfsense_backend/scripts/check_migration_flow.py	Reorders import statements with no functional changes.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant RenderOnLoop as _render_on_loop
  participant Gate as Render Semaphore
  participant Session as AsyncStealthySession
  participant DropOnLoop as _drop_session_on_loop

  Caller->>RenderOnLoop: request render
  RenderOnLoop->>Gate: acquire
  RenderOnLoop->>Session: increment _inflight
  RenderOnLoop->>Session: fetch()
  DropOnLoop->>Session: check _inflight
  alt in-flight renders active
    DropOnLoop->>Session: mark _doomed
  else idle
    DropOnLoop->>Session: close()
  end
  RenderOnLoop->>Session: decrement _inflight
  alt session doomed and last in-flight
    RenderOnLoop->>Session: close()
  end
  RenderOnLoop->>Gate: release

Loading

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ci_mvp

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}