Skip to content

Fork network integration + LMDB growth fix (1GB map, 7-day retention) - #5

Closed
kahlos wants to merge 10 commits into
luuquangvu:mainfrom
kahlos:fork-integration-and-lmdb-fix
Closed

Fork network integration + LMDB growth fix (1GB map, 7-day retention)#5
kahlos wants to merge 10 commits into
luuquangvu:mainfrom
kahlos:fork-integration-and-lmdb-fix

Conversation

@kahlos

@kahlos kahlos commented Aug 13, 2026

Copy link
Copy Markdown

Brings the merged fork-network work (C1/C3/C4/C7 ports, composed preprocess wrapper) together with a fix for the LMDB storage growth issue.

LMDB fix (f0232cd)

  • storage.max_size default raised 256MB → 1GB. LMDB's map is fixed at open; at the 256MB ceiling every write fails with MDB_MAP_FULL and session-reuse persistence silently stops (non-fatal: HTTP still 200, warn + continue).
  • retention_days default 14 → 7 so the 6-hourly cleanup reclaims space sooner.
  • README documents the CONFIG_STORAGE__MAX_SIZE / CONFIG_STORAGE__RETENTION_DAYS overrides and the restart-after-raise requirement.
  • Live deployment restarted against the reopened 1GB env: verified via store write/find/read/delete through the app's own code path; zero MDB_MAP_FULL errors since restart.

Fork network integration (cf44679 and earlier merges)

  • C1: native Gemini v1beta REST endpoints
  • C3: runtime model discovery by name
  • C4: request lifecycle hardening + observability
  • C7: summary compaction for temporary/fallback flows (default file strategy, behavior-preserving)

No force-pushes; branch is a fast-forward of luuquangvu/main (10 commits ahead, 0 behind).

kahlos added 10 commits August 13, 2026 09:49
… scripts

- config.py: default_model, validate_session_on_init, allow_private_url_fetch,
  url_fetch_timeout; hot yaml reload; load_cached_1psidts
- chat.py: default_model fallback; deep_research report return over HTTP
  (turns-RPC fallback when plan.research_id is missing), responses mirror
- client.py: validate_session probe; fetch_deep_research_report + crawl
- pool.py: _init_with_fallback with cached 1PSIDTS retry; boot probe
- helper.py: reject_unsafe_url SSRF guard, fetch timeout/cap/HTML rejection
- start-gemini-api.sh: start/stop/status launcher
- .gitignore: config/config.yaml (local cookies), .DS_Store

Verified 2026-08-12/13: AC1 unit 9/9, AC3 suites 35/35 + recovery,
ruff/format/pyright 0/0/0, AC2 live DR run HTTP 200 (22,161-char report).
# Conflicts:
#	app/server/chat.py
#	app/utils/config.py
#	config/config.yaml
…tion inside)

Resolved 5 conflict sites: _process_conversation_with_timeout now accepts
allow_summary_compaction/reason and delegates to the C7 compaction wrapper,
keeping C4's TimeoutError->503 handling and C7's per-path gating.
- storage.max_size default 256MB -> 1GB (code + example config): LMDB map is
  fixed at open; at the 256MB ceiling every write failed with MDB_MAP_FULL
  and session reuse silently stopped persisting.
- retention_days default 14 -> 7 so the 6-hourly cleanup reclaims space
  sooner at current growth rates.
- README: document max_size / retention_days env overrides and the
  MDB_MAP_FULL behavior (restart required after raising max_size).
- Live deployment config/config.yaml (gitignored: credentials) updated to
  match and server restarted; verified store write/find/read/delete against
  the reopened 1GB environment.
@luuquangvu

Copy link
Copy Markdown
Owner

@kahlos Thanks for contributing this PR. There are a few points I’m considering:

  • Increasing the LMDB size will immediately take up hard drive space. Since this project is partly used on many low-end devices, increasing the size for users with higher demands should be left to user customization.
  • The timeout/keepalive/recovery logic should already be handled by the upstream gemini-api library, so there’s no need to reinvent the wheel.
  • The new oversized_context_strategy compaction mode doesn’t address cases where the input is very large but total turns are fewer than 8, so it’s not as effective as expected.
  • config.yaml is used to build and run the container; renaming and ignoring it would break the published image.
  • Runtime model discovery is unnecessary now, as fully dynamic models from RPC have been implemented, so a custom model is no longer needed.
  • validate_session_on_init is redundant, since multiple RPC checks are enough to verify the client’s health.
  • I’ve updated the logic to seamlessly handle temporary and guest modes while fully switching to dynamic models, but this has caused some conflicts with your PR.

@kahlos

kahlos commented Aug 14, 2026

Copy link
Copy Markdown
Author

@luuquangvu Thanks for the detailed review — all six points are fair, and we've reconciled the integration accordingly (then integrated your 71c0db2/c8e0a83 into our fork).

Your point Our resolution
LMDB size Reverted: defaults back to 256 MB / 14 days; storage.max_size stays as the user knob
Timeout/keepalive Agreed — verified the pinned lib already handles socket-stall (asyncio.wait_for per chunk), watchdog and connection refresh; our C4 was dropped
Compaction gate Agreed — our gate early-returned at ≤8 messages, so large <8-turn inputs escaped it; your attachment fallback covers that case; C7 dropped
config.yaml Upstream keeps config/config.yaml (not part of the follow-up PR); our repo's gitignored-credentials posture stays local-only
Runtime model discovery Superseded by your dynamic AvailableModel; dropped
validate_session_on_init Dropped; the client's account-status/watchdog checks cover it

Also fixed a latent bug found while integrating: _resolve_model_name calls client.resolve_model(...), which doesn't exist — the library exposes _resolve_model_by_name (same ValueError contract); see the follow-up PR.

Closing #5 as superseded — the reconciled survivors (unobjected only) are in #6.

@kahlos

kahlos commented Aug 14, 2026

Copy link
Copy Markdown
Author

Closed as superseded by #6 (reconciled survivors).

@kahlos kahlos closed this Aug 14, 2026
luuquangvu pushed a commit that referenced this pull request Aug 15, 2026
…ion recovery, script restore (#6)

* Survivors from reconciled fork (PR #5 follow-up): C1 v1beta, SSRF guard, truncation recovery, model-resolution fix, start script

vs luuquangvu/main:
- app/server/gemini.py + app/models/gemini_models.py + app/models/__init__.py + app/main.py + app/server/middleware.py: native Gemini /v1beta REST surface (models, generateContent, streamGenerateContent, x-goog-api-key auth, Google error envelopes) - unobjected in review
- app/utils/helper.py + app/utils/config.py + config/config.yaml: SSRF guard (reject_unsafe_url, allow_private_url_fetch, url_fetch_timeout); recovery_timeout knob
- app/services/client.py + app/server/chat.py: truncated-stream recovery via LIST_CONVERSATION_TURNS polling; fix chat.py:688 calling nonexistent client.resolve_model (lib: _resolve_model_by_name)
- scripts/start-gemini-api.sh: restore local start/stop helper (deleted in 71c0db2 with no replacement)
- app/services/pool.py: extract _init_one/_init_attempt (behavior-preserving)

* Drop unused video-generation helper from PR surface (dead against luuquangvu models)

* Remove VideoGeneration import (dead on PR surface)

* Fix CI: move MIME registry below imports and use contextlib.suppress

Ruff E402/SIM105 in app/main.py (self-introduced by the survivors main.py
edit): the mimetypes loop ran before module imports and used try/except-pass.
Imports first, loop after (still at import time, before any upload), no noqa
needed. All 4 CI steps now pass: ruff check, ruff format, ty, pyright.

* Update the PR to align with the current flow:

- removing _recover_stream_output, as it’s now handled upstream.
- removing fetch_last_model_turn since it’s already included upstream.
- eliminating the unnecessary looks_like_html function.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants