ScholarScout v1.6.5

ScholarScout v1.6.5

ScholarScout now moves from a paper-reactive generator toward a more agent-like workflow with gap-first planning, steering controls, stronger run transparency, and a built-in staged update flow for non-Git users.

Highlights

• Gap-first generation: ideas are now synthesized from reusable gap_candidates, not just raw paper lists
• Gap steering modes: choose between Balanced, Breakthrough-heavy, and Practical-first
• Better diversity: ideas are less likely to collapse onto the same small paper cluster
• Gap Monitor: Activity Center now shows synthesized gaps per category while the pipeline runs
• Evidence-aware gap layer: gap coverage and supporting papers now surface across cards, modal detail, copy/export, and run summary
• Richer run summaries: gap steering, total gap candidates, contributed papers, and average supporting depth are now tracked
• Legacy session compatibility: older snapshots and session history without gap-first metadata now restore safely
• Version sync: runtime version sources are aligned so the dashboard badge, VERSION, and packaging metadata stop drifting
• ZIP-user update flow: dashboard now offers Update now to stage a fresh build, preserve config.yaml and data/, and launch the updated app without Git

New update experience

If a newer release is available, ScholarScout can now:

1. Download the latest GitHub release ZIP
2. Extract it into a new versioned folder
3. Copy over config.yaml and data/
4. Install dependencies
5. Launch the updated app automatically

This is especially helpful for Windows and ZIP-based users who are not using git pull.

Included in this release

• New gap_synthesis pipeline stage
• Steering-aware generation prompts and UI persistence
• Backward-compatible session/snapshot normalization
• Dashboard update overlay and staged updater backend
• Documentation for Git and non-Git update paths
• Guard tests for version consistency, updater helpers, gap synthesis edge cases, and session compatibility

Validation

Passed before release:

• pytest tests --ignore=tests/integration --ignore=tests/test_novelty_checker_properties.py -q
• python -m compileall src preview_server.py
• npm test -- --runInBand

Result:

• 175 passed, 3 skipped
• JS tests passed
• compile checks passed

Notes

• The new app update flow stages the update into a new folder instead of overwriting the old one
• The updated app may launch on the next free localhost port if the current instance is still running
• Automatic in-place updater and native desktop updater are not part of this release yet

ScholarScout v1.6.1

v1.6.1 Release Notes (2026-06-16)

Dashboard workflow polish

• Added a shipped Export All button in the main dashboard controls for full-run Markdown export
• Added Export Selected, so users can pick a smaller working set instead of exporting every generated idea
• Export output now carries refinement notes, user-fit context, evidence claims, source papers, and risk flags
• Added a lightweight run summary strip so users can immediately see category spread and the latest session save timestamp

Reusable research setup

• Added lightweight run presets for saving and re-applying favorite category, context, and personality configurations
• Added a visible preference memory summary in Settings so vote reasons start surfacing as readable steering hints
• Added Goal Style steering with suggested defaults from Goal and mode-aware options for Academic, Product, and Develop workflows
• Goal Style now persists through presets, session state, exports, idea cards, idea detail modal, and pipeline snapshots

Stability and release fit

• CI now ignores integration tests explicitly instead of relying on collection-time markers
• Capability route no longer breaks ScholarScout startup when capabilityscout is not installed; the route degrades gracefully with a clear API error
• Stale structural tests for unfinished dashboard experiments are skipped so shipped UI and test expectations stay aligned
• Release notes and shipped scope were tightened so v1.6.x docs no longer overclaim unfinished dashboard shells

Testing

• Added coverage for Goal Style defaults and prompt steering
• Core Python tests, JS tests, compile checks, and local dashboard smoke checks passed for the shipped v1.6.1 path

ScholarScout v1.6.0

v1.6.0 Release Notes (2026-06-15)

Research-grade trust layer

• Evidence Pack on every generated idea: source papers, evidence claims, grounding score, and risk flags
• Invalid or missing paper references now surface explicitly instead of failing silently
• Deep Dive grounding verification and evidence-aware serialization are now part of the default research workflow

Better idea quality

• Self-improvement upgraded from light text cleanup to a critique-and-rewrite loop
• Refined ideas can now explain their main weakness, why the rewritten version is stronger, where the novelty sits, and what execution risk still remains
• Refined ideas are marked directly in the dashboard and idea detail modal

Personalized idea steering

• Added user preference steering for generation and refinement
• New personality controls: work style, scope preference, risk tolerance, output tone, and extra constraints
• Upvote/downvote now supports reason capture, so ScholarScout can learn what the user actually likes or rejects
• Ideas can now show fit-to-user summary, misalignment flags, and user fit score

Dashboard workflow improvements

• Idea detail modal upgraded with a floating action rail for Deep Dive, Export, Copy, Bookmark, Regenerate, and feedback
• Export and copy from the modal now include refinement and user-fit context
• Session rendering remains backward-compatible with older ideas that do not yet have evidence or refinement fields

Reliability and generation fairness

• Idea distribution across selected categories is now dynamic and fairer
• Fixed the old behavior where early categories could consume the full idea budget before later categories were reached
• Quick mode, regenerate, and full pipeline now all accept personality/profile context

Research tooling

• Implementation discovery support is included for finding related code, tools, datasets, and packages
• Health/usage helpers, evidence helpers, and analysis utilities added to support more audit-ready output
• config.yaml removed from version control; config.example.yaml remains the source of truth for setup

API and backend

• Added or extended support for POST /api/implementations, POST /api/refine, and POST /api/roadmap
• Personality-aware request flow now reaches the pipeline and idea generation routes

Testing

• Added coverage for evidence helpers, refinement flow, personalization helpers, implementation discovery, and UI structural checks
• Core Python tests, JS tests, and compile checks passed for the shipped v1.6 ScholarScout path

v1.5.3 — Review Mode, Activity Center & Paper Freshness

v1.5.3 — Review Mode, Activity Center & Paper Freshness (2026-05-26)

Review Mode (new pipeline)

• 4th mode: Literature Synthesis — clusters papers by theme, synthesizes findings per cluster, identifies gaps and open questions
• 6-phase pipeline: Validate → Fetch → Cluster → Synthesize → Cross-cutting → Save
• Cluster cards in idea grid (not idea cards) — click to see full synthesis
• Cross-cutting analysis: timeline, active debates, open questions, reading list
• Review sessions saved to history (visible in Recent tab with proper cluster rendering)
• Context field required for Review mode (validated on Save and Run)

Activity Center (pipeline visualization popup)

• Owl Chase pixel art game — owl catches paper dots while pipeline runs
• Paper drip-feed system: papers spawn one-by-one (600ms interval) for gameplay enjoyment
• Live graph with hover tooltip (shows category/cluster name + paper count)
• Dot repositioning when new categories appear (columns rebalance)
• LLM Chat tab: narrated conversation between Scout and AI
• Adaptive phase list (5 phases default, 6 phases review)
• Mini notification bar when Activity Center closed but pipeline running

Paper Freshness System

• Papers track _used_count in cache — least-used papers prioritized for idea generation
• When all papers exhausted (used 2+ times): auto-widens date range by 14 days, fetches fresh
• papers_exhausted SSE event shown as orange notice in console

Threshold Externalization

• All similarity thresholds moved to config.yaml → thresholds: section
• Config.THRESHOLD_* class attributes read from YAML (7 thresholds total)
• config.example.yaml created as template for contributors

Security & Quality

• config.yaml added to .gitignore (API keys no longer at risk of commit)
• Prompt injection delimiters on paper abstracts passed to LLM
• 14 new unit tests: test_clusterer.py (7) + test_synthesizer.py (7)
• Skill loader now searches REVIEW/ folder

Rate Limit Improvements

• Semantic Scholar: retry reduced from 3x/45s to 2x/10s — skip faster, other sources cover
• Inter-category delay increased from 4s to 6s (fewer arXiv 429s)
• OpenAlex: cs.IR moved to keyword search (concept ID was wrong — returned law papers)
• Polite User-Agent with mailto on S2 requests

UX Polish

• Profile popup hidden by default (was blocking page on load)
• Goal selection syncs immediately on card click (no need to click Save first)
• All alert() replaced with custom toast notifications (slide-in, auto-dismiss)
• "What's New" section no longer hardcodes version number
• Transparency panel shows all 8 sources (was hardcoded to 3)
• Recent tab: review sessions show "X clusters · Y papers" with topic subtitle
• preview_server.py reads version from VERSION file dynamically

Developer Experience

• .github/ISSUE_TEMPLATE/ (bug report + feature request)
• .github/PULL_REQUEST_TEMPLATE.md with version bump checklist
• CONTRIBUTING.md: removed stale @prompt-auditor reference
• package-lock.json removed from .gitignore (commit lockfile for reproducibility)

Breaking changes

• None. Fully backward compatible with v1.5.2 data files.

v1.5.2 — 8 Sources, Smart Routing, English-Only

v1.5.2 — 8 Sources & English-Only (2026-05-25)

Features

• 8 paper sources: arXiv, OpenAlex, Semantic Scholar, PubMed, Crossref, DOAJ, Scopus, DBLP
• Smart source routing: auto-selects best 3-4 sources per category
• Profile popup redesigned: mode-first flow (Academic → Product → Develop), then goals appear
• Subtle hover effects and fade transitions on profile cards
• What's New section for first-time users
• Settings shows all 8 data sources with descriptions
• Docs tab links to scholarscout.neej4.workers.dev/docs
• Live Demo link in README

Changes

• All UI and code strings converted to English
• Bahasa Indonesia section removed from README (output language still available)
• CSS polish: font smoothing, custom scrollbar, shadow variables

New fetchers

• pubmed_fetcher.py — 36M+ biomedical papers (NCBI E-utilities, no key)
• crossref_fetcher.py — 150M+ DOI records (REST, no key)
• doaj_fetcher.py — 9M+ open access articles (no key)
• scopus_fetcher.py — 90M+ records (needs SCOPUS_API_KEY env var)
• dblp_fetcher.py — 6M+ CS conference papers (no key)

Source routing logic

Category	Sources
cs., stat., eess.*	arXiv + S2 + OpenAlex + DBLP
med.*	PubMed + S2 + Crossref + Scopus
bio., q-bio.	PubMed + OpenAlex + Crossref + DOAJ
physics.*	arXiv + S2 + OpenAlex + Crossref
eng.*	Crossref + OpenAlex + S2 + Scopus
chem.*	Crossref + OpenAlex + S2 + PubMed
math.*	arXiv + S2 + OpenAlex + Crossref
soc.*	Crossref + OpenAlex + DOAJ + S2
earth., agri.	Crossref + OpenAlex + DOAJ + PubMed

v1.5.1 — UX, Chunked Generation, CI

What's new

Cleaner controls layout

Action buttons (Profile, Run, Quick, Clear) on the left. Date range, language, and idea count on the right. First-time users see primary actions immediately.

Chunked idea generation

Large requests split into batches of 3 per LLM call. If one batch fails (truncated JSON), remaining batches still succeed. No more lost ideas from a single malformed response.

Cache expiry

Papers older than 7 days auto-pruned from cache. Configurable via \eatures.cache_expiry_days\ in config.yaml.

Feature flags

Centralized in config.yaml:
All overridable via env vars (\SCOUT_REFINE=1, etc).

CI/CD

GitHub Actions on every push/PR:

• Python 3.10 / 3.11 / 3.12 matrix
• pytest + npm test
• Node.js 22

Bug fixes

• Deep Dive modal no longer appears on page refresh
• Structural tests updated to match actual code

---

Upgrade

git pull origin main
pip install -e .

No config changes required. Existing config.yaml works as-is.

Full Changelog: https://github.com/neej4/ScholarScout/blob/main/CHANGELOG.md

v1.5.0 — Trust & Quality

v1.5.0 — Trust & Quality

Three new opt-in features that make generated ideas sharper and more trustworthy.

---

Grounding verification for Deep Dive

Each Deep Dive section is now compared against the source paper using semantic similarity. Sections get a badge:

• Grounded (green) — aligns with source paper (≥65% similarity)
• Partial (yellow) — some claims may be inferred (40-65%)
• Caution (red) — may not be directly supported (<40%)

Enable in Settings → "Verify Deep Dive grounding". Adds ~5s per Deep Dive.

Self-distillation refinement

After generating ideas, the LLM re-reads its own output and refines: removes redundancy, sharpens titles, improves feasibility. Same ideas, better quality.

Enable in Settings → "Self-distillation refinement". Adds 1 extra LLM call per batch.

Prompt sensitivity check

Trend analysis now optionally runs with 2 different prompt framings (consensus vs contradiction). Compares keyword agreement and adjusts confidence score accordingly.

Enable in Settings → "Prompt sensitivity check". Adds 1 extra LLM call per category.

---

Other changes

• Idea card footer fixed — "Check Novelty" and "Show more" always visible (never clipped by overflow)
• Version auto-injected from VERSION file (no more hardcoded version in dashboard)
• All three features off by default — zero behavior change for existing users

From community PR (thanks @Bortlesboat)

• Fixed setuptools build backend for editable installs
• Integration test mocks retargeted to blueprint modules
• Testing docs updated for pip install -e ".[dev]"
• Hypothesis cache added to .gitignore

---

Upgrade: git pull && pip install -r requirements.txt

Full changelog: CHANGELOG.md

v1.4.1

v1.4.1

Cleanup patch.

• Removed light theme (dark only — cleaner, one less thing to maintain)
• Removed requirements-dev.txt (dev deps available via pip install .[dev] from pyproject.toml)
• Updated README testing section

No functional changes. Upgrade: git pull

v1.4.0

ScholarScout v1.4.0

Three ways to turn papers into action.

---

What's new

Three idea generation modes

ScholarScout now generates ideas in three modes from the same papers:

• Academic — thesis topics, methodology, key papers (existing)
• Product — buildable products with MVP features, tech stack, revenue model, competitors
• Develop — features and improvements for your existing project, grounded in recent papers

Develop mode treats your project description as a hard constraint. Every idea must be directly applicable to what you're building.

18 skill profiles

Skills restructured into three categories:

• skills/ACADEMIC/ — 9 profiles (Undergraduate through Grant Proposal)
• skills/PRODUCT/ — 4 profiles (Hackathon, Side Project, AI Tool, Industry R&D)
• skills/DEVELOP/ — 5 profiles (Feature, Integration, Optimization, Extension, Pivot)

Onboarding wizard

3-step setup on first launch: pick provider → test connection → choose categories. Supports custom endpoints (9router, LM Studio, any OpenAI-compatible proxy).

File upload

Drag-and-drop .pdf/.txt/.md/.json in the Profile popup. Content is used as extra context for both Quick mode and full pipeline Run.

Semantic novelty checking

When using Gemini, novelty check now uses text-embedding-004 for cosine similarity instead of token-level Jaccard. Falls back gracefully if embedding API is unavailable.

Architecture refactor

• Server split into 6 Flask Blueprint modules
• SSE streaming response parser (handles local proxies that stream even with stream: false)
• Cache-aware fetching (second run skips API calls if cache has papers)
• pyproject.toml with project metadata
• Semantic Scholar API key support (S2_API_KEY env var)

---

Bug fixes

• Goal selection not reaching pipeline (hidden select missing product/develop options)
• Quick mode ignoring goal (always used academic prompt)
• Profile popup save broken after refactor
• Wizard skip not persisting
• Upload endpoint 404 until restart
• Token budget too low for product mode (truncated JSON)
• Empty file upload accepted silently
• Negative session index wrap-around
• Test connection timeout too long for local endpoints

---

Breaking changes

None. Existing config.yaml and session data work without modification.

---

Install / upgrade

git pull
pip install -r requirements.txt
python preview_server.py

Open http://localhost:5050 — the wizard will guide you if it's your first time.

---

Full changelog: CHANGELOG.md

v1.3.0 — Quick Mode, Quality Scoring, Cross-Pollination

What's new

Quick Mode

Generate ideas in ~10 seconds from cached papers. No fetching, no waiting. Click "Quick" instead of "Run".

Quality Scoring

LLM self-rates each idea 1-10. Ideas below 5 are automatically filtered out.

Cross-Pollination

Analyzer suggests combining research gaps with techniques from other fields.

Saturation Detection

Each category is labeled: saturated, growing, or emerging. Generator adjusts scope accordingly.

Prerequisites

Each idea lists 3-5 skills needed to execute it.

13 Research Profiles

New: Hackathon (4-12h), Side Project (weekends), Industry R&D, AI Tool. Total 13 skills.

Settings Overhaul

Full settings page: LLM provider, pipeline behavior, data source toggles, storage management, about section. Every field has inline help text.

Deep Dive Caching

Second click loads instantly from memory.

Citation Quality Filter

Papers sorted by citation count. High-impact papers analyzed first.

---

Bug fixes

• Deep Dive "Key References" showing [object Object] — fixed
• Deep Dive timeout too short — increased to 180s
• Regenerate failing — rewritten with standalone prompt
• Settings input styling mismatch — fixed
• Variable conflict breaking all buttons — fixed
• ArXiv rate limiting — faster skip, parallel compensates
• OpenAlex wrong results for medical categories — switched to keyword search

---

Full changelog

See CHANGELOG.md