Skip to content

feat: SQLite-backed local store, per-source progress UI, taxonomy per…#2

Merged
HugoMSFT merged 1 commit into
mainfrom
feat/sqlite-store-progress-and-taxonomy-persistence
May 10, 2026
Merged

feat: SQLite-backed local store, per-source progress UI, taxonomy per…#2
HugoMSFT merged 1 commit into
mainfrom
feat/sqlite-store-progress-and-taxonomy-persistence

Conversation

@HugoMSFT

Copy link
Copy Markdown
Owner

…sistence

Local persistence:

  • Add src/local_store.py: thread-safe SQLite store (feedback + state tables, JSON list serialisation, CSV export/import, legacy CSV seeding).
  • Replace ad-hoc in-memory state with the LocalStore on every collection run; CSV becomes a snapshot, the DB is the source of truth.

CSV exports:

  • Reorder TABLE_COLUMNS so Sources and Url are columns 2 and 3 (right after Feedback_ID); previously buried at columns 5 and 29.

Taxonomy long-term persistence:

  • Save keywords / categories / impact_types to data/.json (writable) instead of src/.json. Cascading load order: data/ -> src/ -> in-code defaults.
  • Customizations survive application restarts, PyInstaller frozen builds (read-only src/), and source-tree upgrades.

Collection progress drawer:

  • Add per-source state tracking (pending / running / success / error / skipped) on the server with a lock-protected helper, surfaced via SSE.
  • Replace the static 4-card grid in index.html with a dynamic per-source list rendered from collection_status.source_states; includes status badges, spinner for running, item counts, and a summary line.
  • Mark still-running / pending sources as "error" if the run aborts so the drawer never shows a stuck spinner.

.env handling:

  • Rewrite .env.template to match the variable names actually read by config.py (REDDIT_, GITHUB_TOKEN, ADO_, optional Fabric / Flask).

Frozen build (PyInstaller 6+):

  • Add src/ to spec pathex and pin module hiddenimports so run_web / app / config / etc. resolve at runtime.
  • Update runtime_paths._select_source_dir to probe sys._MEIPASS/src and PROJECT_ROOT/_internal/src layouts.
  • start_feedback_collector.py reconfigures stdout / stderr to UTF-8 (errors='replace') so emoji-laden log lines don't crash on cp1252-redirected streams.

…sistence

Local persistence:
- Add src/local_store.py: thread-safe SQLite store (feedback + state
  tables, JSON list serialisation, CSV export/import, legacy CSV
  seeding).
- Replace ad-hoc in-memory state with the LocalStore on every
  collection run; CSV becomes a snapshot, the DB is the source of truth.

CSV exports:
- Reorder TABLE_COLUMNS so Sources and Url are columns 2 and 3 (right
  after Feedback_ID); previously buried at columns 5 and 29.

Taxonomy long-term persistence:
- Save keywords / categories / impact_types to data/*.json (writable)
  instead of src/*.json. Cascading load order: data/ -> src/ -> in-code
  defaults.
- Customizations survive application restarts, PyInstaller frozen
  builds (read-only src/), and source-tree upgrades.

Collection progress drawer:
- Add per-source state tracking (pending / running / success / error /
  skipped) on the server with a lock-protected helper, surfaced via SSE.
- Replace the static 4-card grid in index.html with a dynamic per-source
  list rendered from collection_status.source_states; includes status
  badges, spinner for running, item counts, and a summary line.
- Mark still-running / pending sources as "error" if the run aborts so
  the drawer never shows a stuck spinner.

.env handling:
- Rewrite .env.template to match the variable names actually read by
  config.py (REDDIT_*, GITHUB_TOKEN, ADO_*, optional Fabric / Flask).

Frozen build (PyInstaller 6+):
- Add src/ to spec pathex and pin module hiddenimports so
  run_web / app / config / etc. resolve at runtime.
- Update runtime_paths._select_source_dir to probe sys._MEIPASS/src
  and PROJECT_ROOT/_internal/src layouts.
- start_feedback_collector.py reconfigures stdout / stderr to UTF-8
  (errors='replace') so emoji-laden log lines don't crash on
  cp1252-redirected streams.
@HugoMSFT
HugoMSFT merged commit 1be8bb0 into main May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant