feat: SQLite-backed local store, per-source progress UI, taxonomy per…#2
Merged
HugoMSFT merged 1 commit intoMay 10, 2026
Merged
Conversation
…sistence Local persistence: - Add src/local_store.py: thread-safe SQLite store (feedback + state tables, JSON list serialisation, CSV export/import, legacy CSV seeding). - Replace ad-hoc in-memory state with the LocalStore on every collection run; CSV becomes a snapshot, the DB is the source of truth. CSV exports: - Reorder TABLE_COLUMNS so Sources and Url are columns 2 and 3 (right after Feedback_ID); previously buried at columns 5 and 29. Taxonomy long-term persistence: - Save keywords / categories / impact_types to data/*.json (writable) instead of src/*.json. Cascading load order: data/ -> src/ -> in-code defaults. - Customizations survive application restarts, PyInstaller frozen builds (read-only src/), and source-tree upgrades. Collection progress drawer: - Add per-source state tracking (pending / running / success / error / skipped) on the server with a lock-protected helper, surfaced via SSE. - Replace the static 4-card grid in index.html with a dynamic per-source list rendered from collection_status.source_states; includes status badges, spinner for running, item counts, and a summary line. - Mark still-running / pending sources as "error" if the run aborts so the drawer never shows a stuck spinner. .env handling: - Rewrite .env.template to match the variable names actually read by config.py (REDDIT_*, GITHUB_TOKEN, ADO_*, optional Fabric / Flask). Frozen build (PyInstaller 6+): - Add src/ to spec pathex and pin module hiddenimports so run_web / app / config / etc. resolve at runtime. - Update runtime_paths._select_source_dir to probe sys._MEIPASS/src and PROJECT_ROOT/_internal/src layouts. - start_feedback_collector.py reconfigures stdout / stderr to UTF-8 (errors='replace') so emoji-laden log lines don't crash on cp1252-redirected streams.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…sistence
Local persistence:
CSV exports:
Taxonomy long-term persistence:
Collection progress drawer:
.env handling:
Frozen build (PyInstaller 6+):