Skip to content

feat(dashboard): post layout and infinite scroll#3

Closed
Marob-nod wants to merge 13 commits into
zkorum:mainfrom
Marob-nod:first-branch
Closed

feat(dashboard): post layout and infinite scroll#3
Marob-nod wants to merge 13 commits into
zkorum:mainfrom
Marob-nod:first-branch

Conversation

@Marob-nod

@Marob-nod Marob-nod commented Aug 20, 2023 •
edited by nicobao
Loading

Copy link
Copy Markdown
Contributor

Closes #1

nicobao
nicobao reviewed Aug 22, 2023
View reviewed changes
Comment thread services/front/src/components/dashboard/Post.tsx Outdated
Comment thread services/front/src/components/dashboard/Post.tsx Outdated
Comment thread services/front/src/components/dashboard/Post.tsx Outdated
@nicobao nicobao changed the title first commit of all times feat(dashboard): post layout and infinite scroll Aug 22, 2023
@Marob-nod

Marob-nod commented Aug 26, 2023

Copy link
Copy Markdown
Contributor Author

Thanks for your comments. I am reviewing my code right now. I can't auto indent my code while saving even if my mode is On in my settings. Do you know how i can add that to VS code ?

@nicobao

nicobao commented Aug 26, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Thanks for your comments. I am reviewing my code right now. I can't auto indent my code while saving even if my mode is On in my settings. Do you know how i can add that to VS code ?

Have you downloaded vscode-eslint extension?
https://github.com/microsoft/vscode-eslint
The front currently does not have prettier, just eslint.
It uses .eslint.cjs. In case it still does not work, try looking at: microsoft/vscode-eslint#1456

@Marob-nod

Marob-nod commented Aug 26, 2023

Copy link
Copy Markdown
Contributor Author

Thanks for your comments. I am reviewing my code right now. I can't auto indent my code while saving even if my mode is On in my settings. Do you know how i can add that to VS code ?

Have you downloaded vscode-eslint extension? https://github.com/microsoft/vscode-eslint The front currently does not have prettier, just eslint. It uses .eslint.cjs. In case it still does not work, try looking at: microsoft/vscode-eslint#1456

Es-lint shows the erros of my code but to indent it correctly

@Marob-nod Marob-nod closed this Aug 26, 2023
@nicobao

nicobao commented Aug 26, 2023

Copy link
Copy Markdown
Member

I think you closed the PR by accident!

@Marob-nod Marob-nod reopened this Aug 26, 2023
@Marob-nod

Marob-nod commented Aug 26, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Oh yeah my bad, i just reopened it !

@nicobao

nicobao commented Aug 28, 2023

Copy link
Copy Markdown
Member

Thanks for your comments. I am reviewing my code right now. I can't auto indent my code while saving even if my mode is On in my settings. Do you know how i can add that to VS code ?

Have you downloaded vscode-eslint extension? https://github.com/microsoft/vscode-eslint The front currently does not have prettier, just eslint. It uses .eslint.cjs. In case it still does not work, try looking at: microsoft/vscode-eslint#1456

Es-lint shows the erros of my code but to indent it correctly

Just added prettier it should work now.

@nicobao

nicobao commented Aug 29, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

For info, I also harmonized the eslint config across the stack

@Marob-nod Marob-nod closed this Sep 10, 2023
nicobao referenced this pull request in nicobao/agora Oct 28, 2025
@nicobao
perf(database): add composite index for moderation queries and enhanc…
4a8cd74 
…e monitoring

This commit implements database performance optimizations and infrastructure
improvements based on load testing analysis.

Database Schema Changes:
- Added composite index on conversation_moderation(conversation_id, moderation_action)
  to optimize lock status checks (V0027 migration)
- Renumbered V0024 migration to V0025 to maintain sequential ordering
- Added V0026 migration for previous schema changes
- The composite index reduces query plan cost for the 4,950 moderation checks
  observed during load testing (~14% of total queries)

Math Updater Improvements - Queue Locking Strategy:
- Fixed race condition in conversation update queue processing
- Changed early locking from processedAt to lastMathUpdateAt
- New logic: requestedAt > lastMathUpdateAt determines if processing needed
- lastMathUpdateAt is only touched by math-updater (not API), preventing race conditions
- Added detailed pg-boss job state logging when singleton jobs are rejected
- Shows whether jobs are RUNNING, QUEUED, or COMPLETED with timing information
- Sets processedAt only if requestedAt didn't change during processing

Math Updater Improvements - Scanner Logic:
- Updated scanner to check requestedAt > lastMathUpdateAt instead of processedAt IS NULL
- Added comprehensive job state queries to pgboss.job table
- Logs job age, running time, and completion time for better observability
- Prevents duplicate job enqueueing when jobs are running or recently completed

Math Updater - Batch Query Implementation (from main, leftover from rebase):
- Opinion updates are batched (1000 opinions per batch) to avoid PostgreSQL parameter limits
- Uses batchArray helper function to split large opinion lists
- Cluster stats organized in lookup maps for efficient batch processing
- Note: These changes were already merged to main but appear in diff due to rebase

Monitoring & Infrastructure:
- Enhanced PostgreSQL monitoring configuration in postgresql-monitoring.conf
- Updated Prometheus configuration for better metrics collection
- Improved Docker Compose setup for development environment
- Added connection pooling and read replica routing configuration

Load Testing:
- Updated scenario to test moderation lock checks under load
- Enhanced API client utilities for better test coverage
- Improved user action simulation for realistic testing

Documentation:
- Added DATABASE.md with comprehensive database architecture documentation
- Updated PERFORMANCE_ANALYSIS.md with October 2025 optimization details
- Added git commit guidelines to CLAUDE.md following Conventional Commits standard
- Includes guideline to verify staged changes and check for accidental commits
- Explicitly instructs not to mention AI assistants in commit messages
- Documented read replica strategy and query optimization approaches

Python Bridge:
- Minor updates to clustering service for better compatibility

Shared Backend Syncing:
- Synced shared-backend code to api and math-updater services
- Updated schema.ts and db.ts across all consuming services
- This is standard rsync distribution of shared database layer code

Impact:
- Prevents math-updater job duplication race conditions
- Marginal query performance improvement (~1-2%) from composite index
- Significantly improved observability through enhanced logging
- Sets foundation for future caching layer implementation
- Better code maintainability with type-safe patterns

Technical Notes:
- The composite index provides minimal gain due to UNIQUE constraint on conversation_id,
  but demonstrates query optimization best practices
- The requestedAt/lastMathUpdateAt strategy is more reliable than processedAt because
  it's single-writer (only math-updater modifies lastMathUpdateAt)
- See PERFORMANCE_ANALYSIS.md Known Limitations #3 for planned caching layer
@nicobao nicobao mentioned this pull request Oct 28, 2025
Merged
nicobao added a commit that referenced this pull request Oct 28, 2025
@nicobao
feat: add ai translation + performance improvements + fix python-brid…
116b9cb 
…ge logic (#400)

* feat(translation): add AI-powered translation for cluster labels and summaries

Implement automatic translation of Polis cluster labels and summaries using Google Cloud Translation API. Translations are generated proactively
   during math updates and on-demand as fallback.

  Translation strategy:
  - Proactive: Translate all cluster labels/summaries during math updates for configured languages
  - On-demand: Translate individual clusters at request time if translation not found in database
  - Frontend automatically sends Accept-Language header based on user's display language preference

  Key changes:
  - Add Google Cloud Translation API integration with authentication
  - Implement cluster translation service with parallel processing (p-limit)
  - Add translation logic to math-updater service for batch translation
  - Add fallback translation in API service for missing translations
  - Include cluster ID in metadata for translation mapping
  - Add database migration for cluster translation storage
  - Configure translation services across all backend services

* perf(database): add composite index for moderation queries and enhance monitoring

This commit implements database performance optimizations and infrastructure
improvements based on load testing analysis.

Database Schema Changes:
- Added composite index on conversation_moderation(conversation_id, moderation_action)
  to optimize lock status checks (V0027 migration)
- Renumbered V0024 migration to V0025 to maintain sequential ordering
- Added V0026 migration for previous schema changes
- The composite index reduces query plan cost for the 4,950 moderation checks
  observed during load testing (~14% of total queries)

Math Updater Improvements - Queue Locking Strategy:
- Fixed race condition in conversation update queue processing
- Changed early locking from processedAt to lastMathUpdateAt
- New logic: requestedAt > lastMathUpdateAt determines if processing needed
- lastMathUpdateAt is only touched by math-updater (not API), preventing race conditions
- Added detailed pg-boss job state logging when singleton jobs are rejected
- Shows whether jobs are RUNNING, QUEUED, or COMPLETED with timing information
- Sets processedAt only if requestedAt didn't change during processing

Math Updater Improvements - Scanner Logic:
- Updated scanner to check requestedAt > lastMathUpdateAt instead of processedAt IS NULL
- Added comprehensive job state queries to pgboss.job table
- Logs job age, running time, and completion time for better observability
- Prevents duplicate job enqueueing when jobs are running or recently completed

Math Updater - Batch Query Implementation (from main, leftover from rebase):
- Opinion updates are batched (1000 opinions per batch) to avoid PostgreSQL parameter limits
- Uses batchArray helper function to split large opinion lists
- Cluster stats organized in lookup maps for efficient batch processing
- Note: These changes were already merged to main but appear in diff due to rebase

Monitoring & Infrastructure:
- Enhanced PostgreSQL monitoring configuration in postgresql-monitoring.conf
- Updated Prometheus configuration for better metrics collection
- Improved Docker Compose setup for development environment
- Added connection pooling and read replica routing configuration

Load Testing:
- Updated scenario to test moderation lock checks under load
- Enhanced API client utilities for better test coverage
- Improved user action simulation for realistic testing

Documentation:
- Added DATABASE.md with comprehensive database architecture documentation
- Updated PERFORMANCE_ANALYSIS.md with October 2025 optimization details
- Added git commit guidelines to CLAUDE.md following Conventional Commits standard
- Includes guideline to verify staged changes and check for accidental commits
- Explicitly instructs not to mention AI assistants in commit messages
- Documented read replica strategy and query optimization approaches

Python Bridge:
- Minor updates to clustering service for better compatibility

Shared Backend Syncing:
- Synced shared-backend code to api and math-updater services
- Updated schema.ts and db.ts across all consuming services
- This is standard rsync distribution of shared database layer code

Impact:
- Prevents math-updater job duplication race conditions
- Marginal query performance improvement (~1-2%) from composite index
- Significantly improved observability through enhanced logging
- Sets foundation for future caching layer implementation
- Better code maintainability with type-safe patterns

Technical Notes:
- The composite index provides minimal gain due to UNIQUE constraint on conversation_id,
  but demonstrates query optimization best practices
- The requestedAt/lastMathUpdateAt strategy is more reliable than processedAt because
  it's single-writer (only math-updater modifies lastMathUpdateAt)
- See PERFORMANCE_ANALYSIS.md Known Limitations #3 for planned caching layer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nicobao nicobao nicobao left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(dashboard): post layout and infinite scroll

2 participants

@Marob-nod @nicobao