Society Speaks

Overview

Society Speaks is a public discussion platform where nuanced debate leads to better understanding and real solutions. Inspired by Pol.is, it features a native consensus-building system that uses machine learning to identify opinion groups, find common ground, and surface divisive topics - all without traditional threaded comments or upvotes.

Community Compact

Society Speaks is committed to building a thriving, transparent, and supportive community. We value every contribution, feedback, and use of the platform. We encourage everyone to use the software freely and ethically, and to share back improvements so everyone benefits. Our goal is to align our success with the success of our community.

Support This Project

Society Speaks is open source, and your support helps us keep it alive! If you find this platform useful, please consider donating to help us continue development and maintenance.

• Sponsor on GitHub

Thank you for your support!

Roadmap & Ideas

To see our future plans and ideas, check out IDEAS.md.

Features

Core Discussion System

• Native Statement System - One-click voting (Agree/Disagree/Unsure) with progressive disclosure
• Threaded Responses - Pro/con/neutral responses to statements with evidence linking
• Evidence Attachments - Citations, URLs, and file uploads via Replit Object Storage
• Moderation Queue - Flag and review system for discussion owners
• Edit Windows - 10-minute edit window for statements, then immutable for integrity
• Programmes - Multi-discussion dialogue containers with phases, themes, cohorts, steward access, and streaming export

Consensus Analysis (Machine Learning)

• Opinion Clustering - PCA dimensionality reduction + Agglomerative clustering
• Consensus Detection - Identifies statements with broad agreement across groups
• Bridge Statements - Finds statements that unite different opinion clusters
• Divisive Statements - Surfaces controversial topics with high disagreement
• Silhouette Scoring - Automatic cluster count optimization
• CSV-First Export - Export analysis results as CSV by default (with optional JSON for integrations)

AI Integration (Optional)

• LLM Support - OpenAI, Anthropic (Claude), and Mistral integration
• AI Summaries - Generate discussion summaries automatically
• Cluster Labeling - AI-generated names for opinion groups
• Semantic Deduplication - Prevent similar statements using embeddings
• User-Provided Keys - Encrypted API key storage (Fernet encryption)

Trending Topics System

• News Fetching - RSS feed integration with feedparser
• Topic Clustering - Automatic grouping of related news articles
• Discussion Generation - Create discussions from trending topics
• Social Posting - Bluesky integration for sharing discussions

User Features

• Individual & Company Profiles - Customizable public profiles
• Geographic Filtering - Location-based discussion discovery
• Topic-Based Filtering - Find discussions by subject area
• Notification System - Stay updated on discussion activity
• Admin Dashboard - User, profile, and discussion management

Security & Performance

• CSRF Protection - Flask-SeaSurf integration
• Secure Headers - Flask-Talisman with CSP
• Rate Limiting - Flask-Limiter on sensitive endpoints
• Redis Caching - Improved performance and session management
• Background Jobs - APScheduler for clustering and cleanup tasks
• Error Tracking - Sentry.io integration

Demo

You can try the live version at https://societyspeaks.io/

Partner integration (embeds & API)

Publishers can embed Society Speaks on their articles and use the Partner API for lookup, snapshots, and optional discussion creation.

• Partner hub: For Publishers — overview, three primitives, embed code generator, Rules of the Record.
• API reference: API Reference — lookup by article URL, snapshot, oEmbed, create discussion; all with curl examples.
• Try the API: API Playground — interactive Swagger UI to call the Partner API from the browser.

Optional: set DEMO_DISCUSSION_ID in your environment (e.g. DEMO_DISCUSSION_ID=123) to show a “See example embed” link on the partner hub so visitors can open a live embed in one click.

Tech Stack

Category	Technology
Backend	Python 3.11+ / Flask 3.0+
Database	PostgreSQL
Caching	Redis
Storage	Replit Object Storage
Frontend	Tailwind CSS
ML/Clustering	scikit-learn, numpy, pandas
Background Jobs	APScheduler
Encryption	cryptography (Fernet)
Social Integration	atproto (Bluesky)
News Fetching	feedparser
Monitoring	Sentry.io
Security	Flask-Talisman, Flask-SeaSurf
Session Management	Flask-Session with Redis
Rate Limiting	Flask-Limiter
Email	Flask-Mail

Requirements

# Core Flask (see requirements.txt for full pinned list)
Flask>=3.0.0
Flask-SQLAlchemy==3.0.5
Flask-Migrate==4.0.0
Flask-Login==0.6.3
Flask-WTF==1.2.2
WTForms==3.2.1
Flask-Session==0.8.0
Flask-Caching==2.3.0
Flask-Talisman==1.1.0
Flask-SeaSurf==2.0.0
Flask-Limiter==3.8.0
Flask-Mail==0.10.0

# Database
psycopg2-binary==2.9.10

# Machine Learning & Data
scikit-learn>=1.3.0
numpy>=1.24.0
pandas>=2.0.0

# Background Tasks
APScheduler==3.10.4

# Encryption
cryptography>=41.0.0

# Social & News
atproto
feedparser

# Utilities
python-slugify==8.0.1
email_validator==2.2.0
python-dotenv

# Monitoring
sentry-sdk==2.17.0

# Infrastructure
redis==5.2.0
replit>=4.1.0
gunicorn==21.2.0

Environment Variables

# Core Configuration
SECRET_KEY=your_secret_key
DATABASE_URL=postgresql://user:password@localhost:5432/society_speaks
REDIS_URL=redis://localhost:6379/0

# Email Configuration
MAIL_SERVER=smtp.your-email-server.com
MAIL_PORT=587
MAIL_USE_TLS=True
MAIL_USERNAME=your_email
MAIL_PASSWORD=your_password

# Error Tracking
SENTRY_DSN=your_sentry_dsn

# Bluesky Integration (optional - for social posting)
BLUESKY_HANDLE=your_handle.bsky.social
BLUESKY_APP_PASSWORD=your_app_password

# Environment
FLASK_ENV=development  # or production

Note: LLM API keys (OpenAI, Anthropic, Mistral) are provided by users in their account settings and stored encrypted. They are not configured as environment variables.

Installation

Local Development

1. Clone the repository:

git clone https://github.com/zuluwill/Societyspeaks.git
cd societyspeaks

2. Set up Python environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

3. Install Node dependencies:

npm install

4. Configure environment:

cp .env.example .env
# Edit .env with your configurations. For Daily Brief vars see .env.brief.example.

5. Set up Redis and PostgreSQL:

# Start Redis server
redis-server

# Create PostgreSQL database
createdb societyspeaks

6. Initialize database:

flask db upgrade
flask seed-db  # Optional: for sample data

7. Compile Tailwind CSS:

# Development (with watch)
npx tailwindcss -i ./app/static/src/input.css -o ./app/static/css/output.css --watch

# Production (minified)
npx tailwindcss -i ./app/static/src/input.css -o ./app/static/css/output.css --minify

8. Run the application:

flask run
# Or for production:
gunicorn --bind 0.0.0.0:5000 run:app

Replit Setup

1. Fork the Repl

2. Configure Secrets:

   • DATABASE_URL
   • SECRET_KEY
   • REDIS_URL
   • SENTRY_DSN (optional)
   • MAIL_* configurations
   • BLUESKY_* configurations (optional)

3. Install dependencies and initialize:

pip install -r requirements.txt
flask db upgrade

Alembic / multiple heads: If flask db upgrade fails with "multiple head revisions", the migration history has branched. Merge heads before deploying: flask db merge heads -m "merge_heads", then run flask db upgrade again.

Project Structure

society_speaks/
├── app/
│   ├── admin/              # Admin dashboard routes and forms
│   │   ├── routes.py
│   │   └── forms.py
│   ├── auth/               # Authentication (login, register, password reset)
│   │   └── routes.py
│   ├── discussions/        # Core discussion functionality
│   │   ├── routes.py       # Discussion CRUD
│   │   ├── statements.py   # Statement voting and management
│   │   ├── consensus.py    # Consensus analysis routes
│   │   ├── moderation.py   # Moderation queue
│   │   └── forms.py
│   ├── help/               # User documentation
│   │   └── routes.py
│   ├── lib/                # Core libraries
│   │   ├── consensus_engine.py  # ML clustering algorithms
│   │   └── llm_utils.py         # LLM integrations
│   ├── profiles/           # User profiles (individual & company)
│   │   ├── routes.py
│   │   └── forms.py
│   ├── settings/           # User settings and API key management
│   │   ├── routes.py
│   │   └── api_keys.py
│   ├── trending/           # Trending topics system
│   │   ├── routes.py
│   │   ├── news_fetcher.py
│   │   ├── clustering.py
│   │   └── social_poster.py
│   ├── templates/          # Jinja2 templates
│   ├── static/             # CSS, JS, images
│   ├── models.py           # SQLAlchemy models
│   ├── routes.py           # Main routes (index, about, etc.)
│   ├── scheduler.py        # APScheduler background jobs
│   └── __init__.py         # App factory
├── docs/                   # Implementation documentation
├── migrations/             # Alembic database migrations
├── scripts/                # Utility scripts
├── config.py               # Configuration classes
├── run.py                  # Application entry point
├── requirements.txt
├── package.json            # Tailwind dependencies
└── tailwind.config.js

Security Features

• CSRF Protection - Flask-SeaSurf on all forms
• Secure Headers - Flask-Talisman with Content Security Policy
• Rate Limiting - Configurable limits on sensitive endpoints
• Session Security - Redis-backed secure sessions
• Input Validation - Character limits, type checking, sanitization
• Encrypted API Keys - Fernet encryption for user LLM keys
• Edit Windows - 10-minute edit window, then statements are immutable
• Soft Deletes - Audit trail preservation
• Permission Checks - Owner/moderator role verification

Monitoring and Performance

• Error Tracking - Sentry.io integration
• Redis Caching - Response caching and session storage
• Database Indexes - Optimized queries on foreign keys
• Denormalized Counts - Avoid expensive COUNT(*) queries
• Background Jobs - Non-blocking clustering analysis
• Pagination - 20 items per page default
• Old Analysis Cleanup - Keeps only 10 most recent per discussion

Documentation

• COMPLETE_SYSTEM_GUIDE.md - Full technical documentation
• QUICK_START.md - Getting started guide
• IDEAS.md - Future plans and roadmap
• CONTRIBUTING.md - Current contribution policy
• CODE_OF_CONDUCT.md - Community standards
• MAINTAINERS.md - Maintainer and governance details
• SECURITY.md - Private vulnerability disclosure process
• .github/LABELS.md - Suggested issue triage labels
• docs/ - Implementation summaries

Contributing

External pull requests are currently paused while the project is maintained by a single contributor.

• Please open an issue for bugs, ideas, or collaboration requests.
• If external contributions are reopened, contribution terms and process will be documented in CONTRIBUTING.md.

Known Issues

• Image upload size limited to 10MB on Replit Object Storage
• Clustering requires minimum 7 users with votes
• Large vote matrices (>1000 users) may slow clustering

Support

For support, please open an issue or contact the maintainers.

Acknowledgments

• Pol.is - Inspiration for consensus-building approach
• scikit-learn - Clustering algorithms
• Tailwind CSS - Styling framework
• Flask - Web framework
• APScheduler - Background job scheduling
• Replit - Hosting and development environment
• AllSides - Media bias ratings reference

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).

• Full text: LICENSE
• Copyright and ownership: COPYRIGHT
• Third-party notices: NOTICE

If you run a modified version of this software over a network, AGPL requires you to make the corresponding source available to users of that service.

Commercial Licensing

If you are a publisher, enterprise, or partner and need non-AGPL commercial terms for embeds, API usage, or distribution, contact the maintainers to discuss a commercial license.

---

Made with care by William Roberts