feat: Introduce comprehensive observability and configuration options

This commit significantly expands the bot's configuration capabilities and introduces new observability features.

Key changes include:

*   **Enhanced `.env.example`**: Added numerous new environment variables for:
    *   Detailed logging configuration (level, format, file rotation, retention).
    *   Sentry integration for error tracking.
    *   Dedicated services: Command Analytics, Queue Telemetry, Lyrics, Scaling, DJ Roles, Chaos, Event Loop Profiling, Search Cache, Shard Supervisor, Metrics, Command Reference, Local Sandbox, Performance, Analytics, Code Coverage, Type Checking.
    *   Fine-grained bot settings: owner ID, prefix, activity, status, intents, sharding, player limits, queue/playlist sizes, track duration, search limits, and command cooldowns.
    *   Updated default values for Lavalink and Redis to `localhost` for easier local development.
    *   Adjusted crossfade defaults.
*   **Type Checking Integration**: Added `pyrightconfig.json` and `scripts/typecheck.sh` to enable static type analysis.
*   **Documentation**: Added `docs/development.md` and updated `FURTHER_DEVELOPMENT.md`, `README.md`, and `config.yml` to reflect new features and configuration.
*   **Core Logic Updates**:
    *   `src/main.py`: Integrated new configuration loading and service initialization based on the expanded settings.
    *   `src/configs/settings.py` and `src/configs/schema.py`: Updated to support the new environment variables and configuration structure.
    *   `src/events/music_events.py`: Added `_telemetry` and `_emit_queue_event` methods, and integrated lyrics fetching.
    *   `src/services/lavalink_service.py`: Updated to use new Lavalink configuration.
    *   `src/services/profile_service.py`: Added event loop profiling.
    *   `src/services/autoplay_service.py`, `src/services/playlist_service.py`: Minor adjustments.
*   **New Services/Modules (untracked files)**: The diff indicates the addition of many new files and directories related to the services mentioned above (e.g., `src/commands/chaos_commands.py`, `src/services/metrics_service.py`, `docs/analytics.md`, `tests/`). While not part of the diff, their presence in `??` suggests a major feature addition.

This commit lays the groundwork for advanced monitoring, debugging, and feature development by centralizing configuration and introducing modular service architecture.

Author: timhauke

FURTHER_DEVELOPMENT.md

@@ -37,13 +37,13 @@
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Lyrics Integration</strong> &mdash; Surface synchronised lyrics (Genius, Musixmatch) within now-playing embeds.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>DJ Permissions Layer</strong> &mdash; Add role-based queue control with an auditable history.
       </label>
     </li>
@@ -55,31 +55,31 @@
   <ul style="list-style: none; padding-left: 0;">
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Multi-Node Lavalink Failover</strong> &mdash; Provision redundant nodes with automatic reconnect and player migration.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Self-Healing Supervisors</strong> &mdash; Watch shard heartbeats and restart or reconnect when anomalies occur.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Structured Alerting</strong> &mdash; Export metrics to Prometheus/Alertmanager for proactive paging.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Chaos Testing Playbook</strong> &mdash; Run scheduled disconnect, latency, and error-injection drills.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Auto Scaling Strategy</strong> &mdash; Integrate with Kubernetes/Nomad to scale shards and nodes based on concurrency dashboards.
       </label>
     </li>
@@ -91,25 +91,25 @@
   <ul style="list-style: none; padding-left: 0;">
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Grafana Dashboards</strong> &mdash; Publish live dashboards for shard health, node stats, and slash command throughput.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Command Analytics Pipeline</strong> &mdash; Stream anonymised command events into a data warehouse for trend analysis.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Real-Time Queue Telemetry</strong> &mdash; Emit queue lifecycle events (play, skip, finish) to webhooks or WebSocket consumers.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Enhanced Slash Feedback</strong> &mdash; Display progress bars and follow-ups for long-running operations.
       </label>
     </li>

README.md

@@ -310,6 +310,10 @@ AUTOPLAY_RANDOM_PICK=true
 <ul>
   <li>Capture <code>stdout</code> for VectoBeat; enable log shipping in production (ELK, CloudWatch, etc.).</li>
   <li>Monitor Lavalink metrics: player count, CPU, memory, frame deficit.</li>
+  <li>Import the bundled Grafana dashboards in <code>docs/grafana</code> for shard latency, node health, and slash command throughput visualisations.</li>
+  <li>Enable the command analytics pipeline (<code>docs/analytics.md</code>) to push anonymised slash usage into your data warehouse.</li>
+  <li>Wire the queue telemetry webhook (<code>docs/queue_telemetry.md</code>) into your status site for real-time “now playing” indicators.</li>
+  <li>Long-running slash commands (e.g. playlist loading) now show live progress embeds so users know what’s happening.</li>
   <li>Regularly patch yt-dlp for source compatibility.</li>
   <li>Monitor Redis availability (<code>INFO</code>/<code>PING</code>) if playlist persistence is enabled.</li>
 </ul>
@@ -321,8 +325,8 @@ AUTOPLAY_RANDOM_PICK=true
 
 <ol>
   <li>Fork the repository and clone your fork.</li>
-  <li>Create a feature branch <code>git checkout -b feature/amazing-improvement</code>.</li>
-  <li>Run <code>python3 -m compileall src</code> before committing.</li>
+<li>Create a feature branch <code>git checkout -b feature/amazing-improvement</code>.</li>
+  <li>Run <code>python3 -m compileall src</code> and <code>scripts/typecheck.sh</code> before committing.</li>
   <li>Submit a pull request describing the motivation, approach, and testing performed.</li>
 </ol>
 

docs/development.md

@@ -0,0 +1,22 @@
+# Development Guide
+
+## Type Checking
+We require pyright to pass before merging. Install it globally via npm:
+
+```bash
+npm install -g pyright
+```
+
+Then run:
+
+```bash
+scripts/typecheck.sh
+```
+
+This runs pyright in strict mode using `pyrightconfig.json`.
+
+## Pre-Commit Checklist
+- `python3 -m compileall src`
+- `scripts/typecheck.sh`
+- `python3 -m pytest` (when tests are added)
+

pyrightconfig.json

@@ -0,0 +1,11 @@
+{
+  "$schema": "https://raw.githubusercontent.com/microsoft/pyright/main/packages/pyright-internal/config.schema.json",
+  "include": ["src"],
+  "venvPath": ".",
+  "venv": null,
+  "reportMissingImports": "warning",
+  "reportMissingTypeStubs": "warning",
+  "reportUntypedClassDecorator": "warning",
+  "pythonVersion": "3.11",
+  "typeCheckingMode": "strict"
+}

scripts/typecheck.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+set -euo pipefail
+if ! command -v pyright >/dev/null 2>&1; then
+  echo "pyright executable not found. Please npm install -g pyright or add it to PATH." >&2
+  exit 127
+fi
+pyright "${@:-src}"