feat: Introduce comprehensive observability and scaling features

This commit introduces a suite of new features focused on observability, performance, and scalability, alongside general improvements and bug fixes.

Key changes include:
*   **Observability**: Add metrics, command analytics, and queue telemetry services with corresponding events and documentation.
*   **Scaling**: Implement shard supervision and scaling commands/services to manage bot instances more effectively.
*   **DJ Permissions**: Introduce a dedicated service for managing DJ-specific permissions.
*   **Lyrics Service**: Add functionality to fetch and display song lyrics in now-playing embeds.
*   **Chaos Engineering**: Integrate chaos commands and service for testing resilience.
*   **Configuration**: Update `.env.example` and `config.yml` schema to support new features and refine existing settings.
*   **Development Tools**: Add `docker-compose.local.yml` and `.env.local.example` for easier local development and sandbox environments.
*   **Documentation**: Expand documentation with new sections on analytics, command reference, development, Grafana, local sandbox, performance, and queue telemetry.
*   **Dependencies**: Update `requirements.txt` to include new libraries for metrics, retries, throttling, and async task management.
*   **Code Refinements**: General code cleanup, improved embed utility, and minor bug fixes across various modules.

Author: timhauke

.env.example

@@ -1,32 +1,68 @@
-# Copy this file to `.env` and populate the secrets before running the bot
+# Core bot settings
+DISCORD_TOKEN=
+CONFIG_PATH=config.yml
 
-# Discord bot token (https://discord.com/developers/applications)
-DISCORD_TOKEN=YOUR_DISCORD_BOT_TOKEN
-
-# Lavalink connection overrides (optional if values already exist in config.yml)
-LAVALINK_HOST=lavalink.example.com
+# Lavalink configuration
+LAVALINK_HOST=localhost
 LAVALINK_PORT=2333
-LAVALINK_PASSWORD=supersecret
+LAVALINK_PASSWORD=localpass
 LAVALINK_HTTPS=false
 LAVALINK_NAME=main
 LAVALINK_REGION=eu
 
-# Redis settings for playlist persistence
-# Redis persistence (optional but required for /playlist features)
-REDIS_HOST=45.84.196.19
-REDIS_PORT=32768
+# Redis configuration
+REDIS_HOST=localhost
+REDIS_PORT=6379
 REDIS_PASSWORD=
 REDIS_DB=0
 
 # Autoplay tuning
 AUTOPLAY_DISCOVERY_LIMIT=10
 AUTOPLAY_RANDOM_PICK=true
 
-# Crossfade controls
-CROSSFADE_ENABLED=true
-CROSSFADE_DURATION_MS=2500
-CROSSFADE_STEPS=12
-CROSSFADE_FLOOR_VOLUME=20
+# Crossfade settings
+CROSSFADE_ENABLED=false
+CROSSFADE_DURATION_MS=2000
+CROSSFADE_STEPS=10
+CROSSFADE_FLOOR_VOLUME=15
+
+# Metrics exporter
+METRICS_ENABLED=false
+METRICS_HOST=0.0.0.0
+METRICS_PORT=9333
+METRICS_INTERVAL=15
+
+# Chaos testing
+CHAOS_ENABLED=false
+CHAOS_INTERVAL_MINUTES=360
+CHAOS_SCENARIOS=disconnect_voice,disconnect_node,inject_error
+CHAOS_GUILD_ALLOWLIST=
+
+# Auto scaling
+SCALING_ENABLED=false
+SCALING_ENDPOINT=
+SCALING_PROVIDER=nomad
+SCALING_AUTH_TOKEN=
+SCALING_INTERVAL_SECONDS=60
+SCALING_COOLDOWN_SECONDS=300
+SCALING_TARGETS=1200,150
+
+# Command analytics
+ANALYTICS_ENABLED=false
+ANALYTICS_ENDPOINT=
+ANALYTICS_API_KEY=
+ANALYTICS_FLUSH_INTERVAL=30
+ANALYTICS_BATCH_SIZE=50
+ANALYTICS_STORAGE_PATH=data/command_analytics.log
+ANALYTICS_HASH_SALT=vectobeat
+
+# Queue telemetry
+QUEUE_TELEMETRY_ENABLED=false
+QUEUE_TELEMETRY_ENDPOINT=
+QUEUE_TELEMETRY_API_KEY=
+QUEUE_TELEMETRY_INCLUDE_GUILD=true
 
-# Optional: specify an alternate configuration file
-# CONFIG_PATH=config.yml
+# Cache tuning
+CACHE_SEARCH_ENABLED=true
+CACHE_SEARCH_TTL_SECONDS=60
+CACHE_SEARCH_MAX_ENTRIES=200

.env.local.example

@@ -0,0 +1,13 @@
+DISCORD_TOKEN=
+# Override endpoints to point at the local sandbox
+LAVALINK_HOST=lavalink
+LAVALINK_PORT=2333
+LAVALINK_PASSWORD=localpass
+LAVALINK_HTTPS=false
+REDIS_HOST=redis
+REDIS_PORT=6379
+REDIS_PASSWORD=
+REDIS_DB=0
+QUEUE_TELEMETRY_ENABLED=false
+SCALING_ENABLED=false
+CHAOS_ENABLED=false

.github/instructions/codacy.instructions.md

@@ -6,13 +6,6 @@
 # Codacy Rules
 Configuration for AI behavior when interacting with Codacy's MCP Server
 
-## using any tool that accepts the arguments: `provider`, `organization`, or `repository`
-- ALWAYS use:
- - provider: gh
- - organization: VectoDE
- - repository: VectoBeat
-- Avoid calling `git remote -v` unless really necessary
-
 ## CRITICAL: After ANY successful `edit_file` or `reapply` operation
 - YOU MUST IMMEDIATELY run the `codacy_cli_analyze` tool from Codacy's MCP Server for each file that was edited, with:
  - `rootPath`: set to the workspace path

FURTHER_DEVELOPMENT.md

@@ -37,13 +37,13 @@
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Lyrics Integration</strong> &mdash; Surface synchronised lyrics (Genius, Musixmatch) within now-playing embeds.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>DJ Permissions Layer</strong> &mdash; Add role-based queue control with an auditable history.
       </label>
     </li>
@@ -55,31 +55,31 @@
   <ul style="list-style: none; padding-left: 0;">
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Multi-Node Lavalink Failover</strong> &mdash; Provision redundant nodes with automatic reconnect and player migration.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Self-Healing Supervisors</strong> &mdash; Watch shard heartbeats and restart or reconnect when anomalies occur.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Structured Alerting</strong> &mdash; Export metrics to Prometheus/Alertmanager for proactive paging.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Chaos Testing Playbook</strong> &mdash; Run scheduled disconnect, latency, and error-injection drills.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Auto Scaling Strategy</strong> &mdash; Integrate with Kubernetes/Nomad to scale shards and nodes based on concurrency dashboards.
       </label>
     </li>
@@ -91,25 +91,25 @@
   <ul style="list-style: none; padding-left: 0;">
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Grafana Dashboards</strong> &mdash; Publish live dashboards for shard health, node stats, and slash command throughput.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Command Analytics Pipeline</strong> &mdash; Stream anonymised command events into a data warehouse for trend analysis.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Real-Time Queue Telemetry</strong> &mdash; Emit queue lifecycle events (play, skip, finish) to webhooks or WebSocket consumers.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Enhanced Slash Feedback</strong> &mdash; Display progress bars and follow-ups for long-running operations.
       </label>
     </li>
@@ -127,19 +127,19 @@
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Scenario Test Harness</strong> &mdash; Mock Lavalink responses for integration-style queue and playback validation.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Command Reference Generator</strong> &mdash; Auto-build slash command documentation from source annotations.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Local Sandbox Stack</strong> &mdash; Ship docker-compose with Lavalink, Redis, and Postgres for contributors.
       </label>
     </li>
@@ -151,19 +151,19 @@
   <ul style="list-style: none; padding-left: 0%;">
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Event Loop Profiling</strong> &mdash; Benchmark coroutine hotspots with `pyinstrument` and asyncio debug facilities.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked disabled />
         <strong>Adaptive Caching</strong> &mdash; Cache heavy search queries with TTLs to curb upstream traffic.
       </label>
     </li>
     <li>
       <label>
-        <input type="checkbox" disabled />
+        <input type="checkbox" checked  disabled />
         <strong>Dynamic Search Limits</strong> &mdash; Tune track search result counts based on latency and guild load.
       </label>
     </li>

README.md

@@ -310,6 +310,10 @@ AUTOPLAY_RANDOM_PICK=true
 <ul>
   <li>Capture <code>stdout</code> for VectoBeat; enable log shipping in production (ELK, CloudWatch, etc.).</li>
   <li>Monitor Lavalink metrics: player count, CPU, memory, frame deficit.</li>
+  <li>Import the bundled Grafana dashboards in <code>docs/grafana</code> for shard latency, node health, and slash command throughput visualisations.</li>
+  <li>Enable the command analytics pipeline (<code>docs/analytics.md</code>) to push anonymised slash usage into your data warehouse.</li>
+  <li>Wire the queue telemetry webhook (<code>docs/queue_telemetry.md</code>) into your status site for real-time “now playing” indicators.</li>
+  <li>Long-running slash commands (e.g. playlist loading) now show live progress embeds so users know what’s happening.</li>
   <li>Regularly patch yt-dlp for source compatibility.</li>
   <li>Monitor Redis availability (<code>INFO</code>/<code>PING</code>) if playlist persistence is enabled.</li>
 </ul>
@@ -321,8 +325,13 @@ AUTOPLAY_RANDOM_PICK=true
 
 <ol>
   <li>Fork the repository and clone your fork.</li>
-  <li>Create a feature branch <code>git checkout -b feature/amazing-improvement</code>.</li>
-  <li>Run <code>python3 -m compileall src</code> before committing.</li>
+<li>Create a feature branch <code>git checkout -b feature/amazing-improvement</code>.</li>
+<li>Run <code>python3 -m compileall src</code> and <code>scripts/typecheck.sh</code> before committing.</li>
+<li>(Optional) Exercise queue scenarios via <code>scripts/run_scenarios.py tests/scenarios/basic_queue.yaml</code> when touching playback logic.</li>
+  <li>Regenerate the slash-command reference via <code>python scripts/generate_command_reference.py</code> before publishing docs changes.</li>
+  <li>Spin up the local sandbox stack via <code>docker compose -f docker-compose.local.yml up</code> (see <code>docs/local_sandbox.md</code>) when you need Lavalink/Redis/Postgres locally.</li>
+  <li>Tune search caching via the <code>cache</code> section in <code>config.yml</code> to reduce repeated Lavlink lookups.</li>
+  <li>Adjust dynamic search limits via <code>search_limits</code> to balance latency vs. search breadth.</li>
   <li>Submit a pull request describing the motivation, approach, and testing performed.</li>
 </ol>
 
@@ -338,3 +347,4 @@ AUTOPLAY_RANDOM_PICK=true
 <p align="center" style="margin-top: 2rem;">
   <em>VectoBeat — engineered for premium audio experiences and operational clarity.</em>
 </p>
+- Profile the event loop with `python scripts/profile_event_loop.py` (requires pyinstrument) and inspect `profiles/event-loop-profile.html` for hotspots.

config.yml

@@ -30,6 +30,15 @@ lavalink:
   https: true
   name: "main"
   region: "eu"
+  # Deprecated single-node fallback. Prefer configuring multiple nodes via ``lavalink_nodes`` below.
+
+lavalink_nodes:
+  - host: "lava-v4.ajieblogs.eu.org"
+    port: 443
+    password: https://dsc.gg/ajidevserver
+    https: true
+    name: "main"
+    region: "eu"
 
 spotify:  # Optional: Für Spotify Support
   client_id: ""  # Setze deinen Spotify Client ID
@@ -59,3 +68,52 @@ redis:
   port: 32768
   password: ""
   db: 0
+
+metrics:
+  enabled: true
+  host: "0.0.0.0"
+  port: 9333
+  collection_interval: 15
+
+chaos:
+  enabled: false
+  interval_minutes: 360
+  scenarios:
+    - disconnect_voice
+    - disconnect_node
+    - inject_error
+  guild_allowlist: []
+
+scaling:
+  enabled: false
+  provider: "nomad"
+  endpoint: "https://nomad.example.com/v1/job/vectobeat/scale"
+  auth_token: ""
+  interval_seconds: 60
+  cooldown_seconds: 300
+  target_guilds_per_shard: 1200
+  target_players_per_node: 150
+  min_shards: 1
+  max_shards: 10
+  min_lavalink_nodes: 1
+  max_lavalink_nodes: 5
+
+analytics:
+  enabled: false
+  endpoint: ""
+  api_key: ""
+  flush_interval_seconds: 30
+  batch_size: 50
+  storage_path: "data/command_analytics.log"
+  hash_salt: "vectobeat"
+
+queue_telemetry:
+  enabled: false
+  endpoint: ""
+  api_key: ""
+  include_guild_metadata: true
+
+cache:
+  search_enabled: true
+  search_ttl_seconds: 60
+  search_max_entries: 200