Production Squad Runbook

Stay calm. If you are Production Squad and facing a production issue, use this high-scannability guide to diagnose, isolate, and mitigate the failure step-by-step.

Important

Informing users and managing customer awareness is our top priority. If customers are experiencing an outage, slowness, or degradation, declare an incident immediately, even if you haven't identified the root cause yet. Follow the Incident Management Protocol. It is always better to over-communicate or declare an incident in error than to remain silent. Think of Perceived Reliability and customer impact first.

---

Table of Contents

• Production Squad Runbook

  • 🚦 Triage Matrix: Where Do I Begin?

  • 🧭 Initial Diagnostic Flows

    • 1. Total Outage or Maintenance Mode
    • 2. Service Degradation (Slowness)
    • 3. Grafana Alerts About Performance
    • 4. Application Errors / Errbit Spikes

  • 🛠️ Common Incidents, Diagnostics & Solutions

    • 🚨 Ruby Gem Version Conflicts
    • 🚨 Severe Performance Degradation
    • 🚨 Sphinx Process Terminated (Search Down)

  • ➕ Contributing: Adding New Incidents

---

🚦 Triage Matrix: Where Do I Begin?

Identify your primary symptom below to jump directly to its action path:

1. Is build.opensuse.org completely down or in Maintenance Mode?
2. Is the site up but running extremely slow (Service Degradation)?
3. Has the team Slack channel received alerts about performance?
4. Are users reporting specific errors, or are exceptions spiking on Errbit?

---

🧭 Initial Diagnostic Flows

1. Total Outage or Maintenance Mode

• Step 1 (Max Priority): Declare an incident immediately by following the Incident Management Protocol. Inform the team and users first, troubleshoot second.
• Step 2: Log into the machine to evaluate basic system state and inspect the Passenger logs:

  ssh obs
  
  # Check system uptime to identify recent reboots
  uptime
  
  # Live-inspect Passenger/Apache error logs
  less /var/log/apache2/passenger_log

2. Service Degradation (Slowness)

• Step 1 (Max Priority): Whether you have noticed the slowness yourself or observed a performance spike on the APM Overview Grafana dashboard, your first action must be to alert our users that an issue is occurring and their experience might be affected. Follow the Incident Management Protocol.
• Step 2: Use Poor Man's Analytics tools detailed in the build.opensuse.org Wiki to detect long-running requests (verify the duration field).

3. Grafana Alerts About Performance

• Step 1 (Max Priority): Follow the Incident Management Protocol to tell the users that their experience might be affected.
• Step 2: Analyze the Grafana alert on Slack. Check if it targets Backend Performance or Passenger instance capacity. (Note: All Grafana timestamps are in UTC).
• ⚠️ False Positives: Ignore Grafana alerts stating Datasource Error. These indicate monitoring infrastructure issues (Influxdb/Grafana), not an application outage.
• Step 3: Identify high-volume or malicious traffic patterns using the Poor Man's Analytics tools detailed in the build.opensuse.org Wiki. Use those scripts to isolate:
  • Spikes from a single IP/Host or User within a short time frame.
  • Excessive hits hitting the exact same Controller endpoint.

4. Application Errors / Errbit Spikes

• Step 1: Open Errbit and analyze the stack backtrace to isolate code-level bugs.
• Step 2: Cross-reference the timeline with the latest production release commits via the OBS GitHub Deployments Dashboard.

---

🛠️ Common Incidents, Diagnostics & Solutions

🚨 Ruby Gem Version Conflicts

• Symptom: App drops into Maintenance Mode immediately following a deployment. Passenger fails to initialize.
• Root Cause: Incompatibility between system-installed gems and the OBS Gemfile (Ruby prohibits loading two distinct versions of the same gem within a single process).
• Mitigation (Choose one):
  • Option A (Rollback): Revert to the previous stable deployment commit and lock (pin) the stable gem version in the Gemfile.
  • Option B (System Upgrade): Explicitly update the system-level gem package via zypper:

    zypper in ruby3.4-rubygem-webrick-1.9.2

---

🚨 Severe Performance Degradation

• Symptom: Slack alerts trigger for Backend performance. System experiences high load or potential DoS (one single attacker) or DDos (many distributed attackers).
• Diagnostic: Locate and read the active or compressed historical production logs:

  # Current log
  less api/log/production.log
  
  # Compressed historical log (example date)
  xzless api/log/production.log-20251128.xz

• Solution:
  • Coordinate with the Buildops team to execute firewall IP blocks.
  • If traffic is internal, contact the responsible OBS users to throttle or optimize their automated tasks.

---

🚨 Sphinx Process Terminated (Search Down)

• Symptom: Application search functionality fails completely.
• Diagnostic: Query journalctl to verify if the process was terminated:

  journalctl -u obs-sphinx.service --since "1 day ago"
  # Look for: "obs-sphinx.service: Killing process XXXXX with signal SIGKILL"

• Solution: Restart the systemd unit:

  systemctl restart obs-sphinx.service

---

➕ Contributing: Adding New Incidents

To add a new incident to this runbook, please follow the established formatting structure to maintain scannability during an outage. Use the 🚨 Incident Name and clearly define the Symptom, Root Cause, and Mitigation/Solution using a clean bulleted list. If applicable, wrap any diagnostic or recovery commands. Don't forget to update the Triage Matrix at the top of the page if the new incident introduces a major new symptom category.