Troubleshooting

Troubleshooting

This page covers common issues when using CompareQueries (leoswing-comparequeries-datasource) in dashboards and Grafana Alerting.

1. Quick Diagnosis Checklist

1. Confirm Grafana version and plugin version.
2. For Grafana 13+, use Target Datasource flow (not legacy RefId).
3. In dashboards, set panel datasource to -- Mixed --.
4. Verify your target datasource query works by itself first.
5. Check CompareQueries datasource auth mode:
   • Default: No Authentication
   • If backend/alerting fails auth: Basic authentication + Service Account Token
6. For alert rules, configure query directly in CompareQueries editor (not panel mixed flow).

---

2. Dashboard Issues

Symptom: "No data" in CompareQueries panel

Possible causes and fixes:

• Target query itself returns no data.
  • Run the same query directly against the target datasource first.
• Panel datasource is not -- Mixed --.
  • Switch panel datasource to -- Mixed --.
• Time range too short for your shift amount.
  • If shift is 1w, use a panel range that still contains historical points.
• Invalid shift syntax.
  • Use formats like 1d, 1w, 30m, 2h.

Symptom: Legacy RefId query stopped working after Grafana upgrade

• Grafana 13+ does not support the old third-party RefId mixed flow.
• Fix:
  1. Open CompareQueries query row.
  2. Click Migrate to Target Datasource.
  3. Select target datasource.
  4. Rebuild query inline.
  5. Save dashboard.

Symptom: Shifted lines are not aligned with current window

• Check Process TimeShift option:
  • Enabled: shifted data is moved to align with current time window.
  • Disabled: historical timestamps stay in original time.

---

3. Alerting Issues

Symptom: Alert rule preview fails with 401/403

• Backend query cannot authenticate to Grafana API.
• Fix:
  1. Open CompareQueries datasource settings.
  2. Set Authentication to Basic authentication.
  3. Set Service Account Token (Viewer or higher).
  4. Set Grafana URL only if auto-detection is incorrect.

Symptom: Alert works in panel but fails in alerting

• Panel mixed flow and backend alerting flow are different.
• In alerting, always configure CompareQueries query directly:
  • Target Datasource
  • Inline target query
  • Time shift rows

---

4. Installation and Upgrade Issues

Symptom: Plugin not visible in datasource list

• Verify plugin ID: leoswing-comparequeries-datasource.
• Restart Grafana after install/upgrade.
• Check server logs for plugin load errors.

Install via CLI:

grafana-cli plugins install leoswing-comparequeries-datasource

## Symptom: UI looks stale after upgrade

- Hard refresh browser cache.
- Restart Grafana service/container.
- Reopen panel editor.

———

## 5. Debugging Tips for Self-Hosted Grafana

Enable debug logs and inspect backend plugin logs:

docker logs -f <grafana-container-name>

Useful log keywords:

- Plugin Request Started
- Executing shifted query
- Proxying query to Grafana
- Plugin Request Completed

For local dev environments, you can use:

- GF_LOG_LEVEL=debug
- GF_LOG_FILTERS=plugin.leoswing-comparequeries-datasource:debug

———

## 6. Before Opening an Issue

Please include:

1. Grafana version
2. CompareQueries plugin version
3. Target datasource type (Prometheus/Loki/Elasticsearch/SQL/other)
4. Whether this is Dashboard or Alerting
5. Query JSON (with secrets removed)
6. Relevant Grafana logs and screenshots

Issue tracker:
https://github.com/leoswing/comparequeries-datasource-rc/issues