CI failure

[Aryan-Gore]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

General

Discussion Details

Hi,

I’m currently facing issues with failing CI checks on my PR and I’m unable to identify the exact cause.

Failing checks:

• codecov/project/tests — coverage decreased slightly (97.71%, -0.10%)
• matplotlib.matplotlib — test failures
• Windows (Python 3.12) — pytest failed
• macOS (Python 3.11, 3.12) — tests failing
• Ubuntu (Minimum Versions) — tests failing

What I’ve tried:

• Ran pre-commit run --all-files
• Ran pytest locally (tests pass on my system)
• Fixed ruff and mypy issues

However, I’m unable to reproduce these CI failures locally.

Request:

Could you please guide me on:

• How to debug these cross-platform test failures?
• Or point out what I might be missing?

Thanks for your time!

[sinnerfilozofiya]

🧪 Debugging CI Failures That Don’t Reproduce Locally

This situation is very common: everything passes locally, but CI fails across multiple environments. The key is understanding that CI != your local machine.

Let’s break this down properly.

---

🔍 What’s Likely Happening

Your failures across:

• Windows
• macOS
• Ubuntu (minimum versions)

→ strongly suggest environment differences, not random failures.

---

🧠 1. Always Start With CI Logs (Most Important Step)

Before guessing, open one failing job and:

• Expand the pytest output بالكامل
• Look for the first actual error, not the summary
• Ignore cascade failures — focus on the root cause

💡 Tip: GitHub collapses logs — expand everything.

---

🎨 2. Matplotlib Issues (Very Common in CI)

CI environments are headless (no GUI), so rendering fails.

Fix it globally:

import matplotlib
matplotlib.use("Agg")

Or set environment variable in your workflow:

MPLBACKEND=Agg

Also consider:

import matplotlib.pyplot as plt
plt.switch_backend("Agg")

---

🪟 3. Cross-Platform Issues (Windows/macOS/Linux)

Path Problems

Avoid hardcoded paths:

❌ "data/file.txt"
✅ use pathlib:

from pathlib import Path
Path("data") / "file.txt"

---

Line Endings (CRLF vs LF)

Windows uses CRLF, Linux/macOS use LF.

If you compare strings/files:

• Normalize:

text.replace("\r\n", "\n")

---

Case Sensitivity

Linux is case-sensitive, Windows is not:

❌ "File.txt" vs "file.txt"
→ passes locally, fails in CI

---

📦 4. Minimum Versions Job (Very Important)

This is often the real issue.

CI installs the lowest allowed dependency versions, while locally you use latest.

Reproduce locally:

pip install -e .[test] --upgrade --upgrade-strategy=only-if-needed

Or even better:

pip install "numpy==<min_version>" "matplotlib==<min_version>"

💡 Check CI logs to see exact versions installed.

---

🧪 5. Hidden State / Environment Differences

CI is clean. Your local machine is not.

Common issues:

• Missing environment variables
• Cached files
• Pre-existing data
• Different working directory

Debug tip:

Print environment inside tests:

import os
print(os.getcwd())
print(os.environ)

---

🧵 6. Parallelism Issues

CI may run tests in parallel.

If tests depend on shared state:

• temp files
• global variables
• static ports

→ they may fail randomly

---

🐳 7. Reproduce CI Locally (Best Technique)

Use act for Linux jobs:

act

This runs your GitHub Actions workflow locally using Docker.

---

📎 8. Upload Logs as Artifacts

Modify your workflow to upload full test logs:

- name: Upload logs
  uses: actions/upload-artifact@v4
  with:
    name: pytest-logs
    path: logs/

This makes debugging MUCH easier.

---

📉 9. Coverage Drop (Codecov)

0.10% drop is negligible.

Possible reasons:

• Platform-specific skipped test
• Conditional logic not triggered in CI

Temporary workaround:

fail_ci_if_error: false

---

🧭 Debug Strategy (TL;DR)

1. Check full CI logs (root error)
2. Fix matplotlib backend (Agg)
3. Verify cross-platform compatibility
4. Reproduce minimum dependency versions
5. Ensure clean environment assumptions
6. Use act for local CI simulation

---

✅ Final Thought

If tests pass locally but fail everywhere in CI, it's almost always:
→ environment mismatch
→ dependency versions
→ OS-specific behavior

---

If you can share one full traceback from CI, the exact issue can usually be pinpointed very quickly.

[alex-gamero]

Hey, just a heads up, the goal here is to help the person asking, not add more noise to the thread. A response that feels copy-pasted or robotic can actually make things worse for them. Keep it human and to the point next time.

[alex-gamero]

IklilIlm's answer covers most of it.
One thing I'd add, though, when tests pass locally but fail on minimum versions specifically, it's often a type annotation or function signature that changed between versions rather than logic. Worth checking if any of your dependencies introduced a breaking change in their API between the minimum version and what you have locally.

Also for the matplotlib failures on macOS specifically, sometimes just setting MPLBACKEND=Agg in the workflow env isn't enough if the backend gets imported before the env variable is read. Setting it explicitly at the top of conftest.py is more reliable.

If you can paste the actual traceback from one of the macOS failures that'd make it a lot easier to pinpoint.

[Bharath-970]

This is a pretty common CI vs local mismatch issue.

If it passes locally but fails across multiple CI environments (Windows/macOS/Linux), it’s usually one of these:

• dependency versions differ (CI may install slightly different versions)
• platform-specific behavior (especially file paths, encodings, or OS-specific logic)
• missing system-level dependencies in CI
• tests relying on environment state (timezones, locale, temp paths, etc.)

A few things that might help narrow it down:

1. Check the exact error logs in CI for one failing job (they can differ across OS)
2. Compare dependency versions:
   • pip freeze locally vs CI logs
3. Try running tests in a clean environment locally:
   • use a fresh venv or container
4. If possible, replicate using something like:
   • tox or nox for multi-environment testing

CI logs tend to bury the actual error, so focusing on the first real failure in one job usually helps a lot.

I’ve been working on a small tool to extract the actual error + suggest fixes from CI logs — happy to share if useful.

[pauldev-hub]

Hi @Aryan-Gore ,

Your issue is classic "works locally, fails on CI" — here's exactly what to fix for each failure.

1. matplotlib test failures (all platforms)
   This is your main culprit. Matplotlib tests fail on CI because there's no display server in headless environments. Add this at the top of your test file or conftest.py:

import matplotlib
matplotlib.use('Agg') # Non-interactive backend

Or set it via environment variable in your workflow YAML:

env:
MPLBACKEND: Agg

2. Windows (Python 3.12) + macOS (3.11, 3.12) pytest failures
   This is almost always a dependency version mismatch. Your local env has different versions than CI.
   Fix it by pinning your deps:

pip freeze > requirements-ci.txt
Then in your workflow:

• run: pip install -r requirements-ci.txt

Also check if you're using any f-strings or syntax that behaves differently on Python 3.12 — it removed some deprecated features.

3. Ubuntu Minimum Versions failing
   Your code likely uses a method/feature not available in the minimum supported version of a dependency. Run this locally to simulate it:
   pip install -c constraints-minimum.txt -e .
   pytest

If no constraints file exists, manually install the oldest versions listed in your setup.cfg or pyproject.toml.

4. Codecov -0.10% coverage drop
   Your new code has no tests written for it. Find the uncovered lines:
   pytest --cov=your_package --cov-report=term-missing
   Add tests for whatever shows as missing.
   Root cause summary — tests pass locally because:
   Your local machine has a display server (matplotlib works)
   Your local Python/package versions differ from CI matrix
   CI tests against minimum versions you don't have locally

👍 If this helped you, please mark it as the answer — it helps others in the community who run into the same issue find the solution faster!

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬