Skip to content

TrueQuery 0.1.6

Choose a tag to compare

@chandrahas-reddy chandrahas-reddy released this 04 Nov 22:04
2ef98f7

TrueQuery 0.1.6: pandas pipelines are checked too.

A Python file that imports pandas is now read for pandas code that runs without an error and gives the wrong data. Same promises as the SQL checker: offline, deterministic, no AI, and pandas is never imported or run.

On by default:

  • Writes that never land (error, reported right after the edit): df[mask]["col"] = v, df["col"][i] = v, df["col"].fillna(0, inplace=True). The write goes to a temporary copy, and the DataFrame never changes.
  • Row-loop writes that are thrown away (error): row["col"] = v inside for i, row in df.iterrows(). The row is a copy.
  • groupby(...).first() / .last() mixing rows (warning): each column takes its own first non-null value, so one output row can combine different input rows.
  • A dedupe after an unstable sort (warning): sort_values("ts").drop_duplicates("id", keep="last") keeps an arbitrary row when timestamps tie.
  • to_sql(if_exists="replace") (warning): the table is dropped and recreated with types pandas guesses, losing its keys, constraints and indexes.

Off by default, available with truequery rules set: a merge with no validate= (a duplicate key multiplies rows), and a groupby with no dropna= (rows with a missing key disappear).

Measured before switching anything on. On 13 public pandas projects (PUDL, Our World in Data, Google's COVID-19 Open Data, PyPSA-Eur, freqtrade, Qlib and more), the 5 rules on by default reported 25 findings: 1 false alarm and 1 real bug (a fill that lands on a copy). The merge check, off by default, found a second real bug. Every rule comes with a proof: a small DataFrame where the wrong code gives the wrong answer, run in the tests.

The receipt says what was covered: pandas in N Python files, and that only DataFrames a file builds or annotates itself are followed.

Heads-up: Python files that import pandas are now checked, and may report findings (exit 1).

Update

claude plugin update truequery@analytically     # Claude Code
muse plugins marketplace update analytically && muse plugins update truequery   # Muse Code
pip install -U truequery                        # the checker on its own

pre-commit and the GitHub Action: use rev: v0.1.6 / @v0.1.6.

CHANGELOG · Evidence and limits · Found a false alarm? Report it here.