Date Math Test Harness

Does an LLM actually get date arithmetic right, or does it just sound confident either way?

This is a small Node script that sends randomized, freshly-generated date-math questions to several LLM APIs — one isolated request per question, no shared context — computes the correct answer independently in code, and grades the raw response against it. It's built to answer one narrow question honestly, with receipts: real questions, real API calls, real pass/fail, not a curated list of "gotcha" examples.

Background: [https://dev.to/maverickyadav/-llms-cant-reliably-do-date-math-and-now-theres-data-4hm2]

Project layout

date-math-bench/
├── .env.example
├── .gitignore
├── LICENSE
├── README.md
├── package.json
├── dates.js       # date helpers + seedable RNG (pure, no I/O)
├── grading.js     # pass/fail checker functions (pure, no I/O)
├── run.js         # test generation, API calls, reporting — the entrypoint
├── test/
│   └── grading.test.js   # selftest for grading.js
└── results/       # archived per-run output (committed on purpose — see Output below)

Kept flat and single-purpose on principle — this only tests date arithmetic. If a broader scope shows up later (arithmetic, unit conversions, etc.) that's a signal to either split into a src/ structure with per-category files, or spin up a sibling repo, not a reason to restructure preemptively now.

Why this exists

Date arithmetic looks like the safest possible thing to hand an LLM — no ambiguity, no judgment call, just counting. That's what makes errors in it dangerous: a wrong answer reads exactly as confident as a right one. This harness exists to actually measure that, per model, instead of relying on a couple of examples caught in casual conversation.

Models tested

Model	Provider	Env var
Claude Sonnet 4.6	Anthropic	ANTHROPIC_API_KEY
Claude Haiku	Anthropic	ANTHROPIC_API_KEY
GPT-4o-mini	OpenAI	OPENAI_API_KEY
Llama 3.3 70B	Groq	GROQ_API_KEY
Qwen3-27B	Groq (qwen/qwen3.6-27b)	GROQ_API_KEY

Only models with a key set will run — you don't need all five. Missing keys are skipped with a console message, not an error.

Heads up on Groq model availability: Groq deprecated llama-3.3-70b-versatile and the old qwen/qwen3-32b in mid-2026. If a call errors out with a deprecation or model-not-found message, override the slug without touching code:

LLAMA_MODEL="openai/gpt-oss-120b" npm test

Check console.groq.com/docs/models for the current lineup if this happens again later.

Setup

git clone <this-repo>
cd date-math-bench
npm install
cp .env.example .env   # then fill in whichever keys you have
export $(cat .env | grep -v '^#' | xargs)   # or just export them directly in your shell
npm test

Every model without a key gets skipped automatically — running with just one key set works fine, it just tests one model instead of five.

Before trusting any real results, it's worth running the grading selftest once:

npm run test:grading

This checks the pass/fail logic itself against hand-picked known-answer cases (30 of them), independent of any API call. A bug in a checker would silently corrupt every number this tool reports without ever showing up as an error, so this is the one part of the repo that gets its own tests.

Reproducibility

Every run picks a seed (random unless SEED is set) and prints it at the top of the output. To reproduce the exact question set from a specific run later:

SEED=<value printed at the top of that run> npm test

Useful if you want to cite specific numbers somewhere and let someone else regenerate the exact same questions to check them.

Options

• TRIALS — trials per category, defaults to 8 (11 categories × 8 ≈ 88 questions per model). With all 5 models that's ~440 requests total — check your provider rate limits first. Use TRIALS=3 for a quick sanity-check run, and the default or higher for numbers you actually intend to report.
• SEED — see Reproducibility above.
• SONNET_MODEL, HAIKU_MODEL, GPT_MODEL, LLAMA_MODEL, QWEN_MODEL — override any model slug, e.g. if a provider has changed their lineup since this was written.

What it tests

Eleven categories, chosen to cover three distinct underlying failure shapes rather than eleven variations on one bug:

Counting errors — the model reaches for a memorized constant instead of computing the specific case:

1. Ordinal day count — elapsed days vs. "day N of a run" (off-by-one on an inclusive start)
2. Leap year span — Feb 1 → Mar 1 day count depends on whether the year is a leap year
3. Century leap-year rule — the /100-but-not-/400 exception (1900 and 2100 are NOT leap years; 2000 and 2400 ARE)
4. Age calculation — age as of a given date, correctly handling a birthday that hasn't occurred yet that year

Directional errors — the magnitude is right, the direction isn't: 2. Day-of-week increment — computing tomorrow's date/weekday 6. Direction (days ago) — subtracting vs. adding, sign errors 8. Backward month arithmetic — subtracting months, including year-boundary wraparound

Category errors — one kind of date operation gets silently treated as a simpler one it resembles: 4. Month-end rollover — "one month after Jan 31" must land on a real calendar date, not "February 31" 5. Business days — business-day counts must skip weekends, not just add calendar days 7. Nth weekday of month — "the third Thursday of November" style date resolution 11. Feb 29 birthday recurrence — a leap-day birthday landing in a non-leap year has no single correct celebration date; grading accepts either real-world convention (Feb 28 or Mar 1) and only fails on an actually invalid date

Output

Console output streams each question, every model's raw answer, and a pass/fail/unparseable tag as it runs (rate-limited requests wait and retry automatically rather than failing), followed by a key/auth-error summary if any keys need attention, a per-category matrix (pass/total per model), an overall summary, and every real failure printed verbatim — question, expected answer, actual response.

Every run writes results two places:

• results.json — scratch copy, always overwritten, gitignored. Fine for quick local checks.
• results/<timestamp>_seed<seed>_trials<N>.json — a permanent, timestamped copy of that exact run. Not gitignored — the intent is to commit the runs whose numbers actually get cited somewhere (like a blog post), so anyone can check them later, and reproduce the exact same question set with SEED=<value from filename> npm test.

Notes on grading

Most categories have exactly one correct answer and are graded by exact/substring match. Two categories are graded on validity rather than a single expected string, because the underlying question genuinely doesn't have one right answer:

• Month-end rollover accepts any date that's a real calendar date in the target month; it only fails on an impossible date like "February 31."
• Feb 29 birthday recurrence accepts either Feb 28 or Mar 1 as a valid celebration date, and only fails if the model outputs Feb 29 for a year that doesn't have one.

This keeps the harness from penalizing a model for picking a defensible convention while still catching outright invalid dates.

License

MIT — see LICENSE.