Green ruff lint (supersedes #31): databento start_date + ruff-clean#32
Merged
Conversation
Every code path ignored it: a cold cache always backfilled from 2000-01-01 (clamped to the GLBX.MDP3 floor 2010-06-06), and a warm cache always resumed from last_date+1, regardless of what was passed. A caller trying to bound a fetch (e.g. 'just the last 3 months') silently got a full-history pull instead, at full Databento API cost, with no error or warning. Checked every caller before deciding how to fix it. grep across the workspace shows no one relies on it working: the function's only real internal caller (update_all_daily_prices) never passes it, both tests never pass it, and every OTHER fetch_daily_ohlc in the tree (npf.ml.labels) is an unrelated function that correctly forwards its own start_date into cotdata.get_prices — not this one. Removed rather than wired in. An explicit TypeError for a future caller beats another silent full-history surprise, and there is no current caller to write a correctness test against if the parameter were implemented instead. A genuinely bounded fetch, if ever needed, should be a new function with its own test proving the bound is honored. Found while building a bounded, one-off Databento divergence-check script in livebook, which deliberately avoided reusing this function for exactly this reason. 81 tests pass (5 apparent failures in test_norgate_provider.py are pre-existing and environment-dependent — COTDATA_STORE unset — confirmed via git stash, unrelated to this change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Correctly pointed out that removal just relocates the problem — a future caller (including my own livebook divergence-check script, which had to write a separate bounded fetch specifically because this function couldn't do it) still needs a bounded fetch, and the test infrastructure to verify it correctly already exists in this file (the mock Historical client pattern). Cold cache (symbol's first-ever fetch): start_date now narrows the fetch floor (still clamped to the GLBX.MDP3 2010-06-06 floor), so a narrow first-time query is actually cheap — this is the real fix for the original API-cost bug. Warm cache (symbol already has history): start_date deliberately does NOT narrow the incremental fetch, which always resumes from the cache's own last_date+1. The cache is a single shared, from-inception series across every caller; letting a later, narrower start_date shrink what gets fetched would silently truncate a cache other callers rely on being complete. start_date still filters what's RETURNED in this case, just not what's fetched or persisted to disk. Both branches are tested against the mock Historical client, including the warm-cache case specifically (pre-seeded 3-day cache, narrower start_date requested, asserts the fetch call still starts from last_date+1 while the returned frame excludes the pre-start row AND the on-disk cache keeps the full series). 6 new tests, 8/8 in this file, 86/86 full suite (COTDATA_STORE set). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds ruff configuration to pyproject.toml and a linting step to CI workflow before tests. Configuration: line-length=100, select=[E,F,W,I], target-version=py39. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The 'Add ruff linting to CI' step (on this databento branch) ran but the code had 137 findings. Pinned ruff==0.15.22 and cleared them: whitespace, import sorting, unused imports, f-strings, and E713 not-in (all safe auto-fixes). The 3 remaining E402 are test imports after sys.modules mock injection (databento/norgate) — those must stay below the mock, so noqa'd at the line; one stray datetime import moved to the top. ruff check src tests clean; all 86 tests pass (with COTDATA_STORE set). Config unchanged (fleet parity: select E/F/W/I, ignore E501). Includes this branch's databento start_date work (fbc808a, c87279e). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #31 (its branch is checked out by another session, so this is a fresh branch off the same tip). Carries #31's databento
start_datework plus the ruff cleanup that greens the lint step: pinned ruff==0.15.22, cleared 137 findings (whitespace/imports/f-strings/E713 auto-fixes; 3 test imports after sys.modules mock injection noqa'd; one datetime import moved up). ruff clean, 86 tests pass. Config unchanged (fleet parity).Once merged, #31 can be closed.
🤖 Generated with Claude Code