Skip to content

auto-delimiter detection: fix IndexError on short files, raise a typed error#606

Merged
jepegit merged 3 commits into
masterfrom
claude/wizardly-nobel-6d881c
Jul 21, 2026
Merged

auto-delimiter detection: fix IndexError on short files, raise a typed error#606
jepegit merged 3 commits into
masterfrom
claude/wizardly-nobel-6d881c

Conversation

@jepegit

@jepegit jepegit commented Jul 21, 2026

Copy link
Copy Markdown
Owner

What

find_delimiter_and_start (cellpy/readers/instruments/base.py) failed to load short files with a bare IndexError: list index out of range — an error naming neither the file nor the cause. A short vendor file (a truncated run, a quick sanity export, a small fixture) is perfectly normal.

Root cause

The function split its sample window into a header part and a data part, but computed the data-part start from the nominal window size rather than the lines actually read:

_find_separator(checking_length_whole - checking_length_header, lines, separators)
# → last_part = lines[checking_length_whole - checking_length_header : -1]

When the file was shorter than that offset, last_part was empty and _find_separator ran list({})[0]IndexError.

Threshold (bisected, not assumed)

The real callers (AutoLoader._auto_formatter, and the same in custom.py) use checking_length_header=100, checking_length_whole=200, putting the split at line 100 — so any file under 102 lines failed. For neware_txt_zero (1 header + D data rows): D=100 → 101 lines crashes, D=101 → 102 lines works, which is exactly the reported "100 fails / 150 works" bracket. (The module default header=30 puts the boundary at 171 rows; both confirmed by bisection.)

Fix

  • _find_separator clamps the header window to the actual line count — min(checking_length_header, n-1) — always leaving ≥1 data row, and drops the possibly-truncated last line only when a data row can still be spared. For any file that already worked (n ≥ 102 at header=100), the inspected window lines[100:-1] is byte-identical → zero behavior change on existing files.
  • A header line + one data line is now enough (verified to the 2-line floor).
  • Genuine failures raise a typed LoaderError naming the file — empty/blank-only files, no consistent delimiter, or an unlocatable header — replacing both the IndexError and the old untyped IOError.

Verification

  • End-to-end through the real neware_txt loader: D = 1, 2, 5, 20, 100, 300 all load (previously 1–100 crashed).
  • New tests/test_find_delimiter_and_start.py — 35 tests: sizes 1/2/5/20/100/101/150 across four delimiters, preamble handling, the minimal 2-line file, and both typed-error paths.
  • No regressions: real neware_uio.csv parity tests and the loader_custom golden (both exercise auto-detection) stay green.

Context

Found while extending the #560 loader-port parity coverage (PR #602); split out to keep that branch scoped. This also removes the reason the synthetic-parity fixtures had to be 300 rows.

🤖 Generated with Claude Code

…d error

find_delimiter_and_start split its sample window into a header part and a
data part, but computed the data-part start from the nominal window size
(checking_length_whole - checking_length_header) rather than the number of
lines actually read. A file shorter than that offset left the data slice
empty, and _find_separator died on list({})[0] with a bare
"IndexError: list index out of range" - naming neither the file nor the cause.

With the real callers (AutoLoader: header=100, whole=200) the split sat at
line 100, so any file under 102 lines failed: neware_txt_zero with 100 data
rows crashed, 150 loaded. A short vendor file - a truncated run, a quick
sanity export, a small fixture - is perfectly normal.

_find_separator now clamps the header window to the actual line count
(min(header, n-1)), always leaving at least one data row, and drops the
possibly-truncated last line only when a data row can still be spared. For any
file that already worked (n >= 102 at header=100) the inspected window is
byte-identical, so existing files are unaffected. A header line plus a single
data row is now enough. Genuine failures - empty/blank-only files, no
consistent delimiter, an unlocatable header - raise a typed LoaderError that
names the file, replacing both the IndexError and the old untyped IOError.

Regression tests cover 1/2/5/20/100/101/150 data rows across four delimiters,
preamble handling, the minimal two-line file, and both error paths. This also
removes the reason the #560 synthetic-parity fixtures had to be 300 rows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@cursor

cursor Bot commented Jul 21, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@jepegit
jepegit enabled auto-merge (squash) July 21, 2026 07:20
@jepegit
jepegit merged commit 0c93407 into master Jul 21, 2026
4 checks passed
@jepegit
jepegit deleted the claude/wizardly-nobel-6d881c branch July 21, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant