CSV error reporting: logical record number or physical line range for multiline fields?

[renfengxing]

🏷️ Discussion Type

Question

Body

I would like feedback on an error-location design in a small Python standard-library CSV merge tool.

The parser accepts quoted multiline fields. For a field-count error it currently reports the source filename, logical record number (header is record 1), and the physical line where that record ends (csv.reader.line_num). A generated example produces:

{"status": "error", "error": "synthetic-invalid.csv: record 2, line 2: field count mismatch", "synthetic_demo": true}

For a multiline record, the physical line can differ from the record number. Would a start/end physical-line range be more useful when fixing an export in an editor, or would it create more confusion than a logical record number? A small synthetic example where the current location would be misleading would help.

The implementation is here, at the published commit; the local demo and format rules give the surrounding context. It validates UTF-8/comma-separated inputs with identical ordered headers and stops on bad input rather than writing a partial CSV. Whitespace is preserved by default; trimming is opt-in. The demo is synthetic, including the four-to-three-record trimming example.

Please use invented rows only, with no customer files, personal data or confidential information.

AI assistance: the code and drafting used Codex (GPT-6). This single post is prepared and submitted by an owner-authorized Codex assistant; the implementation details and demo behavior above were checked against the linked published source and recorded local execution results. The purpose is feedback on this specific reporting tradeoff.

Resubmitted using this category's web form and template after the automated format feedback on the original closed discussion.

Guidelines

• I have read and understood this category's guidelines before making this post.

[hiteshx2026]

I'd recommend reporting both: keep the logical record number and add a start-end physical line range. When the record occupies a single physical line, simply display line N instead of a range.

csv.reader.line_num reports the number of physical lines consumed so far. For multiline records, this points to the line where the record ends. While that is useful, reporting only the ending line can make it difficult to locate the actual problem.

Consider this synthetic example:

id,note
1,"first entry
2,second entry
3,third entry

The quoted field is never closed, so the reader may consume everything through EOF as part of one record. An error reported at line 4 would send someone to 3,third entry, which looks perfectly normal on its own. The actual problem is the opening quote on line 2.

Reporting record 2, lines 2-4 provides the full context and makes the issue easier to investigate.

A second example is a multiline record with an incorrect field count:

id,note,tag
1,ok,a
2,"line one
line two",b,extra
3,ok,c

Here, the third logical record occupies physical lines 3-4 but contains four fields instead of the expected three.

An error such as:

synthetic-invalid.csv: record 3, lines 3-4: field count mismatch

is more informative than simply reporting record 3, line 4.

Why I would keep both:

• The logical record number identifies the record's position in the CSV data.
• The physical line range helps users locate the problematic text directly in editors, terminal output, or Git diffs.
• Keeping both avoids forcing users to translate between CSV records and physical lines.
• Using line N for single-line records keeps the output concise.

Implementation considerations:

1. Capture the starting physical line before reading each record and the ending line immediately after reading it. For example:

start_line = reader.line_num + 1

try:
    row = next(reader)
except StopIteration:
    break

end_line = reader.line_num

2. Open the file with newline='' when using Python's CSV module, as recommended by the standard library.

3. Consider including structured information in the JSON error response, such as record, line_start, line_end, expected_fields, and got_fields. This allows other tools to consume the error without parsing its message.

4. Handle malformed CSV quoting separately from field-count mismatches. In particular, an unterminated quoted field should ideally produce an error such as unterminated quoted field starting at line 2, rather than suggesting that the only problem is an incorrect field count.

Overall, I think adding the physical line range is worth the small increase in message length. Keeping the logical record number alongside it provides useful context while making multiline CSV errors significantly easier to debug.

[renfengxing]

Thanks — I reproduced both synthetic cases with Python’s strict CSV reader. They support keeping the logical record number and adding the physical line span; a single ending line is ambiguous.

The current preview counts the header as record 1 and reports the ending physical line. I haven’t changed it yet. Your examples are useful regression cases for a future diagnostic update, including checking blank-line handling and malformed quotes.

This reply was prepared with AI assistance; the two examples were checked locally.

[hiteshx2026]

Glad the examples were useful. For the follow-up cases:

• Header counting: since the preview counts the header as record 1, I'd state that convention in the message or docs (e.g. "record 3, including header"), so it doesn't differ from what pandas or a spreadsheet shows.
• Blank lines: csv.reader returns [] for a blank line and still advances line_num. A blank line before the failing record shifts the physical line numbers without being a record, so it's worth a regression case.
• Malformed quotes: a stray quote inside an unquoted field (e.g. 2,ab"c,d) behaves differently in strict and non-strict mode. A quote that opens on one line and closes several lines later, swallowing valid rows, is also worth a case, and the error should name the line where the quote opened.

If this answers your question, please consider marking it as the answer so others can find it. Happy to help with more cases.

[sumiEmpire001]

I'd recommend reporting both the logical record number and the physical line range.

The logical record number helps identify which CSV record failed, while the physical line range helps someone locate the actual text in a source file. This is especially useful when quoted fields contain embedded newlines.

One implementation detail worth considering is that "csv.reader.line_num" counts physical lines consumed, not logical records.

You can capture the start and end line numbers like this:

import csv
import io
data = '''id,note
1,"first line
second line"
2,normal
'''
reader = csv.reader(io.StringIO(data))
previous_end = 0

for record_number, row in enumerate(reader):

    start_line = previous_end + 1
    end_line = reader.line_num
    previous_end = end_line

    print({
        "record": record_number,
        "start_line": start_line,
        "end_line": end_line,
        "fields": len(row)
     #})

For this example, the output is:

{'record': 0, 'start_line': 1, 'end_line': 1, 'fields': 2}
{'record': 1, 'start_line': 2, 'end_line': 3, 'fields': 2}
{'record': 2, 'start_line': 4, 'end_line': 4, 'fields': 2}

For a field-count mismatch, I'd suggest an error format such as:

file.csv: record 1, lines 2-3:
expected 3 fields, found 2

This assumes zero-based record numbering, with the header as record 0. If your application uses one-based numbering, adjust the labels accordingly.

One important caveat: this approach works for successfully parsed records, including records with incorrect field counts. For malformed quoting that causes "csv.Error", additional care is needed because the parser may have consumed multiple physical lines before failing.

My recommendation: Keep the logical record number as the primary identifier, add the physical line range as supplementary location information, and document the numbering convention. This would improve debugging without breaking the existing mental model for users.