Fix: Batch processing now loads all rows from file instead of being limited to 200

[Copilot]

Problem

When users imported files with 2,500+ rows for batch processing (single label classification, multi-label classification, or keyword extraction), only 200 batch requests were being sent to the OpenAI API instead of the full dataset. This significantly limited the utility of the batch processing features for large datasets.

Root Cause

The issue was in the import_data() function in file_handling/data_import.py. The function has two phases:

1. Preview Phase: When a file is selected, _refresh_preview() loads only the first 200 rows for UI performance and caches them in _loaded_rows
2. Import Phase: When the user clicks "Import", _do_import() was reusing the cached _loaded_rows instead of reloading the full file

This meant that even though the file contained thousands of rows, only the 200 preview rows were actually imported and used for batch processing.

Solution

Modified _do_import() to always reload the full file without the max_rows limit, regardless of whether preview data exists in the cache. This ensures:

• Preview still loads only 200 rows for fast UI rendering (unchanged)
• Import always loads ALL rows from the file (fixed)
• Batch processing receives the complete dataset (fixed)

Before:

if not _loaded_rows:
    try:
        rows = _load_tabular(path)
    except Exception as e:
        messagebox.showerror("Load error", str(e), parent=dlg)
        return
else:
    rows = _loaded_rows  # BUG: Only 200 rows from preview

After:

# Always load the full file without row limit for import
# (preview may have loaded only 200 rows)
try:
    rows = _load_tabular(path)
except Exception as e:
    messagebox.showerror("Load error", str(e), parent=dlg)
    return

Testing

Validated with files containing 50, 200, 201, 300, 2,500, and 5,000 rows:

• All rows are correctly imported regardless of file size
• Batch processing generates the correct number of API requests
• Preview performance is maintained (still limited to 200 rows for UI)
• Edge case at 201 rows confirmed fixed (the 201st row would have been lost before this fix)

Impact

This fix affects all features that use import_data():

• ✅ Batch Processing: Single label, multi-label, and keyword extraction
• ✅ Live Processing: All processing modes

Users can now process datasets of any size without artificial limitations.

Original prompt

Even when my users are inputting files with 2,500 rows, the software is only sending batches of 200 requests rather than the full amount when using the batch features (single label classification for example). Identify and solve this issue.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.