Skip to content

One-by-one file insert to avoid transaction timeout when inserting large files#226

Merged
kushalbakshi merged 5 commits into
datajoint:main_external-storagefrom
ttngu207:main_external-storage
Nov 5, 2025
Merged

One-by-one file insert to avoid transaction timeout when inserting large files#226
kushalbakshi merged 5 commits into
datajoint:main_external-storagefrom
ttngu207:main_external-storage

Conversation

@ttngu207
Copy link
Copy Markdown
Contributor

@ttngu207 ttngu207 commented Nov 5, 2025

This pull request refactors how result files are inserted into the database within the element_array_ephys/spike_sorting/si_spike_sorting.py module. Instead of batch-inserting file records, the code now inserts each file individually as it is discovered. This change enhances clarity and may improve error handling or logging for individual file insertions.

Refactoring of file insertion logic:

  • Changed from batch insertion (self.File.insert([...])) to per-file insertion (self.File.insert1(...)) for result files in the _run_sorter method.
  • Applied the same per-file insertion pattern in the _sorting_analyzer_compute method for files in the analyzer output directory.
  • Updated the _export_report method to insert report files individually for each file found in the relevant report directories.

@ttngu207 ttngu207 requested a review from kushalbakshi November 5, 2025 14:44
@kushalbakshi kushalbakshi merged commit 481f77b into datajoint:main_external-storage Nov 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants