Transaction timeout during checksum computation for large filepath fields

[dimitri-yatsenko]

Problem

In three-part make() methods, the transaction can expire while checksums are computed for large files with filepath fields. This causes the entire make() to fail, not just a performance degradation.

The checksum computation for large files takes long enough that the database transaction times out before the insert can complete.

Context

This affects the 0.14.x branch. The issue occurs when:

• Working with large files (videos, imaging data, etc.)
• Using filepath attributes that reference these large files
• The checksum computation happens within the transaction
• Computation time exceeds the transaction timeout

Impact

• make() fails entirely for large files
• Data that was already computed is lost
• Users cannot reliably use filepath fields with large files

Branch

maint/0.14

Note

This will be addressed differently in DataJoint 2.0+.

[dimitri-yatsenko]

Fixed in v0.14.8 with the new filepath_checksum_size_limit_insert config option. See PR #1387.