Correctly load chunks with larger headers

[nvoxland]

🚀 🚀 Pull Request

Impact

• Bug fix (non-breaking change which fixes expected existing functionality)
• Enhancement/New feature (adds functionality without impacting existing logic)
• Breaking change (fix or feature that would cause existing functionality to change)

Description

If the header of a chunk is larger than 100 bytes, a ValueError: buffer size must be a multiple of element size error was thrown because we only read the first 100 bytes of the chunk and assume the header will fit in it.

For performance reasons, this still assumes 100 bytes is all that will be needed, but will now get more data if needed.

Things to be aware of

I couldn't find an better way to test the logic than something more like an end-to-end test

Things to worry about

Nothing

[codecov]

Codecov Report

Patch coverage: 78.57% and project coverage change: -1.92% ⚠️

Comparison is base (c4849ca) 83.83% compared to head (98e66a5) 81.91%.
Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2574      +/-   ##
==========================================
- Coverage   83.83%   81.91%   -1.92%     
==========================================
  Files         224      224              
  Lines       24393    24403      +10     
==========================================
- Hits        20450    19990     -460     
- Misses       3943     4413     +470     

Flag	Coverage Δ
unittests	81.91% <78.57%> (-1.92%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
deeplake/core/dataset/dataset.py	92.56% <ø> (-0.23%)	⬇️
deeplake/core/vectorstore/deeplake_vectorstore.py	97.50% <ø> (ø)
deeplake/util/version_control.py	96.73% <ø> (ø)
deeplake/core/serialize.py	96.93% <72.72%> (-0.86%)	⬇️
deeplake/enterprise/convert_to_libdeeplake.py	60.57% <100.00%> (-0.97%)	⬇️
deeplake/enterprise/libdeeplake_query.py	97.14% <100.00%> (ø)

... and 20 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

50.0% Coverage
0.0% Duplication