Fix memory blow up when transforming links

[FayazRahman]

🚀 🚀 Pull Request

Impact

• Bug fix (non-breaking change which fixes expected existing functionality)
• Enhancement/New feature (adds functionality without impacting existing logic)
• Breaking change (fix or feature that would cause existing functionality to change)

Description

• When ingesting links, we are able to hold many of them in memory
• But if verify=True, it will blow up memory when all of it is flushed at the same time
• Verify linked samples in transform (iff verify=True) and use the size of the actual samples to determine when to flush
• Generic code cleanups

Things to be aware of

Things to worry about

Additional Context

[codecov]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (8693b18) 83.79% compared to head (bf478f4) 83.77%.
Report is 9 commits behind head on main.

Files	Patch %	Lines
deeplake/core/transform/transform_tensor.py	90.90%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2751      +/-   ##
==========================================
- Coverage   83.79%   83.77%   -0.02%     
==========================================
  Files         233      233              
  Lines       26461    26533      +72     
==========================================
+ Hits        22172    22229      +57     
- Misses       4289     4304      +15     

Flag	Coverage Δ
unittests	83.77% <96.51%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sonarcloud]

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

2 New issues
0 Security Hotspots
96.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud