[AL-2353] Add delay for transform progress bar updates

[FayazRahman]

🚀 🚀 Pull Request

Impact

• Bug fix (non-breaking change which fixes expected existing functionality)
• Enhancement/New feature (adds functionality without impacting existing logic)
• Breaking change (fix or feature that would cause existing functionality to change)

Description

• When compute functions are really fast, progress bar updates become the slowest part of the chain.
• Update it in intervals of 5 seconds.

Things to be aware of

Things to worry about

Additional Context

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.01% ⚠️

Comparison is base (1c6db8a) 84.90% compared to head (5e38cbb) 84.89%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2515      +/-   ##
==========================================
- Coverage   84.90%   84.89%   -0.01%     
==========================================
  Files         328      328              
  Lines       38837    38846       +9     
==========================================
+ Hits        32973    32979       +6     
- Misses       5864     5867       +3     

Flag	Coverage Δ
unittests	84.89% <100.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
deeplake/core/vectorstore/deeplake_vectorstore.py	89.15% <ø> (ø)
...lake/core/vectorstore/test_deeplake_vectorstore.py	60.42% <100.00%> (+0.25%)	⬆️
deeplake/util/transform.py	95.40% <100.00%> (+0.07%)	⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nvoxland]

Does the performance improvements you saw still remain if you push the logic up to deeplake.core.compute.provider? @FayazRahman

        def sub_func(*args, **kwargs):
            callback_data = {
                "progress": 0,
                "last_progress_update_time": 0,
            }

            def pg_callback(value: int):
                callback_data["progress"] = callback_data["progress"] + value
                if (
                        time.time() - callback_data["last_progress_update_time"] > TRANSFORM_PROGRESSBAR_UPDATE_INTERVAL
                        or callback_data["progress"] == total_length - 1
                ):
                    progress_queue.put(callback_data["progress"])
                    callback_data["last_progress_update_time"] = time.time()
                    callback_data["progress"] = 0

            return func(pg_callback, *args, **kwargs)

[FayazRahman]

@nvoxland It should remain, but I would advice against that inorder to keep the pg_callback within the base class general.