Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[2.0] writing/reading fixed-shape arrays to chunks #842

Merged
merged 99 commits into from
May 13, 2021
Merged

[2.0] writing/reading fixed-shape arrays to chunks #842

merged 99 commits into from
May 13, 2021

Commits on May 5, 2021

  1. method skeleton

    @verbose-void
    verbose-void committed May 5, 2021
    Configuration menu
    Copy the full SHA
    57c0981 View commit details
    Browse the repository at this point in the history

  2. write chunks for array to memory provider (basic)

    @verbose-void
    verbose-void committed May 5, 2021
    Configuration menu
    Copy the full SHA
    6ec9e66 View commit details
    Browse the repository at this point in the history

  3. start implementing chunk writing callstack

    @verbose-void
    verbose-void committed May 5, 2021
    Configuration menu
    Copy the full SHA
    379fa8e View commit details
    Browse the repository at this point in the history

  4. broke up write function into function calls

    @verbose-void
    verbose-void committed May 5, 2021
    Configuration menu
    Copy the full SHA
    e0bd95b View commit details
    Browse the repository at this point in the history

  5. save index map with pickle to storage provider

    @verbose-void
    verbose-void committed May 5, 2021
    Configuration menu
    Copy the full SHA
    b86ad9d View commit details
    Browse the repository at this point in the history

  6. very simple reading & writing to/from chunks

    @verbose-void
    verbose-void committed May 5, 2021
    Configuration menu
    Copy the full SHA
    92bc409 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2021

  1. new write.py (not done) with better structure

    @verbose-void
    verbose-void committed May 6, 2021
    Configuration menu
    Copy the full SHA
    49bb3fc View commit details
    Browse the repository at this point in the history

  2. write index map inside write_array

    @verbose-void
    verbose-void committed May 6, 2021
    Configuration menu
    Copy the full SHA
    4364b26 View commit details
    Browse the repository at this point in the history

  3. rename functions more precisely (chunk_and_write...)

    @verbose-void
    verbose-void committed May 6, 2021
    Configuration menu
    Copy the full SHA
    c4c3a2a View commit details
    Browse the repository at this point in the history

  4. add some batched tests

    @verbose-void
    verbose-void committed May 6, 2021
    Configuration menu
    Copy the full SHA
    812b6f1 View commit details
    Browse the repository at this point in the history

  5. started batched writing

    @verbose-void
    verbose-void committed May 6, 2021
    Configuration menu
    Copy the full SHA
    9d5a138 View commit details
    Browse the repository at this point in the history

  6. require >= 1 cache & don't compress incomplete chunks (also mark them)

    @verbose-void
    verbose-void committed May 6, 2021
    Configuration menu
    Copy the full SHA
    2e2cf19 View commit details
    Browse the repository at this point in the history

  7. remove no_cache tests

    @verbose-void
    verbose-void committed May 6, 2021
    Configuration menu
    Copy the full SHA
    96e5fc3 View commit details
    Browse the repository at this point in the history

  8. read from cache before storage & check for _incomplete chunks

    @verbose-void
    verbose-void committed May 6, 2021
    Configuration menu
    Copy the full SHA
    4e4cc9f View commit details
    Browse the repository at this point in the history

  9. add normalize_and_batchify util with tests

    @verbose-void
    verbose-void committed May 6, 2021
    Configuration menu
    Copy the full SHA
    4369cde View commit details
    Browse the repository at this point in the history

  10. use normalize_and_batchify_shape & update params

    @verbose-void
    verbose-void committed May 6, 2021
    Configuration menu
    Copy the full SHA
    fa427a5 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2021

  1. use "bytes_left_in_last_chunk instead of last_chunk_num_bytes`

    @verbose-void
    verbose-void committed May 7, 2021
    Configuration menu
    Copy the full SHA
    1f40bcb View commit details
    Browse the repository at this point in the history

  2. move cache/storage functions to storage_chain.py

    @verbose-void
    verbose-void committed May 7, 2021
    Configuration menu
    Copy the full SHA
    730120d View commit details
    Browse the repository at this point in the history

  3. some comments for @abhinav

    @verbose-void
    verbose-void committed May 7, 2021
    Configuration menu
    Copy the full SHA
    8d95e4a View commit details
    Browse the repository at this point in the history

  4. removed all caching code

    @verbose-void
    verbose-void committed May 7, 2021
    Configuration menu
    Copy the full SHA
    32a5aa9 View commit details
    Browse the repository at this point in the history

  5. make dummy compressors (chunk-wise/sample-wise)

    @verbose-void
    verbose-void committed May 7, 2021
    Configuration menu
    Copy the full SHA
    eeb8040 View commit details
    Browse the repository at this point in the history

  6. merged write_bytes & write_array to get logic working

    @verbose-void
    verbose-void committed May 7, 2021
    Configuration menu
    Copy the full SHA
    ac91aec View commit details
    Browse the repository at this point in the history

  7. works for single-chunk

    @verbose-void
    verbose-void committed May 7, 2021
    Configuration menu
    Copy the full SHA
    8b4b12b View commit details
    Browse the repository at this point in the history

  8. move getting meta to another function

    @verbose-void
    verbose-void committed May 7, 2021
    Configuration menu
    Copy the full SHA
    0d58bc4 View commit details
    Browse the repository at this point in the history

  9. move index_map operations to index_map.py

    @verbose-void
    verbose-void committed May 7, 2021
    Configuration menu
    Copy the full SHA
    aec1602 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2021

  1. moved tests into separate files (fixed/dynamic) & implement them fully

    @verbose-void
    verbose-void committed May 8, 2021
    Configuration menu
    Copy the full SHA
    85d3e5a View commit details
    Browse the repository at this point in the history

  2. remove redundant tests & change var names to be shorter

    @verbose-void
    verbose-void committed May 8, 2021
    Configuration menu
    Copy the full SHA
    9c79d86 View commit details
    Browse the repository at this point in the history

  3. TODO

    @verbose-void
    verbose-void committed May 8, 2021
    Configuration menu
    Copy the full SHA
    57c6919 View commit details
    Browse the repository at this point in the history

  4. add assertion message for failure case

    @verbose-void
    verbose-void committed May 8, 2021
    Configuration menu
    Copy the full SHA
    151b3cf View commit details
    Browse the repository at this point in the history

  5. Merge branch 'release/2.0' of github.com:activeloopai/Hub into featur… 

    …e/2.0/chunk-engine
    @verbose-void
    verbose-void committed May 8, 2021
    Configuration menu
    Copy the full SHA
    2162fd1 View commit details
    Browse the repository at this point in the history

  6. add dtypes for random arrays

    @verbose-void
    verbose-void committed May 8, 2021
    Configuration menu
    Copy the full SHA
    ac8df10 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2021

  1. mapper yield from mapper (instead of .items())

    @verbose-void
    verbose-void committed May 9, 2021
    Configuration menu
    Copy the full SHA
    fe1dbcd View commit details
    Browse the repository at this point in the history

  2. use actual memoryprovider

    @verbose-void
    verbose-void committed May 9, 2021
    Configuration menu
    Copy the full SHA
    fcf1e43 View commit details
    Browse the repository at this point in the history

  3. add compression / storage provider class in vars

    @verbose-void
    verbose-void committed May 9, 2021
    Configuration menu
    Copy the full SHA
    24a841f View commit details
    Browse the repository at this point in the history

  4. remove unused imports

    @verbose-void
    verbose-void committed May 9, 2021
    Configuration menu
    Copy the full SHA
    bcb2fed View commit details
    Browse the repository at this point in the history

  5. uncommented 5 batch param

    @verbose-void
    verbose-void committed May 9, 2021
    Configuration menu
    Copy the full SHA
    cb39ef0 View commit details
    Browse the repository at this point in the history

  6. create storage provider in util

    @verbose-void
    verbose-void committed May 9, 2021
    Configuration menu
    Copy the full SHA
    abae085 View commit details
    Browse the repository at this point in the history

  7. formatted

    @verbose-void
    verbose-void committed May 9, 2021
    Configuration menu
    Copy the full SHA
    5775c80 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'release/2.0' of github.com:activeloopai/Hub into featur… 

    …e/2.0/chunk-engine
    @verbose-void
    verbose-void committed May 9, 2021
    Configuration menu
    Copy the full SHA
    32f2dc3 View commit details
    Browse the repository at this point in the history

  9. clear storage before test (in case last one failed)

    @verbose-void
    verbose-void committed May 9, 2021
    Configuration menu
    Copy the full SHA
    a36a199 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2021

  1. start adding multi-batch support

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    7d3ecc5 View commit details
    Browse the repository at this point in the history

  2. moved meta validation & updating to functions within meta.py

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    2055644 View commit details
    Browse the repository at this point in the history

  3. removed write_impl, reduce test shape sizes

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    fda38d1 View commit details
    Browse the repository at this point in the history

  4. rewriting chunk array method & deferring more logic to 

    meta.py/index_map.py
    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    8234870 View commit details
    Browse the repository at this point in the history

  5. reduce scope (no appending) & limit tests to this scope

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    733603a View commit details
    Browse the repository at this point in the history

  6. remove dead code

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    b5971a7 View commit details
    Browse the repository at this point in the history

  7. TODOs & remove dead code

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    6ba4f27 View commit details
    Browse the repository at this point in the history

  8. add write_array docstring, create some general functions & typing

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    1ab17e2 View commit details
    Browse the repository at this point in the history

  9. keep track of min/max shape for tensors & validate meta in tests

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    07bb887 View commit details
    Browse the repository at this point in the history

  10. read_array & accept slice as argument

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    8f63e92 View commit details
    Browse the repository at this point in the history

  11. allow slice to be None for reading & return all samples

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    bc686b7 View commit details
    Browse the repository at this point in the history

  12. remove generator of decompressed chunks

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    7d19b7b View commit details
    Browse the repository at this point in the history

  13. fixed typing issues

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    7841a0c View commit details
    Browse the repository at this point in the history

  14. default value isn't None, but is slice (for read index)

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    b1116c2 View commit details
    Browse the repository at this point in the history

  15. moved tests/util -> tests/common

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    fa16cc7 View commit details
    Browse the repository at this point in the history

  16. get random chunk name

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    dbe62cb View commit details
    Browse the repository at this point in the history

  17. rename generator -> chunker (also methods) & add join_chunks tests

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    c0f3111 View commit details
    Browse the repository at this point in the history

  18. read_array returns in docstring & added join_chunks docstring

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    2922f3c View commit details
    Browse the repository at this point in the history

  19. returns docstring

    @verbose-void
    verbose-void committed May 10, 2021
    Configuration menu
    Copy the full SHA
    36e6f7d View commit details
    Browse the repository at this point in the history

Commits on May 11, 2021

  1. docstrings, variable names, & removed index/meta modules

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    831f3ea View commit details
    Browse the repository at this point in the history

  2. removed references to deleted modules

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    065f98b View commit details
    Browse the repository at this point in the history

  3. validate meta

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    fceaa5d View commit details
    Browse the repository at this point in the history

  4. remove unused import

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    c0ba66d View commit details
    Browse the repository at this point in the history

  5. update read docstring

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    d33f311 View commit details
    Browse the repository at this point in the history

  6. validate index_map incomplete chunks

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    fa1b8c1 View commit details
    Browse the repository at this point in the history

  7. add tests for validating actual chunk_sizes & incomplete_chunks

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    8d88fc4 View commit details
    Browse the repository at this point in the history

  8. fixed typing

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    1365527 View commit details
    Browse the repository at this point in the history

  9. move constants, delete unused tests, fixed test case

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    11cff64 View commit details
    Browse the repository at this point in the history

  10. change the way chunk_sizes are being asserted

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    f20683f View commit details
    Browse the repository at this point in the history

  11. extend last chunk if it is incomplete

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    20d4acf View commit details
    Browse the repository at this point in the history

  12. added join_chunks extra test case & pass in join_chunks

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    81d81a8 View commit details
    Browse the repository at this point in the history

  13. remove print

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    ee24503 View commit details
    Browse the repository at this point in the history

  14. raise ValueError instead of assert False

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    4fdf386 View commit details
    Browse the repository at this point in the history

  15. fixed mypy types for last_chunk/chunk_name

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    1c972c5 View commit details
    Browse the repository at this point in the history

  16. update docstrings

    @verbose-void
    verbose-void committed May 11, 2021
    Configuration menu
    Copy the full SHA
    2208911 View commit details
    Browse the repository at this point in the history

Commits on May 12, 2021

  1. Merge branch 'release/2.0' of github.com:activeloopai/Hub into featur… 

    …e/2.0/chunk-engine
    @verbose-void
    verbose-void committed May 12, 2021
    Configuration menu
    Copy the full SHA
    8399fb2 View commit details
    Browse the repository at this point in the history

  2. write pytest-benchmarks for 1GB write/read

    @verbose-void
    verbose-void committed May 12, 2021
    Configuration menu
    Copy the full SHA
    1968f9d View commit details
    Browse the repository at this point in the history

  3. moved get_random_array to util & write tests, also write/read bench

    @verbose-void
    verbose-void committed May 12, 2021
    Configuration menu
    Copy the full SHA
    46b8b2e View commit details
    Browse the repository at this point in the history

  4. skip benchmarks when running pytest .

    @verbose-void
    verbose-void committed May 12, 2021
    Configuration menu
    Copy the full SHA
    5da1e1f View commit details
    Browse the repository at this point in the history

  5. use --benchmark-enable in circleci (override setup.cfg)

    @verbose-void
    verbose-void committed May 12, 2021
    Configuration menu
    Copy the full SHA
    2ca88f9 View commit details
    Browse the repository at this point in the history

  6. print array GBs

    @verbose-void
    verbose-void committed May 12, 2021
    Configuration menu
    Copy the full SHA
    84c5f2e View commit details
    Browse the repository at this point in the history

  7. add static typing for tests/utils

    @verbose-void
    verbose-void committed May 12, 2021
    Configuration menu
    Copy the full SHA
    56c7080 View commit details
    Browse the repository at this point in the history

  8. benchmark 16MB only

    @verbose-void
    verbose-void committed May 12, 2021
    Configuration menu
    Copy the full SHA
    d61e78a View commit details
    Browse the repository at this point in the history

  9. separate benchmark runs from testing runs

    @verbose-void
    verbose-void committed May 12, 2021
    Configuration menu
    Copy the full SHA
    d95fe38 View commit details
    Browse the repository at this point in the history

  10. broke write_array into 2 functions & added docstrings/minor TODOs

    @verbose-void
    verbose-void committed May 12, 2021
    Configuration menu
    Copy the full SHA
    74ad008 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2021

  1. expose core to user

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    bf3cecb View commit details
    Browse the repository at this point in the history

  2. remove core exposure

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    3910d7c View commit details
    Browse the repository at this point in the history

  3. Merge branch 'release/2.0' of github.com:activeloopai/Hub into featur… 

    …e/2.0/chunk-engine
    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    c80f0b7 View commit details
    Browse the repository at this point in the history

  4. fixed list failure when key empty

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    1db82de View commit details
    Browse the repository at this point in the history

  5. run tests with s3 provider & clear only memory providers

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    015be2d View commit details
    Browse the repository at this point in the history

  6. fixed benchmark error

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    23ec487 View commit details
    Browse the repository at this point in the history

  7. ignore pytest_cases for mypy & add static types

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    54eaac7 View commit details
    Browse the repository at this point in the history

  8. don't clear memory after writing for benchmarks

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    eca8355 View commit details
    Browse the repository at this point in the history

  9. don't clear after read benchmark

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    36b1de7 View commit details
    Browse the repository at this point in the history

  10. don't use extend, use b"".join(...)

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    07e0e68 View commit details
    Browse the repository at this point in the history

  11. move get_random_array -> tests/common.py

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    0d0a40a View commit details
    Browse the repository at this point in the history

  12. remove &

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    a2765c4 View commit details
    Browse the repository at this point in the history

  13. added args for _get_last_chunk docstring

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    cf833dd View commit details
    Browse the repository at this point in the history

  14. add TODO

    @verbose-void
    verbose-void committed May 13, 2021
    Configuration menu
    Copy the full SHA
    02c87b6 View commit details
    Browse the repository at this point in the history