Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NPZ replacement format (only) #1047

Merged
merged 97 commits into from
Jul 14, 2021
Merged

NPZ replacement format (only) #1047

merged 97 commits into from
Jul 14, 2021

Commits on Jul 7, 2021

  1. initial

    @farizrahman4u
    farizrahman4u committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    dbd83e3 View commit details
    Browse the repository at this point in the history

  2. format

    @farizrahman4u
    farizrahman4u committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    557df26 View commit details
    Browse the repository at this point in the history

  3. typo

    @farizrahman4u
    farizrahman4u committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    95ce176 View commit details
    Browse the repository at this point in the history

  4. typo

    @farizrahman4u
    farizrahman4u committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    950ac7c View commit details
    Browse the repository at this point in the history

  5. bug fix

    @farizrahman4u
    farizrahman4u committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    b55fc38 View commit details
    Browse the repository at this point in the history

  6. some docs and fix 1D shapes

    @verbose-void
    verbose-void committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    b0f7d88 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'fr_optimizations' of github.com:activeloopai/Hub into f… 

    …r_optimizations
    @verbose-void
    verbose-void committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    3f34045 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'refactor/2.0/chunk-engine' of github.com:activeloopai/H… 

    …ub into fr_optimizations
    @verbose-void
    verbose-void committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    0c83842 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'refactor/2.0/chunk-engine' of https://www.github.com/ac… 

    …tiveloopai/hub into fr_optimizations
    @farizrahman4u
    farizrahman4u committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    64c3037 View commit details
    Browse the repository at this point in the history

  10. add assertion for easy debugging

    @farizrahman4u
    farizrahman4u committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    2281692 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'fr_optimizations' of https://www.github.com/activeloopa… 

    …i/hub into fr_optimizations
    @farizrahman4u
    farizrahman4u committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    4039fe9 View commit details
    Browse the repository at this point in the history

  12. one off

    @farizrahman4u
    farizrahman4u committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    26cd377 View commit details
    Browse the repository at this point in the history

  13. segfault fix

    @farizrahman4u
    farizrahman4u committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    fadc940 View commit details
    Browse the repository at this point in the history

  14. smol fixes

    @farizrahman4u
    farizrahman4u committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    035bacc View commit details
    Browse the repository at this point in the history

  15. add clear cache to memory test in api and fix return in decode

    @verbose-void
    verbose-void committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    1eddf74 View commit details
    Browse the repository at this point in the history

  16. add a better exception for pointer GC

    @verbose-void
    verbose-void committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    3a409e1 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2021

  1. merge conf fix + infer chunk byte size

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    0f58f6e View commit details
    Browse the repository at this point in the history

  2. smol fix

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    ef11252 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'fr_optimizations' of https://www.github.com/activeloopa… 

    …i/hub into fr_optimizations
    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    2fb777a View commit details
    Browse the repository at this point in the history

  4. all fix

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    57d2da7 View commit details
    Browse the repository at this point in the history

  5. chuunk id optims init

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    4dde08a View commit details
    Browse the repository at this point in the history

  6. debug msgs

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    df03495 View commit details
    Browse the repository at this point in the history

  7. fix refcounting bug

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    71ee06c View commit details
    Browse the repository at this point in the history

  8. ren shards->data

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    44b5ade View commit details
    Browse the repository at this point in the history

  9. faster buff load

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    6f086e2 View commit details
    Browse the repository at this point in the history

  10. save 1 memcpy

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    31aa04d View commit details
    Browse the repository at this point in the history

  11. indexing

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    e98b008 View commit details
    Browse the repository at this point in the history

  12. cache data len

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    3b68c57 View commit details
    Browse the repository at this point in the history

  13. cache _num_chunks

    @farizrahman4u
    farizrahman4u committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    2d91772 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2021

  1. chunk engine updates cache size

    @verbose-void
    verbose-void committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    dcea7cc View commit details
    Browse the repository at this point in the history

  2. rename remove -> remove_from_dirty

    @verbose-void
    verbose-void committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    04276f0 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'optimize/uploads' into fr_optimizations

    @verbose-void
    verbose-void committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    bad4186 View commit details
    Browse the repository at this point in the history

  4. remove some sums

    @verbose-void
    verbose-void committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    36002e2 View commit details
    Browse the repository at this point in the history

  5. optims for seq access

    @farizrahman4u
    farizrahman4u committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    fea211d View commit details
    Browse the repository at this point in the history

  6. Merge branch 'fr_optimizations' of https://www.github.com/activeloopa… 

    …i/hub into fr_optimizations
    @farizrahman4u
    farizrahman4u committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    c9bc44a View commit details
    Browse the repository at this point in the history

  7. cache entry

    @farizrahman4u
    farizrahman4u committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    1dfb3c2 View commit details
    Browse the repository at this point in the history

  8. 10s upload speedup

    @verbose-void
    verbose-void committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    c8a9931 View commit details
    Browse the repository at this point in the history

  9. fix mypy

    @verbose-void
    verbose-void committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    16d5fb9 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'optimize/uploads' of github.com:activeloopai/Hub into f… 

    …r_optimizations
    @verbose-void
    verbose-void committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    96280b7 View commit details
    Browse the repository at this point in the history

  11. load chunk ID encoder

    @verbose-void
    verbose-void committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    121753d View commit details
    Browse the repository at this point in the history

  12. mypass binsearch

    @farizrahman4u
    farizrahman4u committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    7c2221e View commit details
    Browse the repository at this point in the history

  13. format

    @farizrahman4u
    farizrahman4u committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    60b73d3 View commit details
    Browse the repository at this point in the history

  14. merge conf fix

    @farizrahman4u
    farizrahman4u committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    07456b5 View commit details
    Browse the repository at this point in the history

  15. rem debug line

    @farizrahman4u
    farizrahman4u committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    8cb1bce View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2021

  1. fr_optimizations_2

    @farizrahman4u
    farizrahman4u committed Jul 10, 2021
    Configuration menu
    Copy the full SHA
    91596e4 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #1039 from activeloopai/fr_optimizations_2 

    Optimize chunk id encoder for sequential access
    @farizrahman4u
    farizrahman4u committed Jul 10, 2021
    Configuration menu
    Copy the full SHA
    9c3c6d5 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2021

  1. optimize tensor iteration

    @farizrahman4u
    farizrahman4u committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    2f28314 View commit details
    Browse the repository at this point in the history

  2. dsiter

    @farizrahman4u
    farizrahman4u committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    7fd0193 View commit details
    Browse the repository at this point in the history

  3. fix test

    @farizrahman4u
    farizrahman4u committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    b63c565 View commit details
    Browse the repository at this point in the history

  4. fix test

    @farizrahman4u
    farizrahman4u committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    ae3c17d View commit details
    Browse the repository at this point in the history

  5. ds iter fixes

    @farizrahman4u
    farizrahman4u committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    069a9f6 View commit details
    Browse the repository at this point in the history

  6. tests

    @farizrahman4u
    farizrahman4u committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    d0306a5 View commit details
    Browse the repository at this point in the history

  7. test

    @farizrahman4u
    farizrahman4u committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    ec4516b View commit details
    Browse the repository at this point in the history

  8. format

    @farizrahman4u
    farizrahman4u committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    f6d71f0 View commit details
    Browse the repository at this point in the history

  9. pytorch training optims

    @farizrahman4u
    farizrahman4u committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    1a107ca View commit details
    Browse the repository at this point in the history

  10. rem bad checks

    @farizrahman4u
    farizrahman4u committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    0cefcd2 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2021

  1. format

    @farizrahman4u
    farizrahman4u committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    318d496 View commit details
    Browse the repository at this point in the history

  2. format + smoll change in encoding format

    @farizrahman4u
    farizrahman4u committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    b1591c4 View commit details
    Browse the repository at this point in the history

  3. minimize searchsorted calls

    @farizrahman4u
    farizrahman4u committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    8f475a9 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2021

  1. refac chunk_id.py

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    a7dd7f1 View commit details
    Browse the repository at this point in the history

  2. more refacc

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    746201c View commit details
    Browse the repository at this point in the history

  3. encode_*->serialize_*

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    440a0b7 View commit details
    Browse the repository at this point in the history

  4. Update hub/core/chunk.py 

    Co-authored-by: dyllan <mccreary@dyllan.ai>
    @farizrahman4u @verbose-void
    farizrahman4u and verbose-void committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    ceae226 View commit details
    Browse the repository at this point in the history

  5. docstring

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    88ab4bb View commit details
    Browse the repository at this point in the history

  6. Merge branch 'fr_optimizations' of https://www.github.com/activeloopa… 

    …i/hub into fr_optimizations
    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    f3c891c View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into fr_optimizations

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    3f3af34 View commit details
    Browse the repository at this point in the history

  8. docstring

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    4b29507 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'fr_optimizations' of https://www.github.com/activeloopa… 

    …i/hub into fr_optimizations
    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    5093220 View commit details
    Browse the repository at this point in the history

  10. merge main

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    fb46618 View commit details
    Browse the repository at this point in the history

  11. rm comments

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    5b38608 View commit details
    Browse the repository at this point in the history

  12. rm unused import

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    4f25b21 View commit details
    Browse the repository at this point in the history

  13. revert dataset.py

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    82ce5be View commit details
    Browse the repository at this point in the history

  14. revert tensor.py

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    35d3a4a View commit details
    Browse the repository at this point in the history

  15. revert test_api.py

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    cb4ea21 View commit details
    Browse the repository at this point in the history

  16. revert ChunkEngine.numpy

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    868004e View commit details
    Browse the repository at this point in the history

  17. revert read_sample_from_chunk

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    c7a1321 View commit details
    Browse the repository at this point in the history

  18. rem unreachable

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    6519986 View commit details
    Browse the repository at this point in the history

  19. remove iter logic

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    0ddc61b View commit details
    Browse the repository at this point in the history

  20. revert pytorch.py

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    609ea67 View commit details
    Browse the repository at this point in the history

  21. revert dataset.py

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    7823060 View commit details
    Browse the repository at this point in the history

  22. reverts

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    390151b View commit details
    Browse the repository at this point in the history

  23. revert tensor.py

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    77ca4f7 View commit details
    Browse the repository at this point in the history

  24. reverts

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    e3ab3bf View commit details
    Browse the repository at this point in the history

  25. fixes

    @farizrahman4u
    farizrahman4u committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    0e5e84c View commit details
    Browse the repository at this point in the history

  26. add chunk size tests

    @verbose-void
    verbose-void committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    0ee485d View commit details
    Browse the repository at this point in the history

  27. Merge branch 'fr_serialization' of github.com:activeloopai/Hub into f… 

    …r_serialization
    @verbose-void
    verbose-void committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    7df6b7c View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2021

  1. fixes

    @farizrahman4u
    farizrahman4u committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    d16d550 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'fr_serialization' of https://www.github.com/activeloopa… 

    …i/hub into fr_serialization
    @farizrahman4u
    farizrahman4u committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    a2fa181 View commit details
    Browse the repository at this point in the history

  3. fixes

    @farizrahman4u
    farizrahman4u committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    1b0973a View commit details
    Browse the repository at this point in the history

  4. rem assert

    @farizrahman4u
    farizrahman4u committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    5895438 View commit details
    Browse the repository at this point in the history

  5. test chunk sizes on memds only

    @farizrahman4u
    farizrahman4u committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    f15d71c View commit details
    Browse the repository at this point in the history

  6. Update hub/core/serialize.py 

    Co-authored-by: dyllan <mccreary@dyllan.ai>
    @farizrahman4u @verbose-void
    farizrahman4u and verbose-void committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    0d7e9f2 View commit details
    Browse the repository at this point in the history

  7. Update hub/core/serialize.py 

    Co-authored-by: dyllan <mccreary@dyllan.ai>
    @farizrahman4u @verbose-void
    farizrahman4u and verbose-void committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    8310e36 View commit details
    Browse the repository at this point in the history

  8. Update hub/core/serialize.py 

    Co-authored-by: dyllan <mccreary@dyllan.ai>
    @farizrahman4u @verbose-void
    farizrahman4u and verbose-void committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    3a8ccc8 View commit details
    Browse the repository at this point in the history

  9. Update hub/core/serialize.py 

    Co-authored-by: dyllan <mccreary@dyllan.ai>
    @farizrahman4u @verbose-void
    farizrahman4u and verbose-void committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    d9a846b View commit details
    Browse the repository at this point in the history

  10. rem assertions

    @farizrahman4u
    farizrahman4u committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    8c3b83b View commit details
    Browse the repository at this point in the history

  11. Merge branch 'fr_serialization' of https://www.github.com/activeloopa… 

    …i/hub into fr_serialization
    @farizrahman4u
    farizrahman4u committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    7007c59 View commit details
    Browse the repository at this point in the history