Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[2.0] writing/reading fixed-shape arrays to chunks #842

Merged
merged 99 commits into from May 13, 2021
Merged

[2.0] writing/reading fixed-shape arrays to chunks #842

merged 99 commits into from May 13, 2021

Conversation

verbose-void
Copy link
Contributor

@verbose-void verbose-void commented May 7, 2021 •
edited

support chunked writing (not appending) for np.arrays -> storage providers with the following qualities:

  • batched/unbatched
  • fixed shape (all samples have same shape)

also adds:

  • in .circleci/config.yaml run pytest & pytest-benchmark separately (with --benchmark-skip & --benchmark-only flags)

things this branch does not do:

  • appending (writing arrays to a key that already had arrays written to it)
  • caching
  • index map chunking
  • compression

@verbose-void
method skeleton
57c0981
@verbose-void
write chunks for array to memory provider (basic)
6ec9e66
@verbose-void
start implementing chunk writing callstack
379fa8e
@verbose-void
broke up write function into function calls
e0bd95b
@verbose-void
save index map with pickle to storage provider
b86ad9d
@verbose-void
very simple reading & writing to/from chunks
92bc409
@verbose-void
new write.py (not done) with better structure
49bb3fc
@verbose-void
write index map inside write_array
4364b26
@verbose-void
rename functions more precisely (chunk_and_write...)
c4c3a2a
@verbose-void
add some batched tests
812b6f1
@verbose-void
started batched writing
9d5a138
@verbose-void
require >= 1 cache & don't compress incomplete chunks (also mark them)
2e2cf19
@verbose-void
remove no_cache tests
96e5fc3
@verbose-void
read from cache before storage & check for _incomplete chunks
4e4cc9f
@verbose-void
add normalize_and_batchify util with tests
4369cde
@verbose-void
use normalize_and_batchify_shape & update params
fa427a5
@verbose-void
use "bytes_left_in_last_chunk instead of last_chunk_num_bytes`
1f40bcb
@verbose-void
move cache/storage functions to storage_chain.py
730120d
@verbose-void
some comments for @abhinav
8d95e4a
@verbose-void verbose-void added the enhancement New feature or request label May 7, 2021
@verbose-void verbose-void self-assigned this May 7, 2021
@todo
Copy link

todo bot commented May 7, 2021

remove this:

https://github.com/activeloopai/Hub/blob/8d95e4aca5fea3465e3fcd5af334276450c9394a/hub/core/chunk_engine/__init__.py#L5-L6

This comment was generated by todo based on a TODO comment in 8d95e4a in #842. cc @activeloopai.

@todo
Copy link

todo bot commented May 7, 2021

change storage type to StorageProvider

https://github.com/activeloopai/Hub/blob/8d95e4aca5fea3465e3fcd5af334276450c9394a/hub/core/chunk_engine/read.py#L9-L14

This comment was generated by todo based on a TODO comment in 8d95e4a in #842. cc @activeloopai.

@todo
Copy link

todo bot commented May 7, 2021

read with slice

https://github.com/activeloopai/Hub/blob/8d95e4aca5fea3465e3fcd5af334276450c9394a/hub/core/chunk_engine/read.py#L10-L15

This comment was generated by todo based on a TODO comment in 8d95e4a in #842. cc @activeloopai.

@todo
Copy link

todo bot commented May 7, 2021

don't use pickle

https://github.com/activeloopai/Hub/blob/8d95e4aca5fea3465e3fcd5af334276450c9394a/hub/core/chunk_engine/read.py#L21-L26

This comment was generated by todo based on a TODO comment in 8d95e4a in #842. cc @activeloopai.

@todo
Copy link

todo bot commented May 7, 2021

don't use pickle

https://github.com/activeloopai/Hub/blob/8d95e4aca5fea3465e3fcd5af334276450c9394a/hub/core/chunk_engine/read.py#L25-L30

This comment was generated by todo based on a TODO comment in 8d95e4a in #842. cc @activeloopai.

@todo
Copy link

todo bot commented May 7, 2021

decode from array instead of dictionary

https://github.com/activeloopai/Hub/blob/8d95e4aca5fea3465e3fcd5af334276450c9394a/hub/core/chunk_engine/read.py#L37-L42

This comment was generated by todo based on a TODO comment in 8d95e4a in #842. cc @activeloopai.

benchislett
benchislett reviewed May 13, 2021
View reviewed changes
.circleci/config.yml Outdated Show resolved Hide resolved
benchislett
benchislett suggested changes May 13, 2021
View reviewed changes
Copy link
Contributor

@benchislett benchislett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The usual nitpicks

@verbose-void verbose-void merged commit 80074a4 into release/2.0 May 13, 2021
@verbose-void verbose-void deleted the feature/2.0/chunk-engine branch May 13, 2021 04:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@benchislett benchislett benchislett requested changes

@imshashank imshashank imshashank approved these changes

@AbhinavTuli AbhinavTuli Awaiting requested review from AbhinavTuli

Assignees

@verbose-void verbose-void

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@verbose-void @imshashank @benchislett