Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[python] dataloader optimization (picking up 1169) #1224

Merged
merged 21 commits into from
Jul 8, 2024
Merged

[python] dataloader optimization (picking up 1169) #1224

merged 21 commits into from
Jul 8, 2024

Commits on Jun 7, 2024

  1. ExperimentDataPipe: configurable array-conversion method 

    `nd.array`, `scipy.coo`, `scipy.csr`
    @ryan-williams
    ryan-williams committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    b8182d8 View commit details
    Browse the repository at this point in the history

  2. group chunks iff shuffle_chunk_count > 1

    @ryan-williams
    ryan-williams committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    f7f6121 View commit details
    Browse the repository at this point in the history

  3. rm scipy.coo method

    @ryan-williams
    ryan-williams committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    9655f8b View commit details
    Browse the repository at this point in the history

  4. add METHODS constant

    @ryan-williams
    ryan-williams committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    54ffe73 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. Use return_sparse_X instead of adding a method kwarg

    @ivirshup
    ivirshup committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    2fd1b14 View commit details
    Browse the repository at this point in the history

  2. Oops, one more thing

    @ivirshup
    ivirshup committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    b74bf67 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into ivirshup/optimize-dataloader

    @ivirshup
    ivirshup committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    f29b4c0 View commit details
    Browse the repository at this point in the history

  4. revert change to shuffle_chunk_count branch

    @ivirshup
    ivirshup committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    664812a View commit details
    Browse the repository at this point in the history

  5. mypy + linting fixes

    @ivirshup
    ivirshup committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    629af82 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. Add test for bug I found

    @ivirshup
    ivirshup committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    8bab49a View commit details
    Browse the repository at this point in the history

  2. Fix bug, possibly speed up dataloaders

    @ivirshup
    ivirshup committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    4c042ee View commit details
    Browse the repository at this point in the history

  3. Remove unused (I think) branch

    @ivirshup
    ivirshup committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    9d5cb73 View commit details
    Browse the repository at this point in the history

  4. Remove more unused code

    @ivirshup
    ivirshup committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    295b308 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2024

  1. Slightly more precise typing

    @ivirshup
    ivirshup committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    137f77d View commit details
    Browse the repository at this point in the history

  2. Reduce additional copies in _tables_to_np

    @ivirshup
    ivirshup committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    4ac8511 View commit details
    Browse the repository at this point in the history

  3. Simplify branching in _ObsAndXSOMAIterator

    @ivirshup
    ivirshup committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    f943404 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into ivirshup/optimize-dataloader

    @ivirshup
    ivirshup committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    c1204bf View commit details
    Browse the repository at this point in the history

  5. typing fix

    @ivirshup
    ivirshup committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    357582e View commit details
    Browse the repository at this point in the history

  6. Revert "typing fix" 

    This reverts commit 357582e.
    @ivirshup
    ivirshup committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    15cb1a8 View commit details
    Browse the repository at this point in the history

  7. Use more simplistic typing to avoid mypy + old python wrath

    @ivirshup
    ivirshup committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    4b1a993 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'main' into ivirshup/optimize-dataloader

    @ivirshup
    ivirshup committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    9cd6df7 View commit details
    Browse the repository at this point in the history