Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added support for cloud object storage (S3, GCS, ADLS, etc.) #1164

Merged
merged 40 commits into from
May 5, 2021
Merged

Added support for cloud object storage (S3, GCS, ADLS, etc.) #1164

merged 40 commits into from
May 5, 2021

Commits on Apr 23, 2021

  1. Added support for remote filesystems (S3, GCS, ADLS, etc.)

    @tgaddair
    tgaddair committed Apr 23, 2021
    Configuration menu
    Copy the full SHA
    d656a17 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2021

  1. Use fsspec to open files

    @tgaddair
    tgaddair committed Apr 24, 2021
    Configuration menu
    Copy the full SHA
    acc71e8 View commit details
    Browse the repository at this point in the history

  2. More file utils

    @tgaddair
    tgaddair committed Apr 24, 2021
    Configuration menu
    Copy the full SHA
    6cdf128 View commit details
    Browse the repository at this point in the history

  3. Fixed fsspec modes

    @tgaddair
    tgaddair committed Apr 24, 2021
    Configuration menu
    Copy the full SHA
    ee399c8 View commit details
    Browse the repository at this point in the history

  4. Fixed various h5py issues

    @tgaddair
    tgaddair committed Apr 24, 2021
    Configuration menu
    Copy the full SHA
    62910b2 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2021

  1. Fixed h5py

    @tgaddair
    tgaddair committed Apr 25, 2021
    Configuration menu
    Copy the full SHA
    188f1c2 View commit details
    Browse the repository at this point in the history

  2. Added remote tests

    @tgaddair
    tgaddair committed Apr 25, 2021
    Configuration menu
    Copy the full SHA
    3712958 View commit details
    Browse the repository at this point in the history

  3. Fixed uploading

    @tgaddair
    tgaddair committed Apr 25, 2021
    Configuration menu
    Copy the full SHA
    5035c07 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into remote-fs

    @tgaddair
    tgaddair committed Apr 25, 2021
    Configuration menu
    Copy the full SHA
    da4e359 View commit details
    Browse the repository at this point in the history

  5. Reduced epochs

    @tgaddair
    tgaddair committed Apr 25, 2021
    Configuration menu
    Copy the full SHA
    028e7b2 View commit details
    Browse the repository at this point in the history

  6. WIP added intermediate format to backend

    @tgaddair
    tgaddair committed Apr 25, 2021
    Configuration menu
    Copy the full SHA
    46459dc View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2021

  1. WIP refactor caching

    @tgaddair
    tgaddair committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    24c0026 View commit details
    Browse the repository at this point in the history

  2. WIP fixed HDF5

    @tgaddair
    tgaddair committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    8482292 View commit details
    Browse the repository at this point in the history

  3. WIP fixed HDF5

    @tgaddair
    tgaddair committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    f14973a View commit details
    Browse the repository at this point in the history

  4. Set default data formats

    @tgaddair
    tgaddair committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    4b1ea8d View commit details
    Browse the repository at this point in the history

  5. Removed create_dataset

    @tgaddair
    tgaddair committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    643260a View commit details
    Browse the repository at this point in the history

  6. Fixed checksum

    @tgaddair
    tgaddair committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    4f32bb3 View commit details
    Browse the repository at this point in the history

  7. Refactor

    @tgaddair
    tgaddair committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    3d53542 View commit details
    Browse the repository at this point in the history

  8. Fixed Dask and Ray

    @tgaddair
    tgaddair committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    0d36166 View commit details
    Browse the repository at this point in the history

  9. Added assertions

    @tgaddair
    tgaddair committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    fe0ab92 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2021

  1. Fixed image features

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    8ef1ad3 View commit details
    Browse the repository at this point in the history

  2. Fixed parquet cache

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    2d8743c View commit details
    Browse the repository at this point in the history

  3. Fixed caching

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    83e8cb7 View commit details
    Browse the repository at this point in the history

  4. Fixed tests

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    9be9de3 View commit details
    Browse the repository at this point in the history

  5. Fixed loading from cached parquet

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    b6996c6 View commit details
    Browse the repository at this point in the history

  6. Fixed Dask tests

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    e38c46e View commit details
    Browse the repository at this point in the history

  7. Test disable parquet tests

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    4596a56 View commit details
    Browse the repository at this point in the history

  8. Close Petastorm reader when finished

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    9b30173 View commit details
    Browse the repository at this point in the history

  9. Added docker deps

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    02bb160 View commit details
    Browse the repository at this point in the history

  10. Try without build-essential

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    318a0af View commit details
    Browse the repository at this point in the history

  11. Added back build-essential

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    8df0839 View commit details
    Browse the repository at this point in the history

  12. build-essential

    @tgaddair
    tgaddair committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    1ef0d86 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2021

  1. Fixed preprocess_for_prediction

    @tgaddair
    tgaddair committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    50b75c0 View commit details
    Browse the repository at this point in the history

  2. Fixed typo

    @tgaddair
    tgaddair committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    b4e043e View commit details
    Browse the repository at this point in the history

Commits on May 2, 2021

  1. Use only alphanumeric characters in filenames

    @tgaddair
    tgaddair committed May 2, 2021
    Configuration menu
    Copy the full SHA
    686f50b View commit details
    Browse the repository at this point in the history

  2. Merge

    @tgaddair
    tgaddair committed May 2, 2021
    Configuration menu
    Copy the full SHA
    22e6134 View commit details
    Browse the repository at this point in the history

  3. Removed expensive length check

    @tgaddair
    tgaddair committed May 2, 2021
    Configuration menu
    Copy the full SHA
    96a7252 View commit details
    Browse the repository at this point in the history

  4. Revert "Removed expensive length check" 

    This reverts commit 96a7252.
    @tgaddair
    tgaddair committed May 2, 2021
    Configuration menu
    Copy the full SHA
    e35fb5e View commit details
    Browse the repository at this point in the history

  5. Upgraded petastorm

    @tgaddair
    tgaddair committed May 2, 2021
    Configuration menu
    Copy the full SHA
    643d559 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2021

  1. Merge branch 'master' into remote-fs

    @w4nderlust
    w4nderlust committed May 5, 2021
    Configuration menu
    Copy the full SHA
    d239777 View commit details
    Browse the repository at this point in the history