Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ADD S3 support for downloading and uploading processed datasets #1723

Merged
merged 57 commits into from
Jan 26, 2021
Merged

ADD S3 support for downloading and uploading processed datasets #1723

merged 57 commits into from
Jan 26, 2021

Commits on Jan 8, 2021

  1. added fsspec and fsspec[s3] adjusted save_to_disk function

    @philschmid
    philschmid committed Jan 8, 2021
    Configuration menu
    Copy the full SHA
    6cf8813 View commit details
    Browse the repository at this point in the history

  2. added reading from s3

    @philschmid
    philschmid committed Jan 8, 2021
    Configuration menu
    Copy the full SHA
    aa90496 View commit details
    Browse the repository at this point in the history

  3. fixed save_to_disk for s3 path

    @philschmid
    philschmid committed Jan 8, 2021
    Configuration menu
    Copy the full SHA
    6dba448 View commit details
    Browse the repository at this point in the history

  4. implemented tests

    @philschmid
    philschmid committed Jan 8, 2021
    Configuration menu
    Copy the full SHA
    ac1d90f View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2021

  1. added filesystem utils to arrow dataset and dataset dict

    @philschmid
    philschmid committed Jan 11, 2021
    Configuration menu
    Copy the full SHA
    49b2fd8 View commit details
    Browse the repository at this point in the history

  2. added tests for filesystem_utils

    @philschmid
    philschmid committed Jan 11, 2021
    Configuration menu
    Copy the full SHA
    7eb6160 View commit details
    Browse the repository at this point in the history

  3. added DatasetDict test

    @philschmid
    philschmid committed Jan 11, 2021
    Configuration menu
    Copy the full SHA
    82f891d View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2021

  1. changed var from s3_ to proc_

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    ed17c5e View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/master'

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    3bbab8c View commit details
    Browse the repository at this point in the history

  3. fixed error in load from disk function

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    574c9dd View commit details
    Browse the repository at this point in the history

  4. fixing directory creation

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    0315025 View commit details
    Browse the repository at this point in the history

  5. removed fs.makedirs since files has to be saved temp local

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    8d072e6 View commit details
    Browse the repository at this point in the history

  6. fixed code quality checks

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    7a70258 View commit details
    Browse the repository at this point in the history

  7. fixed quality check

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    29313ab View commit details
    Browse the repository at this point in the history

  8. added noqa for pytest to work with moto

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    5f6749f View commit details
    Browse the repository at this point in the history

  9. stupid mistake with wrong order at imports

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    354e39f View commit details
    Browse the repository at this point in the history

  10. adjuste boto3 version to work with moto in tests

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    bc36832 View commit details
    Browse the repository at this point in the history

  11. removed pytest fixtures from unittest class

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    57bcbe7 View commit details
    Browse the repository at this point in the history

  12. forgot to remove fixture as parameter...

    @philschmid
    philschmid committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    3493e87 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2021

  1. Make it working with Windows style paths.

    @mfuntowicz
    mfuntowicz committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    b4fa6a9 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #1 from philschmid/add_s3 

    Make it working with Windows style paths.
    @philschmid
    philschmid committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    0be3a11 View commit details
    Browse the repository at this point in the history

  3. fixed code quality

    @philschmid
    philschmid committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    5fbfcd7 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'upstream/master'

    @philschmid
    philschmid committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    b540612 View commit details
    Browse the repository at this point in the history

  5. fixed hopefully the last path problems for WIN

    @philschmid
    philschmid committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    9a1b282 View commit details
    Browse the repository at this point in the history

  6. added Path().pathjoin with posix to load_from_disk for DatasetDict keys

    @philschmid
    philschmid committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    081f4bc View commit details
    Browse the repository at this point in the history

  7. fixed win path problem

    @philschmid
    philschmid committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    2ce5fec View commit details
    Browse the repository at this point in the history

  8. create conditional dataset_dict_split_path for creating correct path … 

    …for win and linux
    @philschmid
    philschmid committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    d346c6f View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2021

  1. added s3 as extra requires

    @philschmid
    philschmid committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    f25a036 View commit details
    Browse the repository at this point in the history

  2. fixed boto imports for docs

    @philschmid
    philschmid committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    df78d8b View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2021

  1. added S3FileSystem with documentation

    @philschmid
    philschmid committed Jan 17, 2021
    Configuration menu
    Copy the full SHA
    e3fa922 View commit details
    Browse the repository at this point in the history

  2. reworked everything for datasets.filesystem

    @philschmid
    philschmid committed Jan 17, 2021
    Configuration menu
    Copy the full SHA
    fb992a5 View commit details
    Browse the repository at this point in the history

  3. documentation and styling

    @philschmid
    philschmid committed Jan 17, 2021
    Configuration menu
    Copy the full SHA
    53a6a4b View commit details
    Browse the repository at this point in the history

  4. added s3fs for documentation

    @philschmid
    philschmid committed Jan 17, 2021
    Configuration menu
    Copy the full SHA
    85f0297 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2021

  1. handle optional s3fs dependency

    @lhoestq
    lhoestq committed Jan 18, 2021
    Configuration menu
    Copy the full SHA
    8885a7b View commit details
    Browse the repository at this point in the history

  2. fix test

    @lhoestq
    lhoestq committed Jan 18, 2021
    Configuration menu
    Copy the full SHA
    b91345c View commit details
    Browse the repository at this point in the history

  3. adjusted doc order and renamed preproc_dataset_path to extract_path_f… 

    …rom_uri
    @philschmid
    philschmid committed Jan 18, 2021
    Configuration menu
    Copy the full SHA
    93a5f5b View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2021

  1. added temp dir when saving

    @philschmid
    philschmid committed Jan 19, 2021
    Configuration menu
    Copy the full SHA
    8b55b89 View commit details
    Browse the repository at this point in the history

  2. fixed quality

    @philschmid
    philschmid committed Jan 19, 2021
    Configuration menu
    Copy the full SHA
    2bf289d View commit details
    Browse the repository at this point in the history

  3. added documentation

    @philschmid
    philschmid committed Jan 19, 2021
    Configuration menu
    Copy the full SHA
    83e4673 View commit details
    Browse the repository at this point in the history

  4. implemented save_to_disk for local remote filesystem with temp dir

    @philschmid
    philschmid committed Jan 19, 2021
    Configuration menu
    Copy the full SHA
    04042ea View commit details
    Browse the repository at this point in the history

  5. fixed documentation example

    @philschmid
    philschmid committed Jan 19, 2021
    Configuration menu
    Copy the full SHA
    ec29076 View commit details
    Browse the repository at this point in the history

  6. fixed documentation for botocore and boto3

    @philschmid
    philschmid committed Jan 19, 2021
    Configuration menu
    Copy the full SHA
    187e01d View commit details
    Browse the repository at this point in the history

  7. Merge branch 'master' of git://github.com/huggingface/datasets

    @philschmid
    philschmid committed Jan 19, 2021
    Configuration menu
    Copy the full SHA
    7785f90 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2021

  1. Update docs/source/filesystems.rst 

    Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
    @philschmid @lhoestq
    philschmid and lhoestq committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    926f31c View commit details
    Browse the repository at this point in the history

  2. Update docs/source/filesystems.rst 

    Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
    @philschmid @lhoestq
    philschmid and lhoestq committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    22b33d7 View commit details
    Browse the repository at this point in the history

  3. Update docs/source/filesystems.rst 

    Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
    @philschmid @lhoestq
    philschmid and lhoestq committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    72440ba View commit details
    Browse the repository at this point in the history

  4. Update src/datasets/arrow_dataset.py 

    Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
    @philschmid @lhoestq
    philschmid and lhoestq committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    ea273a8 View commit details
    Browse the repository at this point in the history

  5. Update src/datasets/arrow_dataset.py 

    Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
    @philschmid @lhoestq
    philschmid and lhoestq committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    5359003 View commit details
    Browse the repository at this point in the history

  6. Update src/datasets/filesystems/__init__.py 

    Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
    @philschmid @lhoestq
    philschmid and lhoestq committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    fd106e4 View commit details
    Browse the repository at this point in the history

  7. Update src/datasets/filesystems/s3filesystem.py 

    Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
    @philschmid @lhoestq
    philschmid and lhoestq committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    878f8b7 View commit details
    Browse the repository at this point in the history

  8. Update src/datasets/filesystems/s3filesystem.py 

    Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
    @philschmid @lhoestq
    philschmid and lhoestq committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    0b1a2f8 View commit details
    Browse the repository at this point in the history

  9. Update src/datasets/load.py 

    Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
    @philschmid @lhoestq
    philschmid and lhoestq committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    a3bebd5 View commit details
    Browse the repository at this point in the history

  10. removed unnecessary @mock_s3

    @philschmid
    philschmid committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    eb69cdb View commit details
    Browse the repository at this point in the history

  11. Update docs/source/filesystems.rst 

    Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
    @philschmid @lhoestq
    philschmid and lhoestq committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    8b7cd48 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2021

  1. Update docs/source/filesystems.rst 

    Co-authored-by: Julien Chaumond <chaumond@gmail.com>
    @philschmid @julien-c
    philschmid and julien-c committed Jan 26, 2021
    Configuration menu
    Copy the full SHA
    9d7f5c6 View commit details
    Browse the repository at this point in the history

  2. Update src/datasets/filesystems/s3filesystem.py 

    Co-authored-by: Julien Chaumond <chaumond@gmail.com>
    @philschmid @julien-c
    philschmid and julien-c committed Jan 26, 2021
    Configuration menu
    Copy the full SHA
    8514bee View commit details
    Browse the repository at this point in the history

  3. Update docs/source/processing.rst 

    Co-authored-by: Julien Chaumond <chaumond@gmail.com>
    @philschmid @julien-c
    philschmid and julien-c committed Jan 26, 2021
    Configuration menu
    Copy the full SHA
    a8738ca View commit details
    Browse the repository at this point in the history