Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Infra for Training #124

Merged
merged 13 commits into from
Jun 14, 2024
Merged

Data Infra for Training #124

merged 13 commits into from
Jun 14, 2024

Commits on Apr 24, 2024

  1. ported over necessary files for file download from benjieg/wip-traini…

    …ng-fork. implemented and tested a dataset only arg version of download.py. Need to reconsider the other arguments.
    Benjamin Genchel authored and bgenchel committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    f1eaa65 View commit details
    Browse the repository at this point in the history
  2. added files for downloading and converting datasets, and a guitarset …

    …to test the ecosystem. Tests are written but tox is currently not passing. Many files reformatted by black.
    bgenchel committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    b6229dd View commit details
    Browse the repository at this point in the history
  3. added sox, apache_beam and mirdata to many more places in tox.ini, py…

    …project.toml and tox.yml, which needs review. Added .jams files to MANIFEST.in to accomodate test file for guitarset. Added check for pytest run to guitarset download to avoid unnecessary full download during testing, need review. tox passing.
    bgenchel committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    bfede5c View commit details
    Browse the repository at this point in the history
  4. simplified / created train environment and testenv in tox.ini and pyp…

    …roject.toml. Moved all tests relating to data into their own test directory. Updated path to resources accordingly.
    bgenchel committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    4975d82 View commit details
    Browse the repository at this point in the history
  5. black formatting, flake linting, mypy corrections across files. Added…

    … test for tf_example_serialization, added / corrected tests for other data / train files.
    bgenchel committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    5f78d3a View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. modify .github/workflows/tox.ini so os name check matches os names li…

    …sted, add .zip files to manifest.in
    bgenchel committed May 1, 2024
    Configuration menu
    Copy the full SHA
    ad479f2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3296466 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. addressing pr comments - undoing incorrect reformat, adding entry for…

    … sox in CONTRIBUTING.md, add copyright notice to download.py, remove apache_beam from general dependencies.
    bgenchel committed May 8, 2024
    Configuration menu
    Copy the full SHA
    2c3d4fd View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. Switch over arguments and defaults to use PortableRunner in emulation of

    Dataflow runner, using our Docker image. Added Dockerfile and .dockerfileignore to Manifest.in
    bgenchel-avail committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    4225ade View commit details
    Browse the repository at this point in the history
  2. added an entrypoint for downloading datasets, figured out how to pass…

    … unknown args / pipeline args to the pipeline along with keyword args, added a README.md in the data folder.
    bgenchel-avail committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    1c58725 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Configuration menu
    Copy the full SHA
    d218e1c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    23f7245 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. tox.ini train -> data

    bgenchel-avail committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    a6634f3 View commit details
    Browse the repository at this point in the history