Skip to content

Conversation

@d4l3k
Copy link
Member

@d4l3k d4l3k commented May 27, 2021

This adds a simple two stage pipeline that can run on KFP. The first stage downloads the tiny imagenet dataset and normalizes the data. The second stage then loads that dataset and trains on it.

Test plan:

python datapreproc/main.py --input_path http://localhost:8000/tiny-imagenet-200.zip --output_path s3://torchx-test/tristanr/data_path/

docker build -t torchx/examples .
docker run --env-file /tmp/aws-env -it --name torchx-classy-vision --rm torchx-classy-vision:latest python3 datapreproc/main.py --output_path s3://torchx-test/tristanr/data_path/
docker tag torchx/examples:latest 495572122715.dkr.ecr.us-west-2.amazonaws.com/torchx/examples:latest
docker push 495572122715.dkr.ecr.us-west-2.amazonaws.com/torchx/examples:latest
./kfp_pipeline.py --data_path s3://torchx-test/tristanr/data_path/ --output_path s3://torchx-test/tristanr/output_path/

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 27, 2021
@facebook-github-bot
Copy link
Contributor

@d4l3k has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@d4l3k d4l3k changed the title examples/datapreproc: added simple data loading and preprocessing component examples/kfp_pipeline: add datapreproc component + a two stage pipeline May 27, 2021
@facebook-github-bot
Copy link
Contributor

@d4l3k has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@d4l3k has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@d4l3k has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@d4l3k has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@d4l3k has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@d4l3k has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@d4l3k merged this pull request in 523eb1a.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants