Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix HF conversion script to upload to S3 after editing the files to be HF compatible #136

Merged
merged 2 commits into from
May 16, 2023

Conversation

dakinggg
Copy link
Collaborator

@dakinggg dakinggg commented May 15, 2023

There is probably some more cleanup that can be done to this script, but this change should at least fix the bug. The bug was that an incomplete folder would be what gets uploaded to cloud.

@dakinggg dakinggg marked this pull request as ready for review May 15, 2023 22:29
@dakinggg dakinggg requested a review from abhi-mosaic May 15, 2023 22:29
Copy link
Member

@abhi-mosaic abhi-mosaic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving to fix the bug. Not 100% sure I understand how moving the object store creation fixes the issue. But as long as you've tested it, it LGTM.

@dakinggg dakinggg merged commit 7cc9e04 into mosaicml:main May 16, 2023
6 checks passed
@dakinggg dakinggg deleted the fix-hf-s3-upload branch June 1, 2023 00:11
bmosaicml pushed a commit that referenced this pull request Jun 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants