Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

experiments: support binary and untracked files in experiment patches #4452

Merged
merged 1 commit into from Aug 24, 2020

Conversation

pmrowla
Copy link
Contributor

@pmrowla pmrowla commented Aug 24, 2020

Thank you for the contribution - we'll try to review it as soon as possible. πŸ™

Will fix #4451

@pmrowla pmrowla added bug Did we break something? A: experiments Related to dvc exp labels Aug 24, 2020
@pmrowla pmrowla self-assigned this Aug 24, 2020
Copy link
Contributor Author

@pmrowla pmrowla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One concern here would be that we could potentially be sticking large binary files into git, which git is bad at handling and why dvc exists in the first place. (But they would only go into the experiments workspace git repo, and not the user's main repo)

I'm not sure if there's a cleaner way for us to handle this situation other than documenting that best practices here would be to either mark these types of large binary intermediate artifacts/outputs as dvc outs, or to gitignore them.

@pmrowla
Copy link
Contributor Author

pmrowla commented Aug 24, 2020

As a potential future improvement, we could maybe consider doing something like generating a directory containing all of the untracked files generated by an experiment run (along with a manifest or something containing the original relpaths for the untracked files), and then storing that as an experiment-specific dvc-tracked directory. But in the meantime, this bugfix is needed for testing/development purposes.

@efiop efiop merged commit bfb98e9 into iterative:master Aug 24, 2020
@pmrowla pmrowla deleted the 4451-experiments-bin-patch branch August 24, 2020 12:56
@skshetry skshetry added the bugfix fixes bug label Aug 31, 2020
@pmrowla pmrowla mentioned this pull request Dec 4, 2020
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: experiments Related to dvc exp bug Did we break something? bugfix fixes bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

experiments: experiment git patching will fail if patch contains binary files
4 participants