-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature request] copy the source code to xps dir upon launch (only *.py and *.yaml files is a good start) #4
Comments
you mean in order to execute this code only or as an indication of what code was being ran ? |
Here is what I use. I think it'd be great to add it to dora default, because it facilitates working with old setups/config to .e.g understand whether/why an experiment does not work anymore. import git
def save_git(git_path):
"""Backup git repo to path for replicability."""
curr_git = git.Repo(__file__, search_parent_directories=True)
curr_date = curr_git.commit().committed_date
bckp_date = 0
if git_path.exists():
bckp_git = git.Repo(git_path)
bckp_date = bckp_git.commit().committed_date
if bckp_date < curr_date:
if git_path.exists():
shutil.rmtree(git_path)
curr_git.clone(git_path) |
@adefossez I mean both for execution and as an indication. |
Execution would be cool, although hard to make in a reliable manner. @kingjr I think you always assume you are working from a clean commit in that case. This is rarely the case in practice. |
See #9 This allows both to keep track of the code used for an XP, as well as protect a pending or preempted XP from future code changes. However, if an XP is stopped and rescheduled (not preempted), the clone will be updated and I do not try to keep track of the different versions used across the lifetime of an XP, only the last one (as this would be way too complex). |
Closing the task now that the option (actually |
This feature is great! Though, we have a blocker for adopting it. We have a rather large repo (>1Gb, many files), and are looking to launch many small It seems the current Thanks, P.S. in the case of amended grids, I suppose more "central copies" would be added in |
Definitely for 1k+ this is going to be painfully slow. I could make shared clone for all jobs scheduled at the same time. The only limitation in that case is that the clone would be stuck at one commit, because it is almost impossible to reference count properly how many times it is used and if it is safe to change its current commit. But this could be solved either with a |
thanks for the fast response! If I'm understanding correctly, it might look like the following? Let me know if I'm misunderstanding.
|
exactly ! the only thing now is that at some point |
awesome! yes I think that would definitely be ok (and that edge case indeed sounds acceptable haha). This version of things would be so cool :). Can't wait to use it if you think it's doable! |
No description provided.
The text was updated successfully, but these errors were encountered: