Do not overwrite prompt artifacts

[mynhardtburger]

Issue: #51

Updates load_prompt_artifacts() to only copy new artifacts.
New artifacts are those which themselves (eg. foo.pt) or their swap/in-progress variant (eg. foo.pt.swp) already exist in the prompt_dir location.

Copying is done in two stages:

1. Copy file with the .swp extension appended to indicate a copy is in progress
2. Rename the file by removing the .swp extension after the copy is completed

[gabe-l-hart]

Thanks for digging into this one Mynhardt! My main question with how it's currently written is what advantage the .swp/rename offers over simply not copying if the named file itself already exists. Depending on the filesystem implementation, this could offer the advantage that the named file itself comes into existence atomically during the rename, but for other filesystems (most notably s3fs), don't guarantee an atomic rename, so that wouldn't always be the case.

[gabe-l-hart]

Thanks for fixing this!

[gabe-l-hart]

One day I'll actually learn pathlib!

[mynhardtburger]

I really like the path building using / :)

[gabe-l-hart]

I'm curious, do you happen to know if iterdir lazily evaluates the contents of the dir, or if it takes a proactive os.listdir and then iterates that (i.e. if the list changes during iteration, what happens)? Not particularly important in this instance, just curious.

[mynhardtburger]

iterdir() is a wrapper for os.listdir. So should be functionally the same:

    def iterdir(self):
        """Iterate over the files in this directory.  Does not yield any
        result for the special paths '.' and '..'.
        """
        for name in os.listdir(self):
            yield self._make_child_relpath(name)

[mynhardtburger]

Thanks for digging into this one Mynhardt! My main question with how it's currently written is what advantage the .swp/rename offers over simply not copying if the named file itself already exists. Depending on the filesystem implementation, this could offer the advantage that the named file itself comes into existence atomically during the rename, but for other filesystems (most notably s3fs), don't guarantee an atomic rename, so that wouldn't always be the case.

Atomic rename guarantees that we don't see multiple states for the same file at the same time. But in our case it doesn't matter much:
Our primary goal is that foo.pt is complete and not in-progress. Whether a foo.pt.swp exists alongside it doesn't matter. That would be the non-atomicy showing itself. s3fs-fuse performs a server side copy for a rename. So, I assume, a pod crash won't impact that action and therefore we can rely on a foo.pt file to always reflect a complete file. Eventually the foo.pt.swp will disappear for all the pods.

On file systems which has atomic rename there is no downside.

If 2 pods happens to start a download for the same foo.pt at the same time, the os.replace() will silently override the older version irrespective of the OS.

[gabe-l-hart]

s3fs-fuse performs a server side copy for a rename. So, I assume, a pod crash won't impact that action and therefore we can rely on a foo.pt file to always reflect a complete file.

Great! This makes a lot of sense

[evaline-ju]

LGTM! Thanks for the contribution!