-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add LocalStorage Implementation #13981
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR @masonmenges! In addition to the comments I left, can you add some more tests for the new LocalStorage
class and tests covering the changes to create_storage_from_url
?
src/prefect/runner/storage.py
Outdated
async def pull_code(self): | ||
""" | ||
Pulls the contents of the configured repository to the local filesystem. | ||
""" | ||
self._logger.debug( | ||
"Checking local file path '%s'...", | ||
self.destination, | ||
) | ||
|
||
local_dir = self.destination | ||
|
||
if local_dir.exists(): | ||
os.chdir(local_dir) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For LocalStorage
, we want pull_code
to be a noop. Changing the directory here will change the current working directory for the entire process, which would break other flows served by the Runner
if they're in other directories.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, this should just pass now and not doing anything else.
src/prefect/runner/storage.py
Outdated
@@ -582,8 +655,12 @@ def create_storage_from_url( | |||
parsed_url = urlparse(url) | |||
if parsed_url.scheme == "git" or parsed_url.path.endswith(".git"): | |||
return GitRepository(url=url, pull_interval=pull_interval) | |||
else: | |||
elif parsed_url.scheme in fsspec.available_protocols(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we might want to check for file
and local
schemes before this check to ensure we don't create a RemoteStorage
instance if a user passes in file://path/to/flow.py
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm splitting the path that gets passed in now on these characters "://" when the scheme is local or file but I'm not sure if there's maybe a better way to handle that though since I think that can get kinda tricky when you start including drive letters into the mix for windows paths.
@desertaxle Ok I think I've got most of this in a good spot now and added some more tests for local storage, I'm still a little unsure of how I'm handling the local and file schemes though so if you have any thoughts there I would greatly appreciate it. |
src/prefect/runner/storage.py
Outdated
""" | ||
step = { | ||
"prefect.deployments.steps.set_working_directory": { | ||
"directory": self.destination |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The destination needs to be cast to a string to ensure it can be saved correctly in the DB.
"directory": self.destination | |
"directory": str(self.destination) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
couple comments!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Please be sure to port these changes to main
also!
This pr introduces a LocalStorage implementation to be used with RunnerDeployments for flow deployed in a local process, this closes a gap sometimes felt by users when using the deploy method to deploy prefect flows instead of serve or through a prefect yaml.
closes #13499
Example
Checklist
<link to issue>
"maintenance
,fix
,feature
,enhancement
,docs
.For documentation changes:
mint.json
for files that are removed or renamed.For new functions or classes in the Python SDK:
docs/mint.json
navigation.