Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git storage extra files #4767

Merged
merged 6 commits into from Jul 16, 2021
Merged

Git storage extra files #4767

merged 6 commits into from Jul 16, 2021

Conversation

zangell44
Copy link
Collaborator

@zangell44 zangell44 commented Jul 13, 2021

Summary

This PR adds support and documentation for loading external files using Git storage.

Changes

  • When extracting a flow from a file, pass the file path as __file__
  • Document Git storage use case where additional files are loaded

Importance

Closes #4754

Checklist

This PR:

  • adds new tests (if appropriate)
  • adds a change file in the changes/ directory (if appropriate)
  • updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)

@zangell44 zangell44 requested a review from zanieb as a code owner July 13, 2021 18:58
tests/utilities/test_storage.py Outdated Show resolved Hide resolved
src/prefect/utilities/storage.py Outdated Show resolved Hide resolved
src/prefect/utilities/storage.py Outdated Show resolved Hide resolved
@zanieb zanieb merged commit 8cc4478 into master Jul 16, 2021
@zanieb zanieb deleted the git-storage-extra-files branch July 16, 2021 19:00
@zanieb zanieb mentioned this pull request Jul 21, 2021
@chicago-joe
Copy link

Will this work with AWS CodeCommit storage as well?

@trymzet
Copy link
Contributor

trymzet commented Jul 22, 2021

This will break on a Jupyter notebook with a NameError because Jupyter does not populate __file__.

@zangell44
Copy link
Collaborator Author

@chicago-joe this will not work with CodeCommit, only Git storage at the moment. Happy to chat a bit further about CodeCommit enhancements or workarounds if you'd like to open an issue

@zangell44
Copy link
Collaborator Author

@trymzet you're correct that Jupyter notebooks (and other execution enironments) do not populate __file__. However, I'm not sure I understand the Prefect specific use case you're describing.

We set __file__ when the Flow is loaded from storage to allow users to access __file__ for loading additional files https://github.com/PrefectHQ/prefect/pull/4767/files#diff-8b9309d6d1ca7b8dd179ef8df5c1e02aa0011a639f2d1753f8eee09ca08cc133R87.

For any code executed outside of loading the flow from storage (which should happen when an agent picks up the flow run), __file__ is not used

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

document loading external files with Git storage
4 participants