-
Notifications
You must be signed in to change notification settings - Fork 543
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Core feature] UX improvement: add --copy flag to pyflyte run for more targeted file copying during fast registration #5343
Comments
Having a For rapid iteration, I think building an image every time code changes is a bit more overhead. I'm thinking of this user story:
What do you think of having a new pyflyte run --remote --install-local |
Notes from discussion in contribs' sync: This could be incorporated into fast registration. A user would specify e.g. The benefit of this mechanism over ImageSpec would be that it can be used regardless of how images are built. Some organizations build images purely in CI instead of using ImageSpec. |
@fg91 For multiple local-packages, does the Python library have code that needs to be complied? For example, a Polars extension plugin that requires Rust to build. In any case, I think there is enough Python only packages out there where |
ImageSpec
and pyflyte run --remote
Note: another simple solution proposed by @thomasjpfan to this is a |
Can we discuss the differences between |
|
Motivation: Why do you think this is important?
Say I have the following flytekit project folder structure:
Where
src
is a pure python internal library that I can independently run/test. Then I want to import modules fromsrc
into my flytekittasks.py
andworkflows.py
files.When I build my
ImageSpec
, I currently have to role my own solution to installsrc
is a pip package (maybe via github).Then, when I want to quickly iterate on it with
pyflyte run --remote
, I use the--copy-all
flag to make suresrc
is in the PYTHONPATH of the container running my tasks.Goal: What should the final outcome look like, ideally?
Add a
--copy
flag, which can be specified multiple times for more targeted inclusion of files during fast registrationDescribe alternatives you've considered
The current workarounds for this are
--copy-all
at iteration time and pip installing via github at ImageSpec build time (there are probably more workarounds here).Propose: Link/Inline OR Additional context
This is part of a broader pain point of the flyte-maintainer-approved way of building a flytekit project. The tutorials/guides in our docs sort of assume you're working on a single file, or some simpler structure that doesn't include having a python project that's separate from flyte task/workflow modules.
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?
The text was updated successfully, but these errors were encountered: