-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Agent Environment Hybrid Execution Model First Pass #670
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you also update the tutorial on Local Flow debugging
? There's a section at the end that discusses building a container locally and using from_file
to detect any serialization errors. Maybe just say "Let's use the example of a DockerEnvironment
because it's easiest and representative of the types of errors you may encounter in other environments" or something.
src/prefect/environments/kubernetes/docker/docker_on_kubernetes.py
Outdated
Show resolved
Hide resolved
…s.py Co-Authored-By: joshmeek <40716964+joshmeek@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Request for MORE DOCS but otherwise 👌
Integration tests proof of concept and with jest projects
Thanks for contributing to Prefect!
Please describe your work and make sure your PR:
CHANGELOG.md
(if appropriate)docs/outline.toml
for API reference docs (if appropriate)What does this PR change?
This PR introduces a whole overhaul of how environments are handled in Prefect. Mainly following the outline in PIN-3.
Introduced:
Does not introduce the DaskOnKubernetes environment discussed in PIN-3. Will be implemented in a later PR.
Why is this PR important?
Environments now have a core Environment class which two base environments inherit from—
LocalEnvironment
andDockerEnvironment
. This now allows us to make platform specific environments which abstract out infrastructure requirements to the environments themselves. For example, the Kubernetes environments subclass theDockerEnvironment
because they use Docker in their deployments. Theirsetup
functions may be used to create any other infrastructure requirements for the flow run. (e.g. creating a Dask scheduler and workers)Closes #599, closes #600, closes #601
Starts working towards #602
Note: base environments do not implement
setup
andexecute
because they are not platform dependent.Note: I welcome input on naming conventions for these environments
The Kubernetes environments are meant to be used in tandem with the Prefect k8s agent (https://github.com/PrefectHQ/k8s-agent). This means that all Kubernetes environments'
execute
function should send out the running of the flow in a Kubernetes job. This is important because the resource collection aspect of the agent looks for completed/failed jobs and then cleans up any resources tied to that job that were created during thesetup
phase.