dont depend on docker for ECR #2
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: adriano <adriano@exa.ai>
jld-adriano
changed the title
Author
|
i tested and it works! both for missing and existing images |
devin-ai-integration bot
added a commit
that referenced
this pull request
Dec 12, 2025
…nv vars This fixes bug #1 where overriding nproc_per_node via with_overrides would not change the number of processes for single-node elastic tasks. For single-node elastic tasks (task_type='python-task'), the _execute method reads PET_NPROC_PER_NODE, PET_NNODES, PET_MAX_RESTARTS, and PET_MONITOR_INTERVAL from environment variables. However, these env vars were never being set in the task template during serialization. The fix adds an environment property override to PytorchElasticFunctionTask that includes the elastic config as environment variables. This ensures that when task_config is modified via with_overrides, the elastic configuration is correctly passed to the pod via environment variables. Combined with the previous fix (dynamic task_type property), this now fully supports: - Bug #1: single-node (1 proc) -> single-node (multiple procs) override - Bug #2: single-node (1 proc) -> multi-node (multiple procs) override Co-Authored-By: carlos@exa.ai <carlosmarques.personal@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
This PR introduces a more efficient and reliable way to check for image existence, especially for images hosted on Amazon ECR. Previously, Flytekit relied solely on Docker to check if an image existed, which can be slow or fail if the Docker daemon is unavailable. By directly querying ECR when applicable, we reduce reliance on Docker and improve performance.
What changes were proposed in this pull request?
This PR implements a new mechanism to check for image existence directly in Amazon ECR before falling back to Docker.
is_ecr_registry,check_aws_cli_and_creds, andcheck_ecr_image_existsinflytekit/image_spec/image_spec.pyto:aws ecr describe-images.ImageSpec.exist(): Theexist()method now:docs/ecr_image_check.mddetailing the feature, its benefits, requirements, and usage.How was this patch tested?
Comprehensive unit tests were added in
tests/flytekit/unit/core/image_spec/test_ecr_check.pyto cover:Check all the applicable boxes
Slack Thread