v1.13.1

Notes

flytekitplugins-kfpytorch

The distributed pytorch and distributed elastic-pytorch tasks in flytekitplugins-kfpytorch by default increase the shared memory limit by mounting an emptyDir volume with medium Memory to to /dev/shm as this is almost always required when working with torch multiprocessing (e.g. multi-processed data loader workers or local worker group in distributed training). To disable this, pass increase_shared_mem=False to task_config=PyTorch/Elastic. Elastic tasks now also set a default join timeout of 15 minutes to prevent timeouts when some worker pods require a node scale-up. This setting can be modified via task_config=Elastic(rdzv_configs{...}).

What's Changed

• Add myself to code owners of flytekit-kf-pytorch by @fg91 in #2556
• Add run_policy to ElasticTask by @bgedik in #2555
• Added entrypoint to imagespec and default builder by @pryce-turner in #2553
• Set default width if fail to get terminal size by @pingsutw in #2558
• Disable rich traceback for papermill plugin by @pingsutw in #2559
• fix datetime in eager workflow by @novahow in #2541
• Fix CSS with Flyte decks by @thomasjpfan in #2565
• Fix FlyteDirectory on Azure by @Tom-Newton in #2564
• Default image builder: Creates HOME if it does not exists by @thomasjpfan in #2562
• Removes isodate requirement by @thomasjpfan in #2568
• Read FLYTE_SDK_DEV_LOGGING_LEVEL from env by @pingsutw in #2571
• print native input/output when cache hit by @pingsutw in #2567
• Fix: Set OMP_NUM_THREADS by default in Elastic by @fellhorn in #2569
• Use logging level in env for default dev logger by @pingsutw in #2572
• Bump certifi from 2023.7.22 to 2024.7.4 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in #2566
• Allow for flytekit version to be specified in default image builder by @thomasjpfan in #2563
• Removes jinja2 dependency by @thomasjpfan in #2570
• Improve error when missing type annotations in task/workflow by @pingsutw in #2549
• Rename databricks task type by @pingsutw in #2574
• remove upper bound of plugin dependencies for flytekit-sqlalchemy by @Mecoli1219 in #2517
• Remove flytekitplugins-deck-standard in Dockerfile by @Future-Outlier in #2582
• Bump certifi from 2024.2.2 to 2024.7.4 by @dependabot in #2581
• handle existing sagemaker deployments gracefully by @samhita-alla in #2400
• Use /opt/micromamba for default image builder by @thomasjpfan in #2578
• Improve error message for missing return by @pingsutw in #2551
• Override Dataclass Serialization/Deserialization Behavior for FlyteTypes by mashumaro by @Future-Outlier in #2554
• Fix FlyteDirectory docs rendering by @ppiegaze in #2584
• Make openai_organization Optional in OpenAI plugins by @Future-Outlier in #2585
• Adds comet-ml plugin by @thomasjpfan in #2550
• Replace super(AsyncAgentBase) with super(DatabricksAgent) by @pingsutw in #2590
• add kubernetes package to official docker image again by @flixr in #2589
• Feat: Improve UX of pytorch-elastic plugin by configuring reasonable defaults by @fg91 in #2543
• Improved date parsing in CLI with meta variables by @kumare3 in #2595
• truncate sagemaker agent outputs and automate idempotence token handling by @samhita-alla in #2588
• [BUG] support setting extended resources for array node map tasks by @pvditt in #2592
• Fix DataClass Json Schema Error for get literal type method by @Future-Outlier in #2587
• Sagemaker dict determinism by @samhita-alla in #2597
• refactor(core): Enhance return type extraction logic by @pingsutw in #2598
• Feat: Make exception raised by external command authenticator more actionable by @fg91 in #2594
• Fix: Properly re-raise non-grpc exceptions during refreshing of proxy-auth credentials in auth interceptor by @fg91 in #2591
• validate idempotence token length in subsequent tasks by @samhita-alla in #2604
• Add nvidia-l4 gpu accelerator by @eapolinario in #2608
• eliminate redundant literal conversion for Iterator[JSON] type by @samhita-alla in #2602
• [FlyteSchema] Fix numpy problems by @Future-Outlier in #2619
• add nim plugin by @samhita-alla in #2475

Full Changelog: v1.13.0...v1.13.1