Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow specifying extra volumes #129

Merged
merged 11 commits into from
Jun 10, 2022
Merged

Allow specifying extra volumes #129

merged 11 commits into from
Jun 10, 2022

Conversation

marrrcin
Copy link
Contributor

@marrrcin marrrcin commented Jun 2, 2022

Description

  • add ability to specify additional volumes, especially memory-backed emptyDir to address

Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

while running PyTorch with multiple GPUs/workers using kedro-kubeflow.

  • refactor configuration to Pydantic, with some simplifications related to defaults ( ⚠️ this PR changes the behaviour of resources requests/limits - they will be always set to defaults in the compiled pipeline, but it's not breaking IMHO).
PR Checklist

kedro_kubeflow/config.py Outdated Show resolved Hide resolved
Copy link
Contributor

@szczeles szczeles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic work!!!!

@marrrcin marrrcin merged commit 0fe1a3b into develop Jun 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants