Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pytorch): Add init container config to avoid DNS lookup failure #1493

Merged
merged 4 commits into from
Dec 2, 2021

Conversation

gaocegege
Copy link
Member

What this PR does / why we need it:

This PR is to add the init container, which checks if the master is already running.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #1482

@gaocegege
Copy link
Member Author

/hold

Signed-off-by: Ce Gao <ce.gao@outlook.com>
Signed-off-by: Ce Gao <ce.gao@outlook.com>
Signed-off-by: Ce Gao <ce.gao@outlook.com>
@coveralls
Copy link

coveralls commented Nov 30, 2021

Pull Request Test Coverage Report for Build 1519323487

  • 65 of 83 (78.31%) changed or added relevant lines in 4 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.8%) to 27.704%

Changes Missing Coverage Covered Lines Changed/Added Lines %
pkg/controller.v1/pytorch/envvar.go 13 15 86.67%
pkg/controller.v1/pytorch/pytorchjob_controller.go 3 7 42.86%
pkg/controller.v1/pytorch/initcontainer.go 48 60 80.0%
Totals Coverage Status
Change from base Build 1517009272: 0.8%
Covered Lines: 1693
Relevant Lines: 6111

💛 - Coveralls

Signed-off-by: Ce Gao <ce.gao@outlook.com>
@gaocegege
Copy link
Member Author

/hold cancel

Copy link
Member Author

@gaocegege gaocegege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/assign @zw0610

PyTorchInitContainerImageDefault = "alpine:3.10"
// PyTorchInitContainerTemplateFileDefault is the default template file for
// the pytorch init container.
PyTorchInitContainerTemplateFileDefault = "/etc/config/initContainer.yaml"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw PyTorchInitContainerImage in the original pytorch-operator, but not PyTorchInitContainerTemplateFileDefault. Do we really need this one?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation in pytorch operator hard coded a path. Here we use a CLI flag to make it configurable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I would suggest @Jeffwan or other pytorch-controller contributor to take a look before merge.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @Jeffwan

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does user have any requirements to custom init container? Could we make it completely transparent to users?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think they may update the resource requests. WDYT

/cc @johnugeorge

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd like to custom the init-container-image bescause of my project worked on a network limitations env,what shou i do?

pkg/controller.v1/pytorch/initcontainer.go Show resolved Hide resolved
pkg/controller.v1/pytorch/initcontainer.go Show resolved Hide resolved
pkg/config/config.go Show resolved Hide resolved
PyTorchInitContainerImageDefault = "alpine:3.10"
// PyTorchInitContainerTemplateFileDefault is the default template file for
// the pytorch init container.
PyTorchInitContainerTemplateFileDefault = "/etc/config/initContainer.yaml"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does user have any requirements to custom init container? Could we make it completely transparent to users?

@Jeffwan
Copy link
Member

Jeffwan commented Dec 1, 2021

It looks good to me. Feel free to unhold it when it's ready to merge.

/lgtm
/hold

@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaocegege, Jeffwan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@zw0610
Copy link
Member

zw0610 commented Dec 2, 2021

/unhold

@google-oss-prow google-oss-prow bot merged commit 4494c4c into kubeflow:master Dec 2, 2021
@gaocegege gaocegege deleted the config branch January 24, 2022 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[bug] Missing init container in PyTorchJob
5 participants