Port deviceManager to windows container manager to enable GPU access #80917

aarnaud · 2019-08-02T15:39:15Z

What type of PR is this?
/kind feature

What this PR does / why we need it:

Port the devicemanager to Windows node to allow device plugins.

An important device plugin (not part of this PR) is the directx device plugin for GPU access: https://github.com/aarnaud/k8s-directx-device-plugin

This is required for machine learning, high-performance computing and computer graphics under Windows using the GPU.

Does this PR introduce a user-facing change?:

NONE

k8s-ci-robot · 2019-08-02T15:39:16Z

Welcome @aarnaud!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2019-08-02T15:39:18Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2019-08-02T15:39:23Z

Hi @aarnaud. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

thomacos · 2019-09-05T11:54:19Z

@derekwaynecarr @sjenning @RenaudWasTaken
We would be also very interested into that feature as soon as possible.

aarnaud · 2019-09-05T12:40:15Z

I hope this PR would be merge.

@thomacos Until merge you can find some build here : https://github.com/aarnaud/k8s-directx-device-plugin/releases

thomacos · 2019-09-05T12:51:52Z

I hope this PR would be merge.

@thomacos Until merge you can find some build here : https://github.com/aarnaud/k8s-directx-device-plugin/releases

That's great!
We just started into looking compiling kubelet.exe (and k8s-directx-device-plugin.exe) ourself, but this makes it way easier for us.
Thanks a lot!

Nevertheless, let's hope this PR finds its way into the official releases, too.

aarnaud · 2019-09-05T13:23:57Z

Also depends of your windows version you have to edit c:\k\start-kubelet.ps1

...
ipmo $helper
 
#Add this line below
Remove-Item c:\var\lib\kubelet\device-plugins\kubelet.sock -ErrorAction Ignore
 
if ($RegisterOnly.IsPresent)
{
    RegisterNode
    exit
}
...

There is a lite issue for Linux Socket on Windows c:\var\lib\kubelet\device-plugins\kubelet.sock

Tested on windows server 2019 with docker 19.03

thomacos · 2019-09-05T17:07:52Z

Works like charm!
Thanks again!

aarnaud · 2019-10-29T07:01:29Z

hi, this PR is ok for you? when can i expect to be merged

matthyx · 2019-11-15T15:04:35Z

/cc
/ok-to-test

michmike · 2020-07-16T20:49:51Z

@yujuhong or @dchen1107 can you please approve?
/assign @dchen1107

thomacos · 2020-07-16T22:55:14Z

@yujuhong or @dchen1107 can you please approve?
/assign @dchen1107

Would sending a candy box to @yujuhong and @dchen1107 help getting their attention?

thomacos · 2020-07-17T12:56:28Z

@klueska
Can you approve this PR?
It falls into your expertise area of device plugins and GPU access.
It is already approved by sig-windows.
The failing tests are unrelated to this PR and are not blocking, see #80917 (comment).

klueska · 2020-07-17T15:34:50Z

The PR itself is pretty straightforward and isolated to a windows-specific file.

My only hesitations in approving are therefore:

I'm not very familiar with windows, so I would want confirmation from sig-windows that this is approved on their end
There are no tests associated with this change, so it's hard to verify that it actually does what is expected

From @ddebroy 's comment above is seems that people are generally OK with the lack of tests so long as #88554 is worked on soon. I am OK with this too.

Can someone from sig-windows drop a final comment about their approval?

michmike · 2020-07-17T18:50:01Z

The PR itself is pretty straightforward and isolated to a windows-specific file.

My only hesitations in approving are therefore:

I'm not very familiar with windows, so I would want confirmation from sig-windows that this is approved on their end

There are no tests associated with this change, so it's hard to verify that it actually does what is expected

From @ddebroy 's comment above is seems that people are generally OK with the lack of tests so long as #88554 is worked on soon. I am OK with this too.

Can someone from sig-windows drop a final comment about their approval?

hi @klueska , sig-windows approves this. Deep Debroy is the TL for sig-windows and i am the co-chair for sig-windows

klueska · 2020-07-17T18:53:52Z

/approve

k8s-ci-robot · 2020-07-17T18:54:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aarnaud, klueska, michmike, thomacos

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/cm/OWNERS~~ [klueska]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fejta-bot · 2020-07-17T21:24:31Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

thomacos · 2020-07-17T21:29:15Z

@klueska Thank you for the quick action!

k8s-ci-robot · 2020-07-18T00:22:10Z

@aarnaud: The following tests failed, say /retest to rerun all failed tests:

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

fejta-bot · 2020-07-18T02:39:15Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

thomacos · 2020-07-19T17:51:49Z

It seems that this PR will make it into the 1.19.0-rc.2 release.
Great!
GPU access under Windows is a very useful feature for many use cases.
Thanks to all, especially of course to @aarnaud, both for this PR and the directx device plugin!
And yes, the next step is #88554.

liggitt · 2020-07-20T20:16:40Z

Opened #93263 with a revert of this since it fails windows kubelet startup

liggitt · 2020-07-21T03:09:31Z

revert is merged, please verify azure CI jobs pass as part of reintroducing this

thomacos · 2020-07-21T09:27:39Z

So it seems to me that going back to the separate default paths for Windows is the natural right choice:

It easily fixes the issue.
It reflects the fundamental difference of what constitutes an absolute path under Windows versus Linux.
It is exactly what was approved by @ddebroy from sig-windows, when he said in his Feb 14: "Is it possible to separate out only the DevicePluginPath const into the constants_windows.go and keep the rest of the constants in constants.go?".

So what are the next steps?
@aarnaud creating a fresh new pull request? Or can an existing PR reintroduced?

thomacos · 2020-07-21T09:50:07Z

/reopen

k8s-ci-robot · 2020-07-21T09:50:19Z

@thomacos: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

thomacos · 2020-07-21T09:56:55Z

@aarnaud, what do you think about the following?
Can you go back to the separate default paths for Windows as I suggested in #80917 (comment).
And then /reopen this pull request?

Obviously this PR won't get into 1.19 anymore. But Code Thaw is today. So I would suggest to aim for getting the fixed version approved and merged as soon as possible before it gets forgotten again.

k8s-ci-robot · 2020-07-21T11:17:27Z

@aarnaud: Failed to re-open PR: state cannot be changed. The pull request cannot be reopened.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 2, 2019

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 2, 2019

k8s-ci-robot requested review from RenaudWasTaken and sjenning August 2, 2019 15:41

aarnaud force-pushed the windows-devicemanager branch from 01edb84 to 4ce09a7 Compare August 2, 2019 15:45

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Aug 2, 2019

k8s-ci-robot requested a review from matthyx November 15, 2019 15:04

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 15, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 16, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 17, 2020

k8s-ci-robot merged commit 242f3d9 into kubernetes:master Jul 18, 2020

jsturtevant mentioned this pull request Jul 20, 2020

Kueblet fails to start due to device plugin manager on windows #93262

Closed

liggitt mentioned this pull request Jul 20, 2020

Fix windows kubelet startup #93263

Merged

jsturtevant mentioned this pull request Jul 21, 2020

Presubmit tests for Windows and release blocking tests #93276

Open

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Jul 21, 2020

aarnaud mentioned this pull request Jul 21, 2020

Port deviceManager to windows container manager to enable GPU access #93285

Merged

jsturtevant mentioned this pull request Jul 21, 2020

sig-windows flaky tests #92771

Closed

github-actions bot mentioned this pull request Jul 22, 2020

Week Ending July 19, 2020 dev-obs/actus#196

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port deviceManager to windows container manager to enable GPU access #80917

Port deviceManager to windows container manager to enable GPU access #80917

aarnaud commented Aug 2, 2019 •

edited by liggitt

k8s-ci-robot commented Aug 2, 2019

k8s-ci-robot commented Aug 2, 2019

k8s-ci-robot commented Aug 2, 2019

thomacos commented Sep 5, 2019

aarnaud commented Sep 5, 2019

thomacos commented Sep 5, 2019

aarnaud commented Sep 5, 2019

thomacos commented Sep 5, 2019

aarnaud commented Oct 29, 2019

matthyx commented Nov 15, 2019

michmike commented Jul 16, 2020

thomacos commented Jul 16, 2020

thomacos commented Jul 17, 2020 •

edited

klueska commented Jul 17, 2020

michmike commented Jul 17, 2020

klueska commented Jul 17, 2020

k8s-ci-robot commented Jul 17, 2020

fejta-bot commented Jul 17, 2020

thomacos commented Jul 17, 2020

k8s-ci-robot commented Jul 18, 2020 •

edited

fejta-bot commented Jul 18, 2020

thomacos commented Jul 19, 2020 •

edited

liggitt commented Jul 20, 2020

liggitt commented Jul 21, 2020

thomacos commented Jul 21, 2020

thomacos commented Jul 21, 2020

k8s-ci-robot commented Jul 21, 2020

thomacos commented Jul 21, 2020 •

edited

k8s-ci-robot commented Jul 21, 2020

Port deviceManager to windows container manager to enable GPU access #80917

Port deviceManager to windows container manager to enable GPU access #80917

Conversation

aarnaud commented Aug 2, 2019 • edited by liggitt

k8s-ci-robot commented Aug 2, 2019

k8s-ci-robot commented Aug 2, 2019

k8s-ci-robot commented Aug 2, 2019

thomacos commented Sep 5, 2019

aarnaud commented Sep 5, 2019

thomacos commented Sep 5, 2019

aarnaud commented Sep 5, 2019

thomacos commented Sep 5, 2019

aarnaud commented Oct 29, 2019

matthyx commented Nov 15, 2019

michmike commented Jul 16, 2020

thomacos commented Jul 16, 2020

thomacos commented Jul 17, 2020 • edited

klueska commented Jul 17, 2020

michmike commented Jul 17, 2020

klueska commented Jul 17, 2020

k8s-ci-robot commented Jul 17, 2020

fejta-bot commented Jul 17, 2020

thomacos commented Jul 17, 2020

k8s-ci-robot commented Jul 18, 2020 • edited

fejta-bot commented Jul 18, 2020

thomacos commented Jul 19, 2020 • edited

liggitt commented Jul 20, 2020

liggitt commented Jul 21, 2020

thomacos commented Jul 21, 2020

thomacos commented Jul 21, 2020

k8s-ci-robot commented Jul 21, 2020

thomacos commented Jul 21, 2020 • edited

k8s-ci-robot commented Jul 21, 2020

aarnaud commented Aug 2, 2019 •

edited by liggitt

thomacos commented Jul 17, 2020 •

edited

k8s-ci-robot commented Jul 18, 2020 •

edited

thomacos commented Jul 19, 2020 •

edited

thomacos commented Jul 21, 2020 •

edited