retry nvidia-device-plugin.sh when failed

[qqshfox]

What this PR does / why we need it:

nvidia-device-plugin.sh could failed due to network issues, apt issues, etc. But the oneshot service can't be restarted on failure.

Service has Restart= setting other than no, which isn't allowed for Type=oneshot services. Refusing.

Changed to Type=simple and set Restart=on-failure.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

Special notes for your reviewer:

[k8s-ci-robot]

Hi @qqshfox. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[joshbranham]

@qqshfox can you please fill out the description with the PR template details?

[qqshfox]

@qqshfox can you please fill out the description with the PR template details?

Updated.

[joshbranham]

Thanks @qqshfox

/ok-to-test

[joshbranham]

/retest

[joshbranham]

@qqshfox will making this a full-blown service be problematic, wherein it will run repeatedly? I can't seem to find the exact documentation on what happens if a systemd unit of type simple exits with a 0. Have you tested this? Looks like you can do oneshot and Restart: on-failure now (depending on version)

systemd/systemd#13754

[qqshfox]

@qqshfox will making this a full-blown service be problematic, wherein it will run repeatedly? I can't seem to find the exact documentation on what happens if a systemd unit of type simple exits with a 0. Have you tested this? Looks like you can do oneshot and Restart: on-failure now (depending on version)

systemd/systemd#13754

Yes. It's tested in our production clusters. It'll be inactive when a unit exits with a 0.

# systemctl status nvidia-device-plugin
● nvidia-device-plugin.service - Prepare AWS GPU instances for Nvidia Kubernetes Device Plugin
   Loaded: loaded (/nvidia-device-plugin/nvidia-device-plugin.service; linked; vendor preset: enabled)
   Active: inactive (dead)

The systemd in the official kops images is still 232 which obviously doesn't support the new behavior.

[joshbranham]

Great, thanks!

/lgtm

[rifelpet]

/test pull-kops-verify-staticcheck

[granular-ryanbonham]

/approved

[rifelpet]

I believe @granular-ryanbonham meant:
/approve
:)

looks good to me too

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qqshfox, rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rifelpet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[johngmyers]

/retest