Provision DSA devices and workqueues through initcontainer #680

ozhuraki · 2021-08-30T08:32:13Z

This PR is adds support for provisioning DSA devices and workqueues with accel-config utility for intel-dsa-plugin
through initcontainer.

Closes: #658

mythi · 2021-08-31T13:09:39Z

let's wait with this a bit still.

build/docker/intel-dsa-initcontainer.Dockerfile

pkg/controllers/dsa/controller.go

codecov-commenter · 2021-09-01T08:40:25Z

Codecov Report

Merging #680 (0bcd3b6) into main (3a35a75) will not change coverage.
The diff coverage is n/a.

❗ Current head 0bcd3b6 differs from pull request most recent head 1966090. Consider uploading reports for the commit 1966090 to get more accurate results

@@           Coverage Diff           @@
##             main     #680   +/-   ##
=======================================
  Coverage   60.06%   60.06%           
=======================================
  Files          32       32           
  Lines        2782     2782           
=======================================
  Hits         1671     1671           
  Misses       1028     1028           
  Partials       83       83

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3a35a75...1966090. Read the comment docs.

ozhuraki · 2021-09-01T08:40:26Z

@mythi

Added a license to DSA initcontainer Dockerfile. Will fix everything as you suggested, thanks for the input.

ozhuraki · 2021-09-01T14:07:57Z

@mythi

Switched initcontainer to debian:buster-slim
Added checksum for accel-config archive
Fixed libaccel-config.so symlink issue
Added InitImage validation to dsadeviceplugin_webhook.go

pkg/apis/deviceplugin/v1/dsadeviceplugin_types.go

ozhuraki · 2021-09-03T06:37:13Z

@mythi @rojkov

InitImage -> ProvisioningImage
Updated the description to "ProvisioningImage is an initcontainer image to configure and enable DSA devices and workqueues with accel-config utility"

ozhuraki · 2021-09-03T07:11:09Z

@mythi

Oops, missed your comments. I will update.

Can we do both easily with just one image? I'm also fine with one InitImage entry per CRD that is tasked to do both if what it needs to do can be customized somehow (e.g. not install some tools but provision only vs do both).

+1
I think having multiple images could be justified in cases where it's not possible due to non-technical reasons, etc (or where it would simplify the task). In this specific case, there aren't such yet.

ozhuraki · 2021-09-03T07:22:35Z

@mythi

ProvisioningImage -> InitImage

mythi

a few additional comments

build/docker/intel-dsa-initcontainer.Dockerfile

ozhuraki · 2021-09-22T12:41:33Z

Changes from the previous version:

Switched DSA initcontainer image to debian:sid-slim
Updated DSA initcontainer image to use libjson-c5
Upgrated accel-config to 3.4.1

TODO/WIP

Check if remount of /sys could be dropped
Enable all idle DSA devices

ozhuraki · 2021-09-30T14:01:31Z

Changes:

ENV -> ARG
accel-config -> 3.4.2
debian:sid-slim -> debian:unstable
Dropped remount
Dropped all volumes, using just /sys/devices
Added enabling of all idle DSA devices/workqueues (1 engine / 1 group / 1 wq (user/dedicated) )

demo/dsa-init.sh

build/docker/intel-dsa-initcontainer.Dockerfile

ozhuraki · 2021-10-01T07:35:19Z

Changes:

Dropped "readonlyrootfs" from DSA initcontainer deployment and operator
Added /sys/devices volume into DSA initcontainer deployment

ozhuraki · 2021-10-01T07:37:59Z

Changes:

Dropped /test from initcontainer

Signed-off-by: Oleg Zhurakivskyy <oleg.zhurakivskyy@intel.com>

Previously idxd kernel module instantiated some default DSA devices and workqueues on boot. This is a sample deployment that provisions DSA devices and workqueues for intel-dsa-plugin with accel-config utility through initcontainer. Signed-off-by: Oleg Zhurakivskyy <oleg.zhurakivskyy@intel.com>

ozhuraki · 2021-10-01T07:41:22Z

Changes:

Added "WORKDIR /dsa-init" to initcontainer

ozhuraki · 2021-10-01T07:49:04Z

Changes:

Added /sys/devices volume into DSA operator

bart0sh · 2021-10-01T07:56:02Z

@ozhuraki 3 more things would be nice to have:

documentation (overall description, deployment instructions, etc)
list of known issues
github issue(s) for the known issue(s) included into the next project milestone

Signed-off-by: Oleg Zhurakivskyy <oleg.zhurakivskyy@intel.com>

mythi · 2021-10-01T08:43:33Z

list of known issues

github issue(s) for the known issue(s) included into the next project milestone

I can cover these in the release notes but a short description and sample

$ kubectl apply -k deployments/dsa_plugin/overlays/dsa_initcontainer/

would indeed be good in the README

mythi

this works and is good IMO. short README update as @bart0sh suggested would be good still.

Signed-off-by: Oleg Zhurakivskyy <oleg.zhurakivskyy@intel.com>

ozhuraki · 2021-10-01T09:18:56Z

@mythi @bart0sh

Added a documentation, please take a look.

mythi · 2021-10-01T10:23:07Z

LGTM. Let's merge

rojkov

LGTM

ozhuraki requested review from bart0sh, kad, mythi and rojkov as code owners August 30, 2021 08:32

ozhuraki force-pushed the dsa-idxd-op branch 3 times, most recently from eddd733 to 37b559a Compare August 31, 2021 09:44

rojkov previously approved these changes Aug 31, 2021

View reviewed changes

mythi requested changes Aug 31, 2021

View reviewed changes

build/docker/intel-dsa-initcontainer.Dockerfile Outdated Show resolved Hide resolved

build/docker/intel-dsa-initcontainer.Dockerfile Outdated Show resolved Hide resolved

pkg/controllers/dsa/controller.go Outdated Show resolved Hide resolved

ozhuraki dismissed rojkov’s stale review via f94f8ed September 1, 2021 08:35

ozhuraki force-pushed the dsa-idxd-op branch from 37b559a to f94f8ed Compare September 1, 2021 08:35

ozhuraki force-pushed the dsa-idxd-op branch from f94f8ed to fedc6c9 Compare September 1, 2021 14:06

mythi reviewed Sep 2, 2021

View reviewed changes

pkg/apis/deviceplugin/v1/dsadeviceplugin_types.go Outdated Show resolved Hide resolved

ozhuraki force-pushed the dsa-idxd-op branch 2 times, most recently from c4da835 to 7bb227f Compare September 3, 2021 06:36

ozhuraki force-pushed the dsa-idxd-op branch from 7bb227f to cae6c07 Compare September 3, 2021 07:21

mythi requested changes Sep 16, 2021

View reviewed changes

build/docker/intel-dsa-initcontainer.Dockerfile Outdated Show resolved Hide resolved

build/docker/intel-dsa-initcontainer.Dockerfile Outdated Show resolved Hide resolved

build/docker/intel-dsa-initcontainer.Dockerfile Outdated Show resolved Hide resolved

ozhuraki force-pushed the dsa-idxd-op branch from cae6c07 to 9545146 Compare September 22, 2021 12:38

ozhuraki force-pushed the dsa-idxd-op branch 4 times, most recently from 59b7191 to 38dcbf1 Compare September 30, 2021 05:50

ozhuraki force-pushed the dsa-idxd-op branch 4 times, most recently from fc29cd7 to f4e7e6a Compare September 30, 2021 19:14

mythi reviewed Oct 1, 2021

View reviewed changes

demo/dsa-init.sh Outdated Show resolved Hide resolved

mythi reviewed Oct 1, 2021

View reviewed changes

build/docker/intel-dsa-initcontainer.Dockerfile Outdated Show resolved Hide resolved

ozhuraki force-pushed the dsa-idxd-op branch 2 times, most recently from 571a0db to 4369336 Compare October 1, 2021 07:32

ozhuraki force-pushed the dsa-idxd-op branch from 4369336 to 2da5f67 Compare October 1, 2021 07:37

ozhuraki added 2 commits October 1, 2021 10:40

images: Add intel-dsa-initcontainer

efb357d

Signed-off-by: Oleg Zhurakivskyy <oleg.zhurakivskyy@intel.com>

ozhuraki force-pushed the dsa-idxd-op branch from 2da5f67 to b62e2a1 Compare October 1, 2021 07:40

ozhuraki force-pushed the dsa-idxd-op branch from b62e2a1 to 1966090 Compare October 1, 2021 07:47

operator: dsa: Add InitImage for initcontainer

94a13fc

Signed-off-by: Oleg Zhurakivskyy <oleg.zhurakivskyy@intel.com>

ozhuraki force-pushed the dsa-idxd-op branch from 1966090 to 94a13fc Compare October 1, 2021 08:26

mythi previously approved these changes Oct 1, 2021

View reviewed changes

dsa: Add a documentation on provisioning with initcontainer

30ebc8e

Signed-off-by: Oleg Zhurakivskyy <oleg.zhurakivskyy@intel.com>

ozhuraki dismissed mythi’s stale review via 30ebc8e October 1, 2021 09:17

rojkov approved these changes Oct 1, 2021

View reviewed changes

mythi merged commit dc1465a into intel:main Oct 1, 2021

ozhuraki deleted the dsa-idxd-op branch December 17, 2021 10:43

Provision DSA devices and workqueues through initcontainer #680

Provision DSA devices and workqueues through initcontainer #680

Uh oh!

Conversation

ozhuraki commented Aug 30, 2021 • edited by mythi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mythi commented Aug 31, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Sep 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ozhuraki commented Sep 1, 2021

Uh oh!

ozhuraki commented Sep 1, 2021

Uh oh!

Uh oh!

ozhuraki commented Sep 3, 2021

Uh oh!

ozhuraki commented Sep 3, 2021

Uh oh!

ozhuraki commented Sep 3, 2021

Uh oh!

mythi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ozhuraki commented Sep 22, 2021

Uh oh!

ozhuraki commented Sep 30, 2021

Uh oh!

Uh oh!

Uh oh!

ozhuraki commented Oct 1, 2021

Uh oh!

ozhuraki commented Oct 1, 2021

Uh oh!

ozhuraki commented Oct 1, 2021

Uh oh!

ozhuraki commented Oct 1, 2021

Uh oh!

bart0sh commented Oct 1, 2021

Uh oh!

mythi commented Oct 1, 2021

Uh oh!

mythi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ozhuraki commented Oct 1, 2021

Uh oh!

mythi commented Oct 1, 2021

Uh oh!

rojkov left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ozhuraki commented Aug 30, 2021 •

edited by mythi

Loading

codecov-commenter commented Sep 1, 2021 •

edited

Loading

mythi left a comment •

edited

Loading