Skip to content
Permalink
Browse files

Update docker image dpdk-app-centos. Mainly cleanup some prints

and minor changes for SR-IOV support.

Signed-off-by: Billy McFall <22157057+Billy99@users.noreply.github.com>
  • Loading branch information
Billy99 committed Sep 12, 2019
1 parent dc9406e commit cb81ee09d9dd8868099b538d57710711b5d6747f
@@ -13,6 +13,8 @@ MAINTAINER Billy McFall <bmcfall@redhat.com>
RUN rpm --import https://mirror.go-repo.io/centos/RPM-GPG-KEY-GO-REPO && curl -s https://mirror.go-repo.io/centos/go-repo.repo | tee /etc/yum.repos.d/go-repo.repo
RUN yum groupinstall -y "Development Tools"
RUN yum install -y wget numactl-devel git golang make; yum clean all
# Debug Tools (if needed):
#RUN yum install -y pciutils iproute; yum clean all

#
# Download and Build APP-NetUtil
@@ -40,6 +42,9 @@ RUN sed -i -e 's/EAL_IGB_UIO=y/EAL_IGB_UIO=n/' config/common_linux
RUN sed -i -e 's/KNI_KMOD=y/KNI_KMOD=n/' config/common_linux
RUN sed -i -e 's/LIBRTE_KNI=y/LIBRTE_KNI=n/' config/common_linux
RUN sed -i -e 's/LIBRTE_PMD_KNI=y/LIBRTE_PMD_KNI=n/' config/common_linux
# Additional Debug if Needed
#RUN sed -i -e 's/CONFIG_RTE_LIBRTE_ETHDEV_DEBUG=n/CONFIG_RTE_LIBRTE_ETHDEV_DEBUG=y/' config/common_base

# DPDK_VER 19.02
#RUN sed -i -e 's/EAL_IGB_UIO=y/EAL_IGB_UIO=n/' config/common_linuxapp
#RUN sed -i -e 's/KNI_KMOD=y/KNI_KMOD=n/' config/common_linuxapp
@@ -50,15 +55,16 @@ RUN make install T=${RTE_TARGET} DESTDIR=${RTE_SDK}
#
# Build TestPmd
#
#WORKDIR ${DPDK_DIR}/app/test-pmd
#COPY ./dpdk-args.c ./dpdk-args.c
#COPY ./dpdk-args.h ./dpdk-args.h
#COPY ./testpmd_eal_init.txt ./testpmd_eal_init.txt
#COPY ./testpmd_launch_args_parse.txt ./testpmd_launch_args_parse.txt
#COPY ./testpmd_substitute.sh ./testpmd_substitute.sh
#RUN ./testpmd_substitute.sh
#RUN make
#RUN cp testpmd /usr/bin/dpdk-app
WORKDIR ${DPDK_DIR}/app/test-pmd
COPY ./dpdk-args.c ./dpdk-args.c
COPY ./dpdk-args.h ./dpdk-args.h
COPY ./testpmd_eal_init.txt ./testpmd_eal_init.txt
COPY ./testpmd_launch_args_parse.txt ./testpmd_launch_args_parse.txt
COPY ./testpmd_substitute.sh ./testpmd_substitute.sh
RUN ./testpmd_substitute.sh
RUN make
RUN cp testpmd /usr/bin/testpmd
RUN cp testpmd /usr/bin/dpdk-app

#
# Build l3fwd
@@ -71,7 +77,8 @@ COPY ./l3fwd_parse_args.txt ./l3fwd_parse_args.txt
COPY ./l3fwd_substitute.sh ./l3fwd_substitute.sh
RUN ./l3fwd_substitute.sh
RUN make
RUN cp build/l3fwd /usr/bin/dpdk-app
RUN cp build/l3fwd /usr/bin/l3fwd
#RUN cp build/l3fwd /usr/bin/dpdk-app

# -------- Import stage.
# Docker 17.05 or higher
@@ -1,6 +1,6 @@
# Docker Image: dpdk-app-centos
This directory contains the files needed to build a DPDK based test image.
This image is based on CentOS (latest) base image built with DPDK 19.02.
This image is based on CentOS (latest) base image built with DPDK 19.08.

The User Space CNI inconjunction with the OVS CNI Library (cniovs) or VPP
CNI Library (cnivpp) creates interfaces on the host, like a vhost-user or
@@ -46,8 +46,8 @@ additional package or two to the image after the import stage.


# Docker Image Details
This Docker Image is downloading DPDK (version 19.02, but the version is
not important) and building it. Once built, changing into the DPDK `testpmd`
This Docker Image is downloading DPDK (version 19.08 to get memif PMD)
and building it. Once built, changing into the DPDK `testpmd`
directory (${DPDK_DIR}/app/test-pmd) and building it.

`testpmd` is a sample DPDK application that comes with DPDK. Typically,
@@ -69,7 +69,8 @@ $ testpmd \
```

This Docker image is tweaking this a little. Before `testpmd` is built, the
testpmd.c file (contains main()) is overwritten with a local copy.
testpmd.c file (contains main()) is updated using 'sed'. See
'testpmd_substitute.sh'.

**NOTE:** If a different version of DPDK is needed or used, this local file
may need to be synchronized with the updated version.
@@ -80,6 +81,12 @@ will generate this list of input parameters, and then pass this private set
of parameters to DPDK functions instead of the inpupt `argc` and `argv`. When
the generated binary is copied to `/usr/bin/`, it is renamed to `dpdk-app`.

The code is leveraging another project, app-netutil
(https://github.com/openshift/app-netutil), which is a library design to be
called within a container to collect all the configuration data, like that
stored in annotations by Userspace CNI, and expose it to a DPDK application
in a clean API.

**NOTE:** For debugging, if `dpdk-app` is called with a set of input parameters,
it will skip the dpdk-args.c code and behave exactly as `testpmd`. Just add
the `sleep` to the pod spec:
@@ -101,10 +108,37 @@ Then get a pod shell:
kubectl exec -it userspace-ovs-pod-1 -- sh
```

Run `dpdk-app` as needed:
Run `dpdk-app` with no parameters, and it will be as if it is called
as the container is started. It also prints out the generated parameter
list, which include the dynamic socketfile path:
```
sh-4.2# dpdk-app
COLLECT Data:
cpuRsp.CPUSet = 0-63
Interface[0]:
IfName="eth0" Name="cbr0" Type=unknown
MAC="5e:6b:7e:19:5b:94" IP="10.244.0.197"
Interface[1]:
IfName="net1" Name="sriov-network-a" Type=SR-IOV
MAC=""
PCIAddress=0000:01:0a.4
Interface[2]:
IfName="net2" Name="sriov-network-b" Type=SR-IOV
MAC=""
PCIAddress=0000:01:02.4
ENTER dpdk-app (testpmd):
myArgc=15
dpdk-app -n 4 -l 1-3 --master-lcore 1 --vdev=virtio_user0,path=/var/lib/cni/usrspcni/34c8ba49b767-net1 --vdev=virtio_user1,path=/var/lib/cni/usrspcni/34c8ba49b767-net2 --no-pci -- --auto-start --tx-first --no-lsc-interrupt --stats-period 60
EAL: Detected 64 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
:
```

Then 'CTRL-C' to exit and re-run `dpdk-app` with input parameters
modified as needed:
```
dpdk-app \
-m 1024 \
-l 1-3 \
-master-lcore 1 \
-n 4 \
@@ -117,13 +151,6 @@ dpdk-app \
--no-lsc-interrupt
```

In a future patch, instead of hardcoding these parameters in dpdk-args.c, the
code will leverage another project, app-netutil
(https://github.com/openshift/app-netutil), which is a library design to be
called within a container to collect all the configuration data, like that
stored in annotations by Userspace CNI, and expose it to a DPDK application
in a clean API.


# Deploy Image
An example of using this Docker image can be found in this same repo under:
@@ -36,6 +36,7 @@ static int getInterfaces(int argc, int *pPortCnt, int *pPortMask) {
int j;
int vhostCnt = 0;
int memifCnt = 1;
int sriovCnt = 0;
int err;
struct InterfaceResponse ifaceRsp;
char macStr[DPDK_ARGS_MAX_MAC_STRLEN];
@@ -147,6 +148,8 @@ static int getInterfaces(int argc, int *pPortCnt, int *pPortMask) {
if (ifaceRsp.pIface[i].Sriov.PCIAddress) {
snprintf(&myArgsArray[argc++][0], DPDK_ARGS_MAX_ARG_STRLEN-1,
"-w %s", ifaceRsp.pIface[i].Sriov.PCIAddress);
sriovCnt++;

free(ifaceRsp.pIface[i].Sriov.PCIAddress);

*pPortMask = *pPortMask | 1 << *pPortCnt;
@@ -271,6 +274,11 @@ static int getInterfaces(int argc, int *pPortCnt, int *pPortMask) {
free(ifaceRsp.pIface[i].Name);
}
} /* END of FOR EACH Interface */


if (sriovCnt == 0) {
strncpy(&myArgsArray[argc++][0], "--no-pci", DPDK_ARGS_MAX_ARG_STRLEN-1);
}
}
else {
printf("Couldn't get network interface, err code: %d\n", err);
@@ -333,8 +341,6 @@ char** GetArgs(int *pArgc, eDpdkAppType appType)

argc = getInterfaces(argc, &portCnt, &portMask);

strncpy(&myArgsArray[argc++][0], "--no-pci", DPDK_ARGS_MAX_ARG_STRLEN-1);

/*
* Initialize APP Specific Options
*/
@@ -361,8 +367,6 @@ char** GetArgs(int *pArgc, eDpdkAppType appType)

argc = getInterfaces(argc, &portCnt, &portMask);

strncpy(&myArgsArray[argc++][0], "--no-pci", DPDK_ARGS_MAX_ARG_STRLEN-1);

/*
* Initialize APP Specific Options
*/
@@ -9,24 +9,28 @@
int myArgc = 0;
char **myArgv = NULL;

printf("ENTER dpdk-app:\n");
printf(" argc=%d\n", argc);
for (i = 0; i < argc; i++) {
printf(" argv[%d]=%s\n", i, argv[i]);
}

if (argc > 1) {
printf("ENTER dpdk-app (l3fwd):\n");
printf(" argc=%d\n ", argc);
for (i = 0; i < argc; i++) {
printf(" %s", argv[i]);
}
printf("\n");

ret = rte_eal_init(argc, argv);
if (ret < 0)
rte_exit(EXIT_FAILURE, "Invalid EAL parameters\n");
argc -= ret;
argv += ret;
} else {
printf("COLLECT Data:\n");
myArgv = GetArgs(&myArgc, DPDK_APP_L3FWD);
printf(" myArgc=%d\n", myArgc);
printf("ENTER dpdk-app (l3fwd):\n");
printf(" myArgc=%d\n ", myArgc);
for (i = 0; i < myArgc; i++) {
printf(" myArgv[%d]=%s\n", i, myArgv[i]);
printf(" %s", myArgv[i]);
}
printf("\n");

ret = rte_eal_init(myArgc, myArgv);
if (ret < 0)
@@ -5,20 +5,24 @@
int myArgc = 0;
char **myArgv = NULL;

printf("ENTER dpdk-app:\n");
printf(" argc=%d\n", argc);
for (i = 0; i < argc; i++) {
printf(" argv[%d]=%s\n", i, argv[i]);
}

if (argc > 1) {
printf("ENTER dpdk-app (testpmd):\n");
printf(" argc=%d\n ", argc);
for (i = 0; i < argc; i++) {
printf(" %s", argv[i]);
}
printf("\n");

diag = rte_eal_init(argc, argv);
} else {
printf("COLLECT Data:\n");
myArgv = GetArgs(&myArgc, DPDK_APP_TESTPMD);
printf(" myArgc=%d\n", myArgc);
printf("ENTER dpdk-app (testpmd):\n");
printf(" myArgc=%d\n ", myArgc);
for (i = 0; i < myArgc; i++) {
printf(" myArgv[%d]=%s\n", i, myArgv[i]);
printf(" %s", myArgv[i]);
}
printf("\n");

diag = rte_eal_init(myArgc, myArgv);
}

0 comments on commit cb81ee0

Please sign in to comment.
You can’t perform that action at this time.