PYNQ Machine Learning and Computer Vision Acceleration Platform

The MLVAP example design combines the PYNQ Composable methodology with DPU on PYNQ. The composable pipeline accelerates image pre-processing for the DPU running machine learning models tackling computer vision tasks. The pipeline supports colour channel conversion, 2 types of resize operations, shifting, scaling and 8-bit fixed point quantisation.

This release of PYNQ-MLVAP was developed for PYNQ 2.7 and DPU-PYNQ 1.4 using Vitis/Vivado 2020.2.

Board support and porting

The MLVAP design was developed for the ADM-XRC-9Z1 System On Module (SOM) using the DPU-PYNQ flow to integrate the DPU instance into a custom Vitis extensible platform. The design can be ported to PYNQ-enabled platforms using the following steps:

1. HLS IP generation: generate the IP repository for the target board.
2. Extensible platform porting: add the pre-processing pipeline to the Vitis platform design.

Quick start

The MLVAP example design comes pre-installed for the ADM-XRC-9Z1 PYNQ image available in the PYNQ-AD repository. Download and boot the prebuilt PYNQ image and get started with the MLVAP example design.

Clone this repository

Note

This project depends on the Vitis Accelerated Libraries and DPU on PYNQ repositories.

git clone --recurse-submodules https://github.com/adps/pynq-mlvap.git

For the rest of the documentation, /path/to/pynq-mlvap will refer to the root of the cloned repository.

Build environment

Note

Rebuilding this design requires a Linux build host with Vitis/Vivado 2020.2 and Docker installed.

Rebuilding the design requires a Linux build environment compatible with the DPU-PYNQ flow. The provided Dockerfile is configurred with compatible XRT and OpenCV installations and dependencies to run Vitis and Vivado mounted from the host system.

The alphadata/pynq-mlvap:v1.0.0 Docker image can be rebuilt for the current user using the following make command.

cd /path/to/pynq-mlvap
make docker_image

The following command will start the container with the pynq-mlvap repo mounted under /workspace/pynq-mlvap and Vitis/Vivado installed on the host, e.g. under /opt/Xilinx, mounted under the same location. Setting a limit of 8 GB RAM with OOM kill disabled (and 4 CPU cores) is recommended to prevent Out Of Memory Exceptions (OOME) killing processes in the container, potentially failing the build.

Note

The 8GB RAM, OOM kill disable and 4 CPU cores are rough guidelines. Adjust allocated resources as needed and available on the build machine. Vivado may try to use more CPU cores than allocated to the container. E.g. on Linux, Vivado uses half of the available CPUs up to the default 8 cores limit (experimental result).

cd /path/to/pynq-mlvap
docker run --rm --init --memory 8g --oom-kill-disable --cpus 4 --volume .:/workspace/pynq-mlvap --mount type=bind,src=/opt/Xilinx,dst=/opt/Xilinx,ro,consistency=cached -it alphadata/pynq-mlvap:v1.0.0 bash

Inside the container, source Vivado and Vitis 2020.2 from the mounted location and XRT using the XRT_2020_2 environment variable. Optionally specify license server/file location for Vivado.

source $XRT_2020_2/setup.sh
source /opt/Xilinx/Vivado/2020.2/settings64.sh
source /opt/Xilinx/Vitis/2020.2/settings64.sh
export LM_LICENSE_FILE=<license server/file location>

Building the design

Note

Before generating the MLVAP Overlay, it is recommended to review the options to generate the HLS IPs. Specify any additional build arguments, e.g. TESTS=0, for the make command below as needed. The main make command below only checks for the presence of the hls/ip_repo directory to avoid overwritting any manually generated IPs. Running make clean_all will delete the IP repo allowing recovery from partial builds caused by errors.

Running the following build command will generate the platform.xsa file if not present for the specified board, integrate the DPU instance using the DPU-PYNQ flow and move the generated files inside the pynq_mlvap folder for deployment. The BOARD variable should match the name of the folder in the boards directory.

cd /workspace/pynq-mlvap
make BOARD=<board>

At the end of the build, the Docker container can be exited and all the build outputs will be present on the build host.

Deploying the design

Note

If the deployment script fails with permission denied, run sudo chown xilinx:xilinx /home/xilinx/ in a Jupyter Lab terminal and try again.

At the end of the build process, the deploy2board.sh script can be used to deploy the design on the target device. Execute the script and provide the IP address of the PYNQ board when prompted. The script will copy the pynq_mlvap directory and the setup files to /home/xilinx/pynq-mlvap on the board running PYNQ.

Deploying using prebuilt files

Alternatively, the prebuilt design for the ADM-XRC-9Z1 can be downloaded from the Alpha Data Support site for PYNQ. Open a Jupyter Lab terminal and run the following commands to download the prebuilt design. The installation steps will be the same for this method.

cd /home/xilinx
wget https://support.alpha-data.com/pub/pynq/v2.7.0/examples/admxrc9z1/archive/pynq-mlvap-v1.0.0.zip
unzip pynq-mlvap-v1.0.0.zip -d pynq-mlvap

Installing the design

Note

The following assumes pynq_dpu>=1.4.0 and pynq_composable>=1.0.2 are already installed and working on the board.

On the board running PYNQ, open a Jupyter Lab terminal to install pynq_mlvap and fetch the provided notebooks that will be delivered to the pynq_mlvap folder.

cd /home/xilinx/pynq-mlvap
pip3 install --no-build-isolation .
pynq get-notebooks pynq_mlvap -p $PYNQ_JUPYTER_NOTEBOOKS

After the installation is complete, we can verify the functionality of the design using the built-in test suite without opening any notebooks.

python3 -m pytest --pyargs pynq_mlvap