computeWorks_examples

Matrix multiplication example performed with OpenMP, OpenACC, BLAS, cuBLAS, and CUDA

Getting Started

This example requires the following packages:

• CUDA Toolkit 10.1
• PGI CE Compiler 19.10

Optional:

• Eclipse IDE C/C++
• Docker CE + NVIDIA-Docker v2
  • PGI Docker image
• Jupyter Notebook

The following installation instructions have been tested on Ubuntu 18.04 and CUDA 10.0+.

OpenACC profiling with NVIDIA driver 418.67 and above requires elevated permissions. See here.

You can achieve this one of two ways.

1. Run command with sudo

sudo LD_LIBRARY_PATH=/usr/local/cuda/extra/CUPTI/lib64:$LD_LIBRARY_PATH ./computeWorks_mm

2. Following Administration instructions.

sudo systemctl isolate multi-user # Stop the window manager
sudo su # Switch to root
modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia-vgpu-vfio nvidia # Unload dependent modules
cd /etc/modprobe.d/
touch nvidia.conf # Create file named nvidia.conf
echo -e "options nvidia "NVreg_RestrictProfilingToAdminUsers=0"" > nvidia.conf
reboot

Installation

CUDA -> more details

1. Download CUDA Toolkit
2. Install (assuming file is in ~/Downloads)

sudo dpkg -i ~/Downloads/cuda-repo-ubuntu1804-10-1-local-10.1.168-418.67_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-<version>/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda

3. Add paths to ~/.bashrc

echo -e "\n# CUDA paths" >> ~/.bashrc
echo -e "export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}" >> ~/.bashrc
echo -e "export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" >> ~/.bashrc
echo -e "export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" >> ~/.bashrc

• The CUPTI directory is required for OpenACC profiling

PGI Community Edition (Bare Metal) -> more details

Skip this step if you prefer to run utilize the PGI compiler in a Docker container.

1. Download PGI CE Compiler
2. Install (assuming file is in ~/Downloads)

export PGI_SILENT=true
export PGI_ACCEPT_EULA=accept
export PGI_INSTALL_DIR=/opt/pgi
export PGI_INSTALL_TYPE=single
export PGI_INSTALL_NVIDIA=true
export PGI_INSTALL_JAVA=true
export PGI_INSTALL_MPI=false
export PGI_MPI_GPU_SUPPORT=false
mkdir -p ~/Downloads/tmp
tar xpfz ~/Downloads/pgilinux-2019-1910-x86-64.tar.gz -C ~/Downloads/tmp
sudo -E ~/Downloads/tmp/install
rm -rf ~/Downloads/tmp

3. Add paths to ~/.bashrc

echo -e "\n# PGI paths" >> ~/.bashrc
echo -e "export PGI=/opt/pgi" >> ~/.bashrc
echo -e "export PATH=/opt/pgi/linux86-64/19.10/bin:$PATH" >> ~/.bashrc
echo -e "export MANPATH=$MANPATH:/opt/pgi/linux86-64/19.10/man" >> ~/.bashrc
echo -e "export LM_LICENSE_FILE=$LM_LICENSE_FILE:/opt/pgi/license.dat" >> ~/.bashrc

Eclipse

1. Download Eclipse IDE C/C++
2. Install (assuming file is in ~/Downloads)

sudo tar xpfz ~/Downloads/eclipse-cpp-2019-03-R-linux-gtk-x86_64.tar.gz -C /opt
sudo ln -s /opt/eclipse/eclipse /usr/local/bin/eclipse

3. Install Nsight Eclipse Plugin -> more details

bash /usr/local/cuda/bin/nsight_ee_plugins_manage.sh install /opt/eclipse

Docker -> more details

1. Remove older Docker versions

sudo apt remove docker docker-engine docker.io containerd runc -y

2. Install Docker CE -> more details

sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update
sudo apt install docker-ce -y

3. Enable Docker commands without sudo

sudo usermod -aG docker $USER

4. Log out and back in

• Confirm $USER is in the docker group

groups

mnicely adm cdrom sudo dip plugdev lpadmin sambashare docker

5. Verify docker runs without sudo

docker container run hello-world

NVIDIA Docker v2 -> more details

1. Remove nvidia-docker v1, if installed.

docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f
sudo apt-get purge nvidia-docker

2. Add nvidia-docker v2 repository

curl -sL https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -sL https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update

3. Install nvidia-docker v2

sudo apt-get install nvidia-docker2 -y

4. Reload Docker daemon

sudo pkill -SIGHUP dockerd

5. (Optional) Modify Docker daemon to storage images in /home versus /var. Usually /home has more space.

sudo nano /etc/docker/daemon.json

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    },
    "experimental": true,
    "graph": "/home/<whoami>/.docker",
    "storage-driver": "overlay2"
}

sudo service docker restart

6. Verify you can launch docker container with access to GPU

docker run --runtime=nvidia --rm nvcr.io/nvidia/cuda:latest nvidia-smi

PGI Community Edition (Docker Image)

This create a Docker image containing the PGI CE Compiler.

1. Download git project

git clone https://github.com/mnicely/computeWorks_examples.git
cd computeWorks_examples/computeWorks_mm/pgi_build

2. Download PGI CE Compiler
3. Move PGI tar to Dockerfile directory (assuming file is in ~/Downloads)

cp ~/Downloads/pgilinux-2019-194-x86-64.tar.gz .

4. Build PGI Docker image

docker build -t cuda-10.1_ubuntu-18.04_pgi-19.10 -f Dockerfile.cuda-10.1_ubuntu-18.04_pgi-19.10 .

Jupyter Notebook

1. Install PIP package manager

sudo apt install python-pip3

2. Install JupyterLab

sudo -H pip3 install jupyter

Usage

Bare Metal

• This approach requires PGI Community Edition (Bare Metal)

1. Download git project

git clone https://github.com/mnicely/computeWorks_examples.git
cd computeWorks_examples/computeWorks_mm

2. Build computeWorks_mm binary

make

3. Run computeWorks_mm <matrixSize | default=1024>

./computeWorks_mm 128

Docker

• This approach requires PGI Community Edition (Docker Image)

1. Download git project

git clone https://github.com/mnicely/computeWorks_examples.git
cd computeWorks_examples/computeWorks_mm

2. Build computeWorks_mm binary

docker run --runtime=nvidia --rm -v $(pwd):/workspace -w /workspace cuda-10.1_ubuntu-18.04_pgi-19.10:latest make

3. Run computeWorks_mm <matrixSize | default=1024>

docker run --runtime=nvidia --rm -v $(pwd):/workspace -w /workspace cuda-10.1_ubuntu-18.04_pgi-19.10:latest ./computeWorks_mm 128

Eclipse IDE C/C++

Eclipse, with Nsight Eclipse Plugins offers full-featured IDE that provides an all-in-one integrated environment to edit, build, debug and profile CUDA-C applications.

1. Open Eclipse

eclipse &

2. Import Project

• File -> Import...
• Select -> Git -> Projects from Git -> [Next >]
• Select Repository Store -> Clone URI -> [Next >]
• Source Git Repository -> URL -> https://github.com/mnicely/computeWorks_examples -> [Next >]
• Branch Selection -> [Next >]
• Local Destination -> [Next >]
• Select a wizard to use for importing projects -> Import existing Eclipse project -> [Next >]
• Import Projects -> [Finish]

Baremetal

• This approach requires PGI Community Edition (Bare Metal)

3. Build Project

• Right click computeWorks_mm -> Build Project or Press Ctrl + B

4. Run Project

• Right click computeWorks_mm -> Run As -> Local C/C++ Application

Docker container

• This approach requires PGI Community Edition (Docker Image)

3. Point to PGI Docker container

• Right click computeWorks_mm -> Properties
• C/C++ Build -> Settings
• Settings -> Container Settings -> select Build inside Docker Image
• Image -> cuda-10.1_ubuntu-18.04_pgi-19.10:latest

4. Run Project

• Right click computeWorks_mm -> Run As -> Local C/C++ Application

JupyterLab

• This approach requires Administration instructions.

1. Open computeWorks_mm.ipynb

cd computeWorks_examples/computeWorks_mm/jupyter
jupyter-notebook computeWorks_mm.ipynb