Development Tips

The following page contains various small guides on setting up development, as well as developing on our development server Jizo.

Creating an Accelerate workspace

Accelerate consist of various packages, and sometimes multiple packages need to be updated to change something. To make this easier, it can be a good idea to create a workspace so multiple packages can be edited and compiled together.

Setting up a workspace

1. Start by creating a folder for your workspace.
2. Now clone the source code of all the packages you wish to edit into that folder
3. Create a stack.yaml, consisting of all those packages.

An example

Let's say we want to make a workspace for the accelerate and accelerate-llvm (which consists of several packages) packages, with an example package for our testing. The file structure should look like the following:

- accelerate-workspace
  - accelerate
    - [source files]
  - accelerate-llvm
    - accelerate-llvm
      - [source files]
    - accelerate-llvm-native
      - [source files]
    - accelerate-llvm-ptx
      - [source files]
    - ...
  - example
    - [source files]

Now a stack.yaml file can be created like follows:

resolver: nightly-2021-07-09

packages:
- accelerate
- accelerate-llvm/accelerate-llvm
- accelerate-llvm/accelerate-llvm-native
- accelerate-llvm/accelerate-llvm-ptx
- example

extra-deps:
- github: llvm-hs/llvm-hs
  commit: 351683ed1c2f6b329e62439d42a158b1d804b846
  subdirs:
    - llvm-hs
    - llvm-hs-pure

Now running stack build example should build the example package, which can use the accelerate packages you defined in the packages: header. Making changes to any of those packages will be propagated to the example package.

Enabling debugging

If you wish to enable debugging for Accelerate, you can add the following to your stack.yaml.

flags:
  accelerate:
    debug: true

Running Nvidia Nsight on Jizo

To run Nvidia Nsight on Jizo, some port-forwarding is required as Gemini is used as a proxy. As such, following the following instructions to set up proper SSH tunnels:

SSH Tunnel from Gemini to Jizo

Login to Gemini, then execute the following command:

ssh <USERNAME>@jizo.science.uu.nl -L2202:127.0.0.1:22 -L49152:127.0.0.1:49152 -nNT

This will tunnel port 22 on Jizo to port 2202 on Gemini. If 2202 is already in use, chose another port. The same applies to 49152, which is used for interactive profiling. Feel free to chose in the range 49152-49215

SSH Tunnel from local machine to Gemini

Now, set up a SSH tunnel to Gemini from your local machine:

ssh <SOLISID>@gemini.science.uu.nl -L2202:127.0.0.1:2202 -L49152:127.0.0.1:49152 -nNT

This would bridge port 2202 on Gemini to 2202 on your local machine, as well as 49152

Configure NVIDIA Nsight Compute

Now add a new project in NVIDIA Nsight Compute, and you can configure the host as follows:

• IP/Host Name: 127.0.0.1
• User Name: Username on Jizo
• SSH Private Key: Private key location
• SSH Key Passphrase: Password of the SSH key
• Port: 2202

You can now use the "Interactive Profile" option with NVIDIA Nsight. If using the "Profile" option (which has more functionality) it will give a warning that it will fail on retrieving the output file. Just grab it using scp instead:

scp -P 2202 <USERNAME>@127.0.0.1:/tmp/var/<FILENAME>.ncu-rep .

where is the filename you configured when starting the profile. For example, if it's the default "output.prof":

scp -P 2202 <USERNAME>@127.0.0.1:/tmp/var/output.prof.ncu-rep .

That file can now be manually be viewed in Nsight

Developing on the CUB

To develop the CUB, some compiler flags might be needed to make sure compiling goes smoothly. Below here are the instructions on how to compile the CUB with NVCC using GCC.

Requirements

• The CUB source code
• GCC 8
• NVCC

Compiling

Let's say you want to compile a file called main.cu that is in the root of the CUB source code. Compiling can then be done as follows:

nvcc -lstdc++ -ccbin gcc-8 -I"cub" main.cu -o main

Below is an explination of the flags:

LSTDC++

The CUB uses C++ header files, so the lstdc++ flag is needed to make sure the binary is linked against the C++ headers.

CCBin

Using GCC 8 to compile is nessecary, so this instructs NVCC to use GCC 8 instead of another version. If GCC 8 is the only GCC binary on your system, removing this flag will be possible.

I"cub"

Include the "cub" folder for compiling. This is the folder container the CUB source code. Any changes made there will be used when compiling "main.cu"