Skip to content

Latest commit

 

History

History
168 lines (115 loc) · 12.9 KB

Readme.md

File metadata and controls

168 lines (115 loc) · 12.9 KB
Raw

Hastlayer Vitis Driver

Overview

This project contains the communication service used to connect with Vitis Unified Software Platform devices, such as the Xilinx Alveo U250 FPGA accelerator card via the OpenCL library.

Note that the SH scripts in this project should use LF line endings! You'll get errors such as -bash: $'\r': command not found otherwise.

For Nimbix-specific instructions see the Nimbix docs.

Requirements

  • The system running the FPGA card must be 64-bit Linux (e.g. Ubuntu 18.04.2 LTS or CentOS 7.6). The installation instructions can be found here in the platform documentation.
  • The device-specific software requirements can be found in the card's Getting Started page, e.g. Alveo U250.
  • Hastlayer has its own software requirements which can be found in the repository's GettingStarted.md file.

Preparation

Even after everything is installed, you have to make sure that the executing user's environment variables are correctly set by sourcing the setup scripts as described in the documentation. You can add these commands into the ~/.bashrc file to avoid having to type them every time. If running from the cloud, like Nimbix, this is probably handled automatically.

For setup instructions on the Nimbix cloud see the Nimbix-specific instructions.

Cross Compilation

If you want to build for a platform not in your /opt/xilinx/platforms directory, you can set the XILINX_PLATFORM environment variable to the directory that contains the platform directories. For example: export XILINX_PLATFORM=$HOME/platforms. If the environment variable is not set or if its value isn't an existing directory then /opt/xilinx/platforms is used as fallback.

Be sure that all .NET software dependencies are on the same version on both the target and the source computers. Otherwise the source code won't match during transformation. This will result in a different Transformation ID and the XCLBIN file won't be found, prompting a recompilation on the target machine. To mitigate this risk you can try some of the following strategies:

Cross Compilation with Docker

This way you can compile on your Windows machine, or any machine where you don't want to install XRT permanently. Note that you still need to download the complete Vitis XDK separately for licensing reasons and it takes about 125GB (and at least 50GB more temporarily) to set up the image. Of course you need Docker installed too. However there are no alternatives on Windows so please bear with it. Following these steps you will get a container with Vitis XDK and .NET Core 3.1 SDK installed. Please remember not to distribute the resulting image!

  1. Download the Xilinx Vitis 2020.2: All OS Installer Single-File version from the Vitis Downloads.
  2. Extract the Xilinx_Unified_2020.2_* folder from it (tar xzf Xilinx_Unified_2020.2_*) and copy the folder into Container inside this project.
  3. Download the XRT, deployment platform and development platform packages for CentOS 7:
    • You can download the latest released packages from the Getting Started section of the card's product page (eg. U250).
    • If you are using Azure, all device specific packages must be downloaded from the Xilinx Lounge instead. The NP servers currently require the RedHat / CentOS 7.6 & 7.8 files from the Alveo U250 Gen3x16 XDMA Platform 2.1 Downloads section. Make sure to download both packages for XRT and Deployment Target Platform, not just the ones marked Azure.
  4. Copy the files into the Container/platform folder.
    • If you downloaded the Azure packages too, make sure you don't end up with multiple versions of the same package. For example at the time of writing this document the main Deployment Target Platform archive contained the package xilinx-u250-gen3x16-xdma-validate-2.1-2948414.noarch.rpm. However a newer, Azure-specific version xilinx-u250-gen3x16-xdma-validate-2.1-3005608.1.noarch.rpm was also up for download. Such a clash causes a multilib version problem. That can be resolved by removing older one and keeping the Azure-specific version.
  5. Extract any tar.gz archive in Container/platform and delete the archives. In the end you should only have rpm files.
  6. Copy the centos7-install.sh to the Container as well.
  7. Open a shell of your choice while still in the Container folder and type docker build -t vitis . to create an image. This will take a while.
  8. Open Docker Desktop to verify that the "vitis" image appeared in the Images tab.
  9. Clean up after the build is finished with the docker builder prune -a -f command. If you notice excessive RAM usage by the Vmmem process then close down Docker Desktop, shut down WSL with the wsl --shutdown command, then restart Docker Desktop and continue with the next steps.
  10. Go back to Docker Desktop and click Run on the "vitis" image.
  11. Expand the Optional Settings and create a shared directory by selecting a Host Path (this can be anywhere), and entering "/data" into the Container Path field.
  12. Switch to the Containers / Apps tab in Docker Desktop and click on the CLI (>_) button.
  13. A window with sh shell will appear. Type bash as it already has the XRT setup configured.
  14. Copy your Hastlayer project into the shared folder and access it through the /data directory.

As you can see it was as simple as 1, 2, 13!

If you'd like to move the Docker WLS files (which can be upwards of 100 GB) to another folder/drive then follow this guide.

Using Vitis on Azure NP Servers

If you want work with an Alveo card on an Azure VM, you need to pick the Azure-specific device (currently only Azure Alveo U250). This alters some of the automatic compilation steps. After compilation it submits your binary to an attestation server (via Azure Blob Storage) for automatic approval.

Preparation

You have to set up an NP10s virtual machine via the Azure Portal.

  1. Go to the Select an image page.
  2. Enter "xilinx" into the search bar.
  3. Select the Centos 7 deployment image (Xilinx Alveo U250 Deployment VM - Centos7.8).
  4. Fill out the Create a virtual machine form:
    • Fill the Virtual machine name field.
    • Set the Region to East US, West US2, West Europe or SouthEast Asia.
    • Click See all sizes and search for "np10s".
    • Set up the administrator account's SSH key or specify the user/password.
  5. Click Next : Disks and decide if you need additional disks.
  6. The rest should be fine as-is, click Review + create.

Troubleshooting: if you don't see the Standard_NP10s size option, see the first FAQ item on this page as you might need to request quota or the available regions might've changed since the writing of this document.

Once the VM is running, you have to install the correct platform and runtime files from the Xilinx Lounge, the Vitis SDK and the .NET runtime or SDK. Transfer all package files into the same directory, navigate into it and type the following to install them at once:

Ubuntu

sudo apt install *.deb
source ubuntu-install.sh

CentOS 7

sudo yum localinstall *.rpm
source centos7-install.sh

You can use any SSH client of your choice but we recommend checking out MobaXterm.

Configuration

The approval process requires addition configuration. Fill out and add the below AzureAttestationConfiguration property to the CustomConfiguration in your appsettings.json file.

{
    "HardwareGenerationConfiguration": {
        "CustomConfiguration": {
            "AzureAttestationConfiguration": {
                "StorageAccountName": "From portal.",
                "StorageAccountKey": "From portal.",
                "ClientSubscriptionId": "From portal.",
                "ClientTenantId": "From portal."
            }
        }
    }
}

To get this information from your Azure account:

  1. Go to the Azure Portal.
  2. Click Storage Accounts.
  3. Select your account or create a new one with Blob Storage.
  4. The Subscription ID on the Overview page becomes ClientSubscriptionId.
  5. Click Settings | Access Keys on the side bar.
  6. Click the Show Keys button.
  7. The "Storage account name" field becomes StorageAccountName and the "Key" becomes StorageAccountKey.
  8. Go back to the home page and select Active Directory.
  9. The Tenant ID on the Overview page becomes ClientTenantId.

Additionally, during compilation only a netlist is generated instead of a bitstream so there won't be any reports based on simulation data.

Execution

As of writing this document, the Azure VMs aren't meant for online compilation and there isn't enough space to install the full Vitis SDK. So you have to build and run attestation on a separate machine. The build machine must have Vitis 2020.2, but doesn't need Alveo hardware.

On the build machine:

  1. Navigate to the application directory.
  2. Configure appsettings.json as described above.
  3. Run the Hastlayer application with the "Azure Alveo U250" device selected. It will terminate with an exception after the build is finished because there is no hardware.
  4. Copy the application to the Azure VM. If you already did that, copy the HardwareFramework/bin directory and the appsettings.json file.

On the Azure virtual machine:

  1. Run export XILINX_VITIS=/; source /opt/xilinx/xrt/setup.sh to set up the environment. You may want to copy this into your ~/.bashrc for future convenience.
  2. Navigate to the application directory.
  3. Run the Hastlayer application with the same parameters you did on the build machine.

Other Remarks

If you ever get an error [XRT] ERROR: some device is already programmed due to a crashed or interrupted execution, you can reset the card using xbutil reset command. See more info about the Xilinx Board Utility here.

If you just want to generate a simulation report, you can do that without the full build by configuring the VitisBuildConfiguration.SynthesisOnly custom configuration in the appsettings.json or by adding the following command line argument:

--HardwareGenerationConfiguration:CustomConfiguration:VitisBuildConfiguration:SynthesisOnly true

Available system memory is an important factor. Unless you have a lot, you won't be able to compile multiple projects at the same time. When V++ runs out of available memory it rather crashes than waits so it's best practice to not do anything involved on the machine during compilation.

HBM is used by default on the cards that support it (Alveo U50 and U280) but only one slot. This means that only 256MB memory can be utilized with HBM. To use larger programs with devices that have both HBM and DDR memory, you must disable HMB during compilation either by editing the UseHbm property of OpenClConfiguration in appsettings.json, programmatically changing the value of this in HardwareGenerationConfiguration or using the following command line switch:

--HardwareGenerationConfiguration:CustomConfiguration:OpenClConfiguration:UseHbm false

Like with every other HardwareGenerationConfiguration change, adding this setting will alter the hash resulting in a unique xclbin file so both options can be compiled ahead of time in case you want to automate selecting between them based on the input. Also note, that Alveo U50 only has HBM memory so this option will have no real effect on it.