Skip to content

Commit

Permalink
Merge pull request #39 from arnobaer/readme
Browse files Browse the repository at this point in the history
updated readme
  • Loading branch information
arnobaer committed Apr 25, 2023
2 parents e3042c6 + cb6e667 commit 6e58952
Showing 1 changed file with 108 additions and 49 deletions.
157 changes: 108 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,30 @@

Firmware for the CMS Global Trigger (uGT).

This repository contains the algorithm part of the CMS uGT firmware. It has been written for the Imperial MP7, a Virtex-7 based AMC module.
This repository contains the algorithm part of the CMS uGT firmware. It has been
written for the Imperial MP7, a Virtex-7 based AMC module.

Documentaion of [uGT firmware](doc/mp7_ugt_firmware_specification/pdf/gt-mp7-firmware-specification.pdf).

The uGT is versioned with the following scheme:

* Major version: Changes in firmware structure, changes that require significant interventions at P5.
* Major version: Changes in firmware structure, changes that require significant
interventions at P5.
* Minor version: New features for trigger decisions.
* Rev version: Bug fixes, change/add scripts.

In addition there are versions (with similar schemes) for the following firmware parts of uGT:
In addition there are versions (with similar schemes) for the following firmware
parts of uGT:
* Framework ([frame.vhd](firmware/hdl/payload/frame.vhd))
* Global Trigger Logic ([gtl_module.vhd](firmware/hdl/payload/gtl_module_tpl.vhd))
* Final Decision Logic ([fdl_module.vhd](firmware/hdl/payload/fdl_module.vhd))

Following software tool versions used to create L1Menu and menu depended VHDL files:
Following software tool versions used to create L1Menu and menu depended VHDL
files:

| utm | TME | VHDL Producer |
|:-:|:-:|:-:|
| 0.11.0 | 0.15.1 | 2.14.0 |
| 0.11.2 | 0.15.2 | 2.14.0 |

Following dependencies exist for uGT firmware:

Expand All @@ -32,36 +36,51 @@ Following dependencies exist for uGT firmware:
| [IPB_FW](https://github.com/ipbus/ipbus-firmware) | v1.4


## Setup environment for simulation and build
## Setup environment

First make sure to checkout the branch or tag to work with.

```bash
git clone https://github.com/cms-l1-globaltrigger/mp7_ugt_legacy.git
cd mp7_ugt_legacy
git checkout <branch/tag>
```

If not already done create a Python virtual environment and install all required
dependencies including [IPBB](https://github.com/ipbus/ipbb) and lxml.

If not already done create a Python virtual environment and install required dependencies including [IPBB](https://github.com/ipbus/ipbb) and lxml.
```bash
git clone https://github.com/cms-l1-globaltrigger/mp7_ugt_legacy.git <mp7_ugt_legacy_path>
cd <mp7_ugt_legacy_path>
git checkout <branch>
python3 -m venv env
. env/bin/activate
pip install -U pip
pip install -r scripts/requirements.txt
```

A template file for setting up the envirionment variables is available [setup_env_sim_synth_tpl.sh](setup_env_sim_synth_tpl.sh). Copy this file to e.g. "setup_env_sim_synth.sh", edit the values for the variables and execute the script with ". setup_env_sim_synth.sh".
A template file for setting up the required environment variables is available
[setup_env_sim_synth_tpl.sh](setup_env_sim_synth_tpl.sh). Copy this file to e.g.
`setup_env_sim_synth.sh`, edit the values for the variables and source the
script.

```bash
. setup_env_sim_synth.sh
```

Example for "setup_env_sim_synth.sh"
Example for `setup_env_sim_synth.sh`

```bash
## Simulation (Questasim)
export MODELSIM_ROOT=/opt/mentor
export MODELSIM_PATH=${MODELSIM_ROOT}/questasim/bin
export PATH=${MODELSIM_PATH}:${PATH}
export MTI_VCO_MODE=64
export MGLS_LICENSE_FILE=1717@heros.hephy.at
export MGLS_LICENSE_FILE=1234@license.server
export UGT_GITLAB_USER_NAME=hbergaue
export UGT_QUESTASIMLIBS_DIR=/opt/mentor
export UGT_QUESTASIM_SIM_PATH=${MODELSIM_ROOT}
export UGT_VIVADO_QUESTASIMLIBS_VERSION=2019.2
export UGT_QUESTASIM_LIBS_PATH=${UGT_QUESTASIMLIBS_DIR}/questasimlibs_vivado_${UGT_VIVADO_QUESTASIMLIBS_VERSION}
export UGT_BLK_MEM_GEN_VERSION_SIM=blk_mem_gen_v8_4_4

## Synthesis (Vivado)
export VIVADO_BASE_DIR=/opt/xilinx/Vivado
export UGT_VIVADO_BASE_DIR=${VIVADO_BASE_DIR}
Expand All @@ -70,18 +89,23 @@ export UGT_BLK_MEM_GEN_VERSION_SYNTH=blk_mem_gen_v8_4_4
source ${VIVADO_BASE_DIR}/${UGT_VIVADO_VERSION}/settings64.sh
```

## Simulate
## Simulation

Simulation of VHDL module "gtl_fdl_wrapper.vhd" with Questa simulator for 6 ugt modules.
Simulation of VHDL module `gtl_fdl_wrapper.vhd` with Questa simulator for 6 ugt modules.

* This is a description running script [run_simulation_questa.py](scripts/run_simulation_questa.py) for simulation.
* If Questasim libraries for a certain Vivado version do not exist, they have to be created for the selected Questasim version with script [run_compile_simlib.py](scripts/run_compile_simlib.py):
* If Questasim libraries for a certain Vivado version do not exist, they have to
be created for the selected Questasim version with script [run_compile_simlib.py](scripts/run_compile_simlib.py):
```bash
cd <mp7_ugt_legacy_path>
python3 scripts/run_compile_simlib.py --vivado <Vivado version (e.g. 2019.2)> --questasim <Questasim version (e.g. 10.7c)> --output <output directory for generated libraries>
python scripts/run_compile_simlib.py \
--vivado <vivado_version (e.g. 2019.2)> \
--questasim <path to Questasim (e.g. /opt/mentor/10.7c/questasim)> \
--output <output directory for generated libraries>
```

Following table shows which Questasim libraries for a certain Vivado version have been created and tested (version of IP 'blk_mem_gen' in libraries is relevant for Global Trigger logic):
Following table shows which Questasim libraries for a certain Vivado version
have been created and tested (version of IP 'blk_mem_gen' in libraries is
relevant for Global Trigger logic):

| Vivado | blk_mem_gen | 10.7c | 2021.1_2 |
|:-:|:-:|:-:|:-:|
Expand All @@ -97,38 +121,48 @@ Be sure that setup of environment is done (see chapter above).
Run simulation using Questa.

*Remarks:*
- MIF files (for mass over deltaR) are located in 'mp7_ugt_legacy/firmware/sim' for simulation.
- MIF files (for mass over deltaR) are located in `mp7_ugt_legacy/firmware/sim`
for simulation.

Running script

```bash
cd <mp7_ugt_legacy_path>
python3 scripts/run_simulation_questa.py <path to menu xml file (in repo or local)> --tv <testvector file path> [--ignored]
python scripts/run_simulation_questa.py <xml_file> --tv <testvector_file> [--ignored]
```

*Note:* inspect for default values and other arguments

```bash
python3 scripts/run_simulation_questa.py -h
python scripts/run_simulation_questa.py -h
```

Example 1

```bash
cd <mp7_ugt_legacy_path>
python3 scripts/run_simulation_questa.py https://raw.githubusercontent.com/herbberg/l1menus/master/2022/L1Menu_Collisions2022_v0_1_6-d1/xml/L1Menu_Collisions2022_v0_1_6-d1.xml --tv https://raw.githubusercontent.com/herbberg/l1menus/master/2022/L1Menu_Collisions2022_v0_1_6-d1/testvectors/TestVector_L1Menu_Collisions2022_v0_1_6.txt --ignored
python scripts/run_simulation_questa.py https://raw.githubusercontent.com/cms-l1-globaltrigger/cms-l1-menu/master/2022/L1Menu_Collisions2022_v1_4_0-d1/xml/L1Menu_Collisions2022_v1_4_0-d1.xml \
--tv https://raw.githubusercontent.com/cms-l1-globaltrigger/cms-l1-menu/master/2022/L1Menu_Collisions2022_v1_4_0-d1/testvectors/TestVector_L1Menu_Collisions2022_v1_4_0_ttbar.txt \
--ignored
```

Example 2

```bash
cd <mp7_ugt_legacy_path>
python3 scripts/run_simulation_questa.py <local path>/github/herbberg/l1menus/2022/L1Menu_Collisions2022_v0_1_6-d1/xml/L1Menu_Collisions2022_v0_1_6-d1.xml --tv <local path>/github/herbberg/l1menus/2022/L1Menu_Collisions2022_v0_1_6-d1/testvectors/TestVector_L1Menu_Collisions2022_v0_1_6.txt --ignored
python scripts/run_simulation_questa.py ./cms-l1-menu/2022/L1Menu_Collisions2022_v1_4_0-d1/xml/L1Menu_Collisions2022_v1_4_0-d1.xml \
--tv ./cms-l1-menu/2022/L1Menu_Collisions2022_v1_4_0-d1/testvectors/TestVector_L1Menu_Collisions2022_v1_4_0_ttbar.txt \
--ignored
```

## Build `mp7_ugt` firmware using IPBus builder (IPBB) [for 6 modules]
## Build all modules

*Remarks:*
- FW versions greater or equal 0x1130 are used for uGMT v6.1.0 and unpacker (with bug fix for shifted raw eta).
- FW versions greater or equal 0x1120 and lower 0x1130 are used for uGMT v6.0.0_patch1 and unpacker (with bug: shifted raw eta).
- FW versions greater or equal `0x1150` used for muon scouting.
- FW versions greater or equal `0x1130` are used for uGMT v6.1.0 and unpacker (with bug fix for shifted raw eta).
- FW versions greater or equal `0x1120` and lower `0x1130` are used for uGMT v6.0.0_patch1 and unpacker (with bug: shifted raw eta).

The firmware uses the ipbb build tool, and requires the ipbus system firmware. If you are going to build on a computer outside of the CERN network, then you will need to run kerberos (kinit username@CERN.CH). These instructions assume that you have your Xilinx Vivado licensing already setup for your enviroment.
The firmware uses the ipbb build tool, and requires the ipbus system firmware.
If you are going to build on a computer outside of the CERN network, then you
will need to run kerberos (kinit username@CERN.CH). These instructions assume
that you have your Xilinx Vivado licensing already setup for your enviroment.

* The ugt repo [mp7 ugt_legacy](https://github.com/cms-l1-globaltrigger/mp7_ugt_legacy) is a fork of [svn2git ugt](https://gitlab.cern.ch/hbergaue/ugt/blob/master/mp7_ugt) repo.
- added [ugt_strategy.tcl](firmware/ucf/ugt_strategy.tcl) for ugt specific strategy and inserted it into [top.dep](firmware/cfg/top.dep).
Expand All @@ -139,7 +173,7 @@ The firmware uses the ipbb build tool, and requires the ipbus system firmware. I
- script [checkIpbbSynth.py](scripts/checkIpbbSynth.py) for checking used FPGA resources.
- script [fwpackerIpbb.py](scripts/fwpackerIpbb.py) for packing firmware files in a tar file.

* The [MP7](https://gitlab.cern.ch/cms-cactus/firmware/mp7) firmware needs to be adapted for Global Trigger firmware.
* The [MP7](https://gitlab.cern.ch/cms-cactus/firmware/mp7) firmware needs to be adapted for Global Trigger firmware.
* A script [mp7patch.py](scripts/mp7patch.py) make this changes in the following MP7 files (tag v3.0.0):
- [area_constraints.tcl](https://gitlab.cern.ch/cms-cactus/firmware/mp7/-/blob/v3.0.0/boards/mp7/base_fw/common/firmware/ucf/area_constraints.tcl)
- [mp7_brd_decl.vhd](https://gitlab.cern.ch/cms-cactus/firmware/mp7/-/blob/v3.0.0/boards/mp7/base_fw/mp7xe_690/firmware/hdl/mp7_brd_decl.vhd)
Expand All @@ -149,63 +183,84 @@ The firmware uses the ipbb build tool, and requires the ipbus system firmware. I

### Workflow

Be sure that setup of environment is done (see chapter above).
Make sure to setup ypur local bash environment (see above).

Run kerberos for outside of CERN network.

```bash
kinit <username>@CERN.CH
```

Run synthesis script (for all 6 modules).

```bash
cd <mp7_ugt_legacy_path>
python3 scripts/run_synth_ipbb.py <path to menu xml file (in repo or local)> --ugturl <URL ugt git repo> --ugt <ugt tag in repo> --build <build-version> -p <work dir>
python scripts/run_synth_ipbb.py <xml_file> \
--ugturl <ugt_url_to_git_repo> \
--ugt <ugt_tag_or_branch> \
--build <build_version> \
-p <working_dir>
```

*Note:* inspect default values for arguments using

```bash
python3 scripts/run_synth_ipbb.py -h
python scripts/run_synth_ipbb.py -h
```

Example 1

```bash
cd <mp7_ugt_legacy_path>
python3 scripts/run_synth_ipbb.py https://raw.githubusercontent.com/herbberg/l1menus/master/2022/L1Menu_Collisions2022_v0_1_6-d1/xml/L1Menu_Collisions2022_v0_1_6-d1.xml --ugturl https://github.com/cms-l1-globaltrigger/mp7_ugt_legacy --ugt master --build 0x113d -p <local path>/work_synth/production
python scripts/run_synth_ipbb.py https://raw.githubusercontent.com/cms-l1-globaltrigger/cms-l1-menu/master/2022/L1Menu_Collisions2022_v1_4_0-d1/xml/L1Menu_Collisions2022_v1_4_0-d1.xml \
--ugturl https://github.com/cms-l1-globaltrigger/mp7_ugt_legacy \
--ugt master \
--build 0x113d \
-p ~/work_synth/production
```

Example 2

```bash
cd <mp7_ugt_legacy_path>
python3 scripts/run_synth_ipbb.py <local path>/github/herbberg/l1menus/2022/L1Menu_Collisions2022_v0_1_6-d1/xml/L1Menu_Collisions2022_v0_1_6-d1.xml --ugturl https://github.com/cms-l1-globaltrigger/mp7_ugt_legacy --ugt master --build 0x113d -p <local path>/work_synth/production
python scripts/run_synth_ipbb.py ./cms-l1-menu/2022/L1Menu_Collisions2022_v1_4_0-d1/xml/L1Menu_Collisions2022_v1_4_0-d1.xml \
--ugturl https://github.com/cms-l1-globaltrigger/mp7_ugt_legacy \
--ugt master \
--build 0x113d \
-p ~/work_synth/production
```

After all syntheses have finished, check results:

```bash
$ python3 scripts/check_synth_ipbb.py <path of build_xxxx.cfg>
python scripts/check_synth_ipbb.py <path of build_xxxx.cfg>
```

If timing errors occur (and bit files is not generated), check timing errors in file:

```bash
<project path>/proj/module_<nr>/module_<nr>/module_<nr>.runs/impl_1/top_timing_summary_postroute_physopted.rpt
```

Afterwards execute the following command for every module with timing errors to generate bit file:

```bash
$ vivado -mode batch -source <path to scripts/vivado_write_bitstream.tcl> -tclargs <project path> <module number (e.g.: 0)>
vivado -mode batch -source <path to scripts/vivado_write_bitstream.tcl> -tclargs <project path> <module id (e.g.: 0)>
```

After successfully created bit files, execute the following command to create tar file for HW:

```bash
$ python3 scripts/fwpacker_ipbb.py <path of build_xxxx.cfg>
python scripts/fwpacker_ipbb.py <path of build_xxxx.cfg>
```

## Build for one module (with IPBB)
## Build single module

Run kerberos for outside of CERN network

```bash
kinit <username>@CERN.CH
```

If not already done create a Python virtual environment and install required dependencies including [IPBB](https://github.com/ipbus/ipbb) and lxml.
If not already done create a Python virtual environment and install all required dependencies including [IPBB](https://github.com/ipbus/ipbb) and lxml.

```bash
python3 -m venv env
. env/bin/activate
Expand All @@ -214,39 +269,43 @@ pip install -r scripts/requirements.txt
```

Create a local working area

```bash
ipbb init <working_area_path>
cd <working_area_path>
ipbb add git https://github.com/ipbus/ipbus-firmware.git -b v1.4
ipbb add git https://:@gitlab.cern.ch:8443/hbergaue/mp7.git -b mp7fw_v3_0_0_mp7_ugt
ipbb add git https://github.com/cms-l1-globaltrigger/mp7_ugt_legacy -b <master or branch name or tag name>
ipbb add git https://github.com/cms-l1-globaltrigger/mp7_ugt_legacy -b <branch/tag>
```

Copy VHDL producer output and patched files to mp7_ugt (`gtl_module.vhd`, `gtl_pkg.vhd`, `algo_mapping_rop.vhd` and `gt_mp7_top_pkg.vhd`)

Source Vivado

Create project

```bash
ipbb proj create vivado module_<module number> mp7:../mp7_ugt_legacy
cd proj/module_<module number>
ipbb proj create vivado module_<module_id> mp7:../mp7_ugt_legacy
cd proj/module_<module_id>
ipbb vivado make-project --single
```

Run implementation, synthesis

```bash
ipbb vivado synth
ipbb vivado impl
```

Generate a bitfile

```bash
ipbb vivado package
deactivate
```

## Contact

Bernhard Arnold [<bernhard.arnold@cern.ch>]<br>
Herbert Bergauer (Developer) [<herbert.bergauer@oeaw.ac.at>]<br>
Manfred Jeitler [<Manfred.Jeitler@cern.ch>]

0 comments on commit 6e58952

Please sign in to comment.