Skip to content

Agora-wireless/Agora

master
Switch branches/tags
Code

Latest commit

* better verbosity in initial calibration
* use single gain in json
* modify retries for calibration
* Added some helper text
* Switching to udp
* Add Core-Thread Allocation Summary
* Increase number of data streams (need to add variable)
* Files ready for testing.   Mac UDP packet size < max

* Debugging updates.   Fixed a few indexing issues.
Tested with MacSender not running (looks ok)
Tested with MacSender running in UDP mode with no data availble (looks ok)

* Tested with the FileReader with full packets and fixed size partial packets. Looks good for 1 user.

* print UL and DL bit rates in config
* Added hooks for stream #2
* Fixed error setting initial core offset
* disable RBIndicator
* Config file changes to sync updates
* Updated the e2e test parameters because the sim send rate increased
* Pushing a sim mac configuration similar to the operational hardware configuration
* Removing alignment of the mac packets, since they are just a grouping of bytes after decoding.
1) Readability improvements for the mac packets
2) Removed the padding, and extra variables that enforced alignments.

* Added core assignment summary to end
* reducing the thread core footprint
* Fixed an offset issue with the mac packet
* Synced basestation and user mac thread updates
* Fixed output and tested with the channel sim in AWGN mode
* added memory alignment check
* Switched around the TX scheduling to match socket thread expects
* Modified the core assignment logic to wrap around and exclude a core list.
* Added core exclusion list
* Allow pinning same thread to same core
* Use int type for rx_offset in the sync process

* Matlab script to analyze down-link data recorded in ue_worker
* More verbose errors in initial calibration!
* Print pilot SNR range (across antenna) for each user
* Fix bug in radio_id calculation for two channel scenario.
* Functions to save armadillo matrices and vectors to file.
* Add a separate moving sum calibration buffer
* Set initial calib buffer to one.
* Fix bug in passing calib buffer argument to DoZF and PacketTXRX.
* Turn off initial calib by default
* Better boolean naming for recording frame index
* Fix dual-channel downlink seg fault by using UeAntNum instead of UeNum in PhyStats
* Set calib vector values for first few frames to one when initial calib is off. Print ZF matrix condition number.
* Fixed indexing issue (keeps incrementing)
* Print reciprocal calibration pilot SNR stats
* Fix bug in dual-channel UE pilot TX scheduling
* Move ZF condition number updates to PhyStats
* Send uplink calibration symbol in txrx instead of FPGA buffer.
* sw/hw framer mode for base station
* Modified the config to allow larger frames in hardware
* Disable rcond calculation by default.
* Fix error for using AVX2 FlexRAN on AVX512 capable machine
* Allow DPDK to select NIC with given MAC addresses
* Load iris serials from json. Unify bs and ue configs.
* Remove deprecated reciprocal test
* Fix uplink breaking due to incorrect Direction setting
* Fix bad indexing for ref radio
* Fixed memory leak upon program exit
* Change Channel to UeChannel for clients
* Better handling of reference nodes
* Fix ref radio bug
* Fix handling of reference node in uplink-only mode.
* Updating the e2e test to align cores on falcon
* Modified UeNum to UeAntNum on the basestation side
* Separate Function for Reciprocal Calib functions
* updated powder configs (ul and dl)
* Added additional retries on all ::make calls
* Use single JSON config in channel simulator
* Fix out of bound rx buffer offset
* Pick last peak in FindBeacon
* Disabled untested configurations

Co-authored-by: rdoosty <doost@rice.edu>
Co-authored-by: obejarano <obejarano@rice.edu>
Co-authored-by: Jian Ding <jianding17@gmail.com>
Co-authored-by: Andrew Sedlmayr <as61@localhost>
f020d5c

Git stats

Files

Permalink
Failed to load latest commit information.

Build Status

Agora is a complete software realization of real-time massive MIMO baseband processing.

Some highlights:

  • Agora currently supports 64x16 MU-MIMO (64 RRU antennas and 16 UEs) with 20 MHz bandwidth and 64QAM modulation, on a 36-core server with AVX512 support.
  • Agora is configurable in terms of numbers of RRU antennas and UEs, bandwidth, modulation orders, LDPC code rates.
  • Agora supports an emulated RRU and UEs with a high-performance packet generator.
  • Agora has been tested with real RRUs with up to 64 antennas and up to 8 UEs. The RRU and UE devices are available from Skylark Wireless.

Contents

Building Agora

Agora currently only builds and runs on Linux, and has been tested on Ubuntu 16.04, 18.04, and 20.04. Agora requires CMake 2.8+ and works with both GNU and Intel compilers with C++17 support.

Setting up the build environment

  • Setup CI: run

    $ ./config_ci.sh
    
    • Note for developers: You must run this command before checking out your new feature branch. Do not use _ in your branch name. Use - instead.
  • See scripts/ubuntu.sh for required packages, including Linux packages, gtest, Armadillo, and SoapySDR, and the corresponding versions. Run ./scripts/ubuntu.sh to install these packages.

  • Download and install Intel libraries:

    • Install Intel compiler and MKL, refer to INTELLIB_README.md.

    • Set required environment variables by sourcing setvars.sh. If oneAPI is installed in /opt, run source /opt/intel/oneapi/setvars.sh.

    • Install Intel FlexRAN's FEC SDK for LDPC encoding and decoding:

      • NOTE: Compiling FlexRAN requires Intel compiler with version <= 19.0.4. Newer versions of Intel compiler can also work, but require a patch for resolving conflicts with FlexRAN.
        Please contact the current Agora developers to get the patch.
      • Download Intel FlexRAN's FEC SDK to /opt.
      • Compile FlexRAN as follows:
       $ sudo chmod -R a+rwX FlexRAN-FEC-SDK-19-04/ # Allow all users read-write access 
       $ cd /opt/FlexRAN-FEC-SDK-19-04/sdk/ 
       $ sed -i '/add_compile_options("-Wall")/a \ \ add_compile_options("-ffreestanding")' cmake/intel-compile-options.cmake 
       $ ./create-makefiles-linux.sh 
       $ cd build-avx512-icc # or build-avx2-icc 
       $ make -j
       
    • Optional: DPDK

      • Refer to DPDK_README.md for configuration and installation instructions.

Building and running with emulated RRU

We provide a high performance packet generator to emulate the RRU. This generator allows Agora to run and be tested without actual RRU hardware.
The following are steps to set up both Agora and the packet generator:

  • Build Agora. This step also builds the emulated RRU, a data generator that generates random input data files, an end-to-end test that checks correctness of end results for both uplink and downlink,
    and several unit tests for testing either performance or correctness of individual functions.

     $ cd Agora
     $ mkdir build
     $ cd build
     $ cmake ..
     $ make -j
     
  • Run end-to-end test to check correctness (uplink, downlink and combined tests should all pass if everything is set up correctly).

     $ ./test/test_agora/test_agora.sh 10 out # Runs test for 10 iterations
     
  • Run Agora with emulated RRU traffic

    • NOTE: We recommend running Agora and the emulated RRU on two different machines.
      If you are running them on the same machine, make sure Agora and the emulated RRU are using different set of cores, otherwise there will be performance slow down.

    When running Agora and the emulated RRU on two different machines, the following steps use Linux networking stack for packet I/O.
    Agora also supports using DPDK to bypass the kernel for packet I/O. See DPDK_README.md for instructions of running emulated RRU and Agora with DPDK.

    • First, return to the base directory (cd ..), then run
    $ ./build/data_generator --conf_file data/tddconfig-sim-ul.json
    

    to generate data files.

    • In one terminal, run
    $ ./build/agora --conf_file data/tddconfig-sim-ul.json
    

    to start Agora with uplink configuration.

    • In another terminal, run
    $ ./build/sender --num_threads=2 --core_offset=1 --frame_duration=5000 --enable_slow_start=1 --conf_file=data/tddconfig-sim-ul.json
    

    to start the emulated RRU with uplink configuration.

  • Run Agora with channel simulator and clients

    • First, return to the base directory (cd ..), then run
    $ ./build/data_generator --conf_file data/chsim.json
    

    to generate data files.

    • In one terminal, run
    $ ./build/user --conf_file data/chsim.json
    

    to start clients with combined uplink & downlink configuration.

    • In another terminal, run
    $ ./build/chsim --bs_threads 1 --ue_threads 1 --worker_threads 2 --core_offset 24 --conf_file data/chsim.json
    
    • In another terminal, run
    $ ./build/agora --conf_file data/chsim.json
    

    to start Agora with the combined configuration.

    • Note: make sure Agora and sender are using different set of cores, otherwise there will be performance slow down.
  • Run Agora with channel simulator, clients, and mac enabled.

    • Compile the code with
    $ cmake .. -DENABLE_MAC=true
    
    • Uplink Testing (--conf_file mac-ul-sim.json)
    • Downlink Testing (--conf_file mac-dl-sim.json)
    • Combined Testing (--conf_file mac-sim.json)
      • Terminal 1:
        $./build/data_generator --conf_file data/mac-sim.json
      
      to generate data files.
        $./build/user --conf_file data/mac-sim.json
      
      to start users.
      • Terminal 2:
      $ ./build/chsim --bs_threads 1 --ue_threads 1 --worker_threads 2 --core_offset 28 --bs_conf_file data/mac-sim.json --ue_conf_file data/mac-sim.json
      
      to run the channel simulator
      • Terminal 3:
        $ ./build/macuser --enable_slow_start 1 --conf_file data/mac-sim.json
      
      to run to user mac app. Specify --data_file "" to generate patterned data and --conf_file options as necessary.
      • Terminal 4:
      $ ./build/agora --conf_file data/mac-sim.json
      
      run agora before running macbs. Run macuser -> agora -> macbs in quick succession.
      • Terminal 5:
      $ ./build/macbs --enable_slow_start 1 --conf_file data/mac-sim.json
      
      to run to base station mac app. specify --data_file "" to generate patterned data and --conf_file options as necessary.
    • Note: make sure agora / user / chsim / macuser / macbs are using different set of cores, otherwise there will be performance slow down.
  • To run with real wireless traffic from Faros/Iris hardware UEs, see the Agora with real RRU section below.

Building and running with real RRU

Agora supports a 64-antenna Faros base station as RRU and Iris UE devices. Both are commercially available from Skylark Wireless and are used in the POWER-RENEW PAWR testbed.
Both Faros and Iris have their roots in the Argos massive MIMO base station, especially ArgosV3. Agora also supports USRP-based RRU and UEs.

We recommend using one server for controlling the RRU and running Agora, and another server for controlling the UEs and running the UE code.

Agora supports both uplink and downlink with real RRU and UEs. For downlink, a reference node outside the array (and synchronized) is required for reciprocity calibration.
Note: Faros RRU and Iris UEs can be discovered using the pyfaros tool. You can use this tool to find the topology of the hardware connected to the server.

We describe how to get the uplink and downlink demos working. Below XX can be replaced with either ul and dl.

  • Rebuild the code on both servers for RRU side the UE side.
    • For Faros RRU and Iris UEs, pass -DUSE_ARGOS=on -DUSE_UHD=off to cmake
    • For USRP-based RRU and UEs, pass -DUSE_ARGOS=off -DUSE_UHD=on to cmake
    • Run make -j to recompile the code.
  • Run the UE code on the server connected to the Iris UEs
    • For Iris UEs, run the pyfaros tool in the data directory as follows:
      $ python3 -m pyfaros.discover --json-out
      
      This will output a file named topology.json with all the discoverable serial IDs included.
    • Modify data/topology.json by adding/removing serials of client Irises you'd like to include from your setup.
    • For USRP-based RRU and UEs, modify the existing data/topology.json and enter the appropriate IDs.
    • Run ./build/data_generator --conf_file data/XX-hw.json to generate required data files.
    • Run ./build/user --conf_file data/XX-hw.json.
  • Run Agora on the server connected to the Faros RRU
    • scp over the generated file data/LDPC_orig_XX_data_512_ant2.bin from the client machine to the server's data directory.
    • Rebuild the code
      • Run make -j to compile the code.
    • For Faros RRU, use the pyfaros tool the same as with the UEs to generate a new data/topology.json
    • Modify data/topology.json by adding/removing serials of your RRU Irises, and the hub.
    • Run ./build/agora --conf_file data/XX-hw.json.

Running performance test

To test the real-time performance of Agora for processing 64x16 MU-MIMO with 20 MHz bandwidth and 64QAM modulation, we recommend using two servers (one for Agora and another for the emulated RRU) and DPDK
for networking. In our experiments, we use 2 servers each with 4 Intel Xeon Gold 6130 CPUs. The servers are connected by 40 GbE Intel XL710 dual-port NICs.

  • NOTE: We recommend using at least 10 GbE NIC and a server with more than 10 cores for testing real-time performance of 8x8 MU-MIMO. For 8x8 MU-MIMO, our test on a machine with AVX-512 and CPU frequency
    of 2.3 GHz support shows that at least 7 worker cores are required to achieve real-time performance. Additionally, Agora requires one core for the manager thread and at least 1 core for network threads.\

We change "worker_thread_num" and "socket_thread_num" to change the number cores assigned to of worker threads and network threads in the json files, e.g., data/tddconfig-sim-ul.json.
If you do not have a powerful server or high throughput NICs, we recommend increasing the value of --frame_duration when you run ./build/sender, which will increase frame duration and reduce throughput.

To process 64x16 MU-MIMO in real-time, we use both ports of 40 GbE Intel XL710 NIC with DPDK (see DPDK_README.md) to get enough throughput for the traffic of 64 antennas.
(NOTE: For 100 GbE NIC, we just need to use one port to get enough thoughput.)

To reduce performance variations, we did the following configurations for the server that runs Agora:

  • NOTE: These steps are not strictly required if you just wanted to try out Agora and do not care about performance variations.
  • Disable Turbo Boost to reduce performance variation by running
    $ echo "0" | sudo tee /sys/devices/system/cpu/cpufreq/boost
    
  • Set CPU scaling to performance by running
    $ sudo cpupower frequency-set -g performance
    
    where cpupower can be installed through
    $ sudo apt-get install -y linux-tools-$(uname -r)
    
  • Turn off hyper-threading. We provide an example bash script (scripts/tune_hyperthread.sh), where the core indices are machine dependent.
  • Set IRQ affinity to direct OS interrupts away from Agora's cores. We direct all the interrupts to core 0 in our experiments.
    We provide an example bash script (scripts/set_smp_affinity.sh), where the IRQ indices are machine dependent.

The steps to collect and analyze timestamp traces are as follows:

  • Enable DPDK in Agora. Make sure it is compiled / configured for supporting your specific hardware NICs (see DPDK_README.md).

  • We use data/tddconfig-sim-ul.json for uplink experiments and data/tddconfig-sim-dl.json for downlink experiments.
    In our paper, we change “antenna_num”, “ue_num” and “symbol_num_perframe” to different values to collect different data points in the figures.

  • Generate source data files by running

    $ ./build/data_generator --conf_file data/tddconfig-sim-ul.json
    
  • Run Agora as a real-time process (to prevent OS from doing context switches) using

    $ sudo LD_LIBRARY_PATH=${LD_LIBRARY_PATH} chrt -rr 99 ./build/agora --conf_file data/tddconfig-sim-ul.json
    

    (NOTE: Using a process priority 99 is dangerous. Before running it, make sure you have directed OS interrupts away from cores used by Agora. If you have not done so, run

    $ sudo LD_LIBRARY_PATH=${LD_LIBRARY_PATH} ./build/agora --conf_file data/tddconfig-sim-ul.json
    

    instead to run Agora as a normal process.)

  • Run the emulated RRU using

    $ sudo LD_LIBRARY_PATH=${LD_LIBRARY_PATH} ./build/sender --num_threads=2 --core_offset=0 \
      --conf_file=data/tddconfig-sim-ul.json --frame_duration=5000 --enable_slow_start=1
    

    For DPDK, add --server_mac_addr= and set it to the MAC address of the NIC used by Agora.

  • The timestamps will be saved in data/timeresult.txt after Agora finishes processing. We can then use a MATLAB script to process the timestamp trace.

  • We also provide MATLAB scripts for uplink and downlink that are able to process multiple timestamp files and generate figures reported in our paper.

Contributing to Agora

Agora is open-source and open to your contributions. Before contributing, please read this.

Acknowledgment

Agora was funded in part by NSF Grant #1518916 and by the NSF PAWR project.

Documentation

Check out Agora Wiki for Agora's design overview and flow diagram that maps massive MIMO baseband processing to the actual code structure. Technical details and performance results can be found in

  • Jian Ding, Rahman Doost-Mohammady, Anuj Kalia, and Lin Zhong, "Agora: Real-time massive MIMO baseband processing in software," in Proc. of ACM CoNEXT, December 2020 (PDF, video).

Doxygen documentation generation for Agora can be initiated by running the following command from the repository root directory: doxygen Agora_doxygen.conf The latest hosted output is located at Agora Doxygen

Other community resources can be found at the RENEW Wireless Wiki

Contact

Jian Ding (jian.ding@yale.edu)