RAN650 and Intel E810-XXVDA4T network adapter

[hakegard]

We are trying to connect Benetel RAN650 to the srsRAN using the E810 network card from Intel. We have some problems with synchronization and have tried both PTP and GPS synchronization. DPDK is not installed.

• Connecting the O-RU to the O-DU using GPS synchronization:
  • phc2sys is used to synchronize the system clock to the PHC of the NIC.
  • Wireshark shows that packets are flowing through the FH interface. Both U- and C-plane packets (type 1 and 3) are transmitted from the O-DU, while U-plane packets are transmitted from the O-RU.
  • The kpi.sh application in the O-RU shows that some U-plan packets arrive late (some 10s or some 100s per log line). Number of TX packets are 0.
  • The gnb-log on OFH level shows late downlink messages. May this indicate that the problem is within the O-DU and not the sync between the O-DU and O-RU?
  • The UEs do not see the network, and there is no indication of RRC frames being transmitted.
• Connecting the O-RU to the O-DU using PTP synchronization.
  • The initiation script in the O-RU stops waiting the PTP sync, although logging shows that the O-RU receives PTP packets and synchronises.
  • Wireshark shows that the workstation transmits PTP delay requests and the O-RU PTP delay responses. The O-RU does not send PTP delay requests.

Has anyone experienced something similar and has some device on how to proceed?

[hakegard]

We are still struggling with this issue. UEs connecting to the network using a USRP as RU are not seeing the network using the RAN650 as RU, and there are no signs in the logs that RRC connection establishment is initiated. The CU/DU runs on an AMD Ryzen Threadripper with 64 CPUs and 256 GB RAM. A 4x4 MIMO antenna from Panorama (DWMM4[G]-6-60) is used, and the operation frequency is in the n77u band. The current status is:

• GPS synchronization between DU and RU works fine.
• The configuration files in the srsRAN documentation are used. By adjusting the delay profile parameters t1a…, the kpi.sh application in the RU indicates that all UP and CP packets arrive on time. The number of TX packets is still zero.
• After testing different RU parameter settings, the long PRACH format with LF compression format enabled is selected.
• Monitoring the fronthaul interface with wireshark shows that O-FH packets go in both directions over the fiber. None of the other PCAP files indicate any communication between the UEs and the DU.
• Combinations of MIMO settings (2x2, 4x2 and 4x4), bandwidths (20MHz, 40 MHz, 80 MHz and 100 MHz), and gaincontrol (15-30 dB) are tested. Also, other parameters are modified (such as downlink scaling in the RU and various parameters in the DU) and tested. Increasing the distance to UEs, in case saturation in the UEs is the problem, is tested.
• The gnb.log file consists of lines like:
  

Any assistance or proposals on how to solve this issue are very appreciated. Advice from Benetel support is to get assistance from the srsRAN community… Configuration files, logs etc. can be provided if it might help in the diagnostics.

[ismagom]

Hi @hakegard ,

I share here some config it works well in our setup:

gnb config:

cell_cfg:
  dl_arfcn: 640000
  band: 78
  channel_bandwidth_MHz: 100
  tdd_ul_dl_cfg:
    nof_dl_symbols: 6
    nof_ul_symbols: 4
    nof_dl_slots: 7
    nof_ul_slots: 2
  ssb: 
     ssb_period: 10
     ssb_block_power_dbm: -21
  prach: 
      prach_config_index: 7
      prach_root_sequence_index: 1
      zero_correlation_zone: 0
      prach_frequency_start: 0
  pusch:
    dmrs_additional_position: 2
  pdsch:
    dmrs_additional_position: 2

ru_ofh:
  t1a_max_cp_dl: 470
  t1a_min_cp_dl: 419
  t1a_max_cp_ul: 336
  t1a_min_cp_ul: 285
  t1a_max_up: 345
  t1a_min_up: 294
  ta4_max: 200
  ta4_min: 0
  is_prach_cp_enabled: false
  compr_method_ul: bfp
  compr_bitwidth_ul: 9
  compr_method_dl: bfp
  compr_bitwidth_dl: 9
  compr_method_prach: bfp
  compr_bitwidth_prach: 9
  enable_ul_static_compr_hdr: true
  enable_dl_static_compr_hdr: true
  iq_scaling: 10

Make sure the TDD pattern matches in the RU config. This is then the config we use in the RU (/etc/ru_config.cfg)

# MIMO mode - the radio can operate in 1_3, 2_4, 1_2_3_4_4x2 or 1_2_3_4_4x4 mode
# Variable name : mimo_mode
# Valid options : 1_3
#                 2_4
#                 1_2_3_4_4x2
#                 1_2_3_4_4x4
mimo_mode=1_2_3_4_4x2

# FPGA downlink scaling in steps of 6 dB (0, 6, 12 and 18 can be set)
# Variable name : downlink_scaling
# Valid options : 0
#                 6
#                 12
#                 18
downlink_scaling=6

# PRACH format
# Variable name : prach_format
# Valid options : short
#                 long
prach_format=long

# Short form PRACH compression settings (note uncompressed mode is not supported when RU is in 4x2 or 4x4 mode)
# Variable name : compression
# Valid options : static_uncompressed
#                 dynamic_uncompressed
#                 static_compressed
#                 dynamic_compressed
compression=static_compressed

# Long form PRACH compression settings
# Variable name : lf_prach_compression_enable
# Valid options : true
#                 false
lf_prach_compression_enable=true

# C-Plane per-symbol workaround
# Variable name : cplane_per_symbol_workaround
# Valid options : enabled
#                 disabled
cplane_per_symbol_workaround=disabled

# CUPLANE dl couplling sectionID              
# Variable name : cuplane_dl_coupling_sectionId
# Valid options : enabled                      
#                 disabled                     
cuplane_dl_coupling_sectionID=disabled         
                                               
# FlexRAN PRACH workaround                     
# Variable name : flexran_prach_workaround     
# Valid options : enabled                      
#                 disabled                     
flexran_prach_workaround=disabled              
                                               
#DL_UL special slot tunning                    
#Variable name : dl_ul_tuning_special_slot     
#Valid options : 0x0                           
#                0xfd00000                     
dl_ul_tuning_special_slot=0xfd00000            

Also, I would recommend sticking with 100 MHz 4x2 configuration. I think it's what works best.

Once you see no lates in kpi.sh, can you check if you see signal radiating out of the antenna 1?

[hakegard]

Hi @ismagom,

Thanks for your response. The spectrum at the output of antenna 1 looks like this:

The center frequency is 3.88 GHz, so the signal is within the bandwidth 3.83-3.93 GHz. It is however sparse compared to the spectrum from the USRP (40 MHz BW, one TX antenna, no UE connected):

The configuration files are as follows:

gnb config:

/etc/ru_config.cfg:

/etc/tdd.xml:

It seems as only the two first antennas transmit power:

When starting the gnb, we get the following terminal output:

Sorry for the long message. Hopefully, it might give an indication why we are not able to connect.

[ismagom]

Check the gnb config. It's not the same as the one I provided.

Do you see lates/earlies in kpi.sh command in the RU?

[hakegard]

Thanks again @ismagom for spending time on this. It is much appreciated. I have ordered the gnb config to make sure that it is as similar to yours as possible.

amf:
  addr: 127.0.0.5                                               
  bind_addr: 127.0.0.1                                            

cell_cfg:
  dl_arfcn: 658666                                                
  band: 77                                                    
  common_scs: 30                                                  
  plmn: "24271"                                                   
  tac: 7                                                          
  pci: 1                                                          
  nof_antennas_dl: 4                                              
  nof_antennas_ul: 2     
  
  channel_bandwidth_MHz: 100                                      
  tdd_ul_dl_cfg: 
    nof_dl_symbols: 6                       
    nof_ul_symbols: 4                      
    nof_dl_slots: 7                         
    nof_ul_slots: 2                                           
  ssb:
    ssb_period: 10
    ssb_block_power_dbm: -21
  prach: 
      prach_config_index: 7
      prach_root_sequence_index: 1
      zero_correlation_zone: 0
      prach_frequency_start: 0
  pusch:
    dmrs_additional_position: 2
  pdsch:
    dmrs_additional_position: 2

ru_ofh:
  t1a_max_cp_dl: 470
  t1a_min_cp_dl: 419
  t1a_max_cp_ul: 336
  t1a_min_cp_ul: 285
  t1a_max_up: 345
  t1a_min_up: 294
  ta4_max: 200
  ta4_min: 0                                            
  is_prach_cp_enabled: false                                      
  compr_method_ul: bfp                                            
  compr_bitwidth_ul: 9                                            
  compr_method_dl: bfp                                            
  compr_bitwidth_dl: 9                                            
  compr_method_prach: bfp                                         
  compr_bitwidth_prach: 9                                         
  enable_ul_static_compr_hdr: true                                
  enable_dl_static_compr_hdr: true                               
  iq_scaling: 10                                                
  cells:
    - network_interface: enp74s0f0np0                                 
      ru_mac_addr: 70:b3:d5:xx:xx:xx                               
      du_mac_addr: 50:7c:6f:xx:xx:xx                              
      vlan_tag: 3                                              
      prach_port_id: [4]                                          
      dl_port_id: [0, 1, 2, 3]                                          
      ul_port_id: [0,2] 

There are a few differences. We have the n77 version of RAN650, so the lowest frequency allowed is 3.7 GHz. We know however that the UEs handle our frequency band as it works with the USRPs. kpi.sh indicates that all packets arrive on time:

root@benetelru:~# kpi.sh
 SAMPLE_TIME          | RX_TOTAL             | RX_ON_TIME           | RX_EARLY             | RX_LATE              | RX_ON_TIME_C         | RX_EARLY_C           | RX_LATE_C            | TX_TOTAL             
 15:58:58.439973      | 4124                 | 3959                 | -0                   | -0                   | 164                  | -0                   | -0                   | -0                   
 15:58:59.434145      | 7295                 | 6958                 | -0                   | -0                   | 335                  | -0                   | -0                   | -0                   
 15:59:00.432488      | 7463                 | 7360                 | -0                   | -0                   | 360                  | -0                   | -0                   | -0                   
 15:59:01.433490      | 6204                 | 5877                 | -0                   | -0                   | 202                  | -0                   | -0                   | -0                   
 15:59:02.432892      | 16853                | 16121                | -0                   | -0                   | 1014                 | -0                   | -0                   | -0                   
 15:59:03.432391      | 22516                | 20921                | -0                   | -0                   | 1262                 | -0                   | -0                   | -0                   
 15:59:04.433627      | 14893                | 14045                | -0                   | -0                   | 774                  | -0                   | -0                   | -0       

Selecting prach_frequency_start=0 gives the following warning:

Warning: Configured PRACH occasion collides with PUCCH RBs ([0..1) intersects [0..3)). Some interference between PUCCH and PRACH is expected.

If we select e.g. prach_frequency_start=2 there is no warning.

Unfortunately, the result is as before, also when varying parameters such as the gaincontrol and also bandwidth (40 MHz) as we know that the UEs support this bandwidth. Any suggestion on what else that can be tried? Is it normal to get these warnings in the gnb log file?

2024-05-07T09:03:36.748628 [OFH     ] [W] Detected '4' late downlink C-Plane messages in the transmitter queue for slot '478.2'
2024-05-07T09:03:36.748633 [OFH     ] [W] Detected '4' late downlink U-Plane messages in the transmitter queue for slot '478.2', symbol '0'
2024-05-07T09:03:36.748688 [OFH     ] [W] Failed to dispatch Control-Plane type 1 message for slot '499.2'
2024-05-07T09:03:36.748693 [OFH     ] [W] Failed to dispatch message in the downlink data flow User-Plane for slot '499.2'

[ismagom]

Hi @hakegard ,

• Ignore the PRACH warning, the Benetel RU only works with prach_frequency_start: 0
• Do not use 40 MHz. the RU only supports well 100 MHz.
• Set prach_port_id: [4,5] in the gnb.yml
• Have you verified the DL signal? I suggest capturing it with a spectrum analyzer that supports 5G or capturing it with a file and visualizing in Matlab that it looks good
• Why are you using plmn: "24271"? What sim card and core network do you have?

Have you been able to attach a UE with this configuration using a USRP? I would probably start there. See https://docs.srsran.com/projects/project/en/latest/tutorials/source/cotsUE/source/index.html

[andrepuschmann]

Another idea you could test is with a configuration flag that circumvents an issue we've seen with some Benetel RU models where the transmission on antenna port1 is corrupted. It's not 100% confirmed so not part of the application note yet. Anyway, try to follow the suggestions in https://docs.srsran.com/projects/project/en/latest/tutorials/source/oranRU/source/rus/r550.html and configure the RU in 100MHz, Long-format, etc. and make sure timing/sync is perfect.

Then in the config file swap the first and second DL antenna port, i.e., use:

  dl_port_id: [1, 0, 2, 3]

Then start the gNB and see if the UE can find the cell and sends a PRACH. Run the gnb with PHY logs in info and you should see some activity.

If this doesn't work, try:

ru_ofh:
  ..
  is_dl_broadcast_enabled: true
  ..
  cells:
      dl_port_id: [0, 1]
      ul_port_id: [0, 1]
cell_cfg:
  ..
  nof_antennas_dl: 2
  nof_antennas_ul: 2
  ..

Or try both anyway and report your findings back.

[hakegard]

Hi @ismagom and @andrepuschmann,

Thanks for the updates. I modified the config file as you proposed and now both the modem and the phone connect to the network. I have only tested the 4x2 configuration. The data rates we obtain are about 300 Mbps DL and 35 Mbps UL with the 7:2 TDD scheme.

@ismagom, 24271 is the PLMN of the 5G SA license we have obtained from the Norwegian Communications Authority. The license is for 80 MHz bandwidth, so I hope that it will be possible to also operate with smaller bandwidth than 100 MHz.

[ismagom]

Which of the changes fixed the issue?

You should reach better speeds. Can you share the console trace and gnb log with level warning?

Thanks

[hakegard]

The UEs are able to connect with both prach_port_id: [4,5] and prach_port_id: [4,5,6,7] and with both dl_port_id: [1, 0, 2, 3] and dl_port_id: [0, 1, 2, 3]. It is required that ul_port_id: [0,1] and not ul_port_id: [0,2], and as you wrote prach_frequecy_start: 0.

See the console trace below. The RU transmit power is relatively low for this trace. Tests with higher transmit power generally gives the same mcs, although sometimes it jump up to 28. The cqi is then 15. The DL data rate is about the same with higher transmit power, although rates above 400 Mbps have been seen. The gnb log file is attached.
gnb.log

  |--------------------DL---------------------|-------------------------UL------------------------------
 pci rnti | cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  rsrp  mcs  brate   ok  nok  (%)    bsr    ta  phr
   1 4601 |  11   4   19   4.9M  250    2   0%      0 |  27.7 -15.7   27   247k   47    5   9%      0   0us   10
   1 4601 |  11   4   20    93k   28    0   0%      0 |  26.5 -18.8   28    67k   16    1   5%    745   0us   11
   1 4601 |  11   4   20   405k   96    4   4%      0 |  26.0 -18.6   28   230k   48    1   2%      0   0us   11
   1 4601 |  11   4   19   519k  110    2   1%      0 |  25.8 -18.1   28   536k   45    3   6%      0   0us   12
   1 4601 |  11   4   19   118k   49    0   0%      0 |  24.0 -20.3   28   125k   22    6  21%      0   0us   14
   1 4601 |  11   4   19   576k   66    0   0%  7.04k |  23.5 -20.4   28   304k   19    4  17%      0   0us   12
   1 4601 |  11   4   19   1.4M  150    0   0%      0 |  25.3 -20.2   28   196k   48    0   0%      0   0us   12
   1 4601 |  11   4   19    81k   27    0   0%      0 |  22.4 -20.8   27   540k   15    6  28%      0   0us   12
   1 4601 |  11   4   20   417k   81    0   0%      0 |  25.5 -19.3   28   254k   40    0   0%      0   0us   11
   1 4602 |  12   3   20    14k   10    7  41%      0 |  31.0  -9.5   25    30k    7    0   0%      0   0us    4
   1 4601 |  11   4   20    15M  409    5   1%  8.73k |  25.3 -18.5   28   644k   71    6   7%  3.91k   0us   20
   1 4602 |  12   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    4
   1 4601 |  11   4   16    81M  591   42   6%      0 |  24.7 -19.0   27   516k   74    3   3%      0   0us   11
   1 4602 |  11   3   20   2.2k    2    0   0%      0 |  34.5  -9.4   28   8.7k    2    0   0%      0   0us    5

          |--------------------DL---------------------|-------------------------UL------------------------------
 pci rnti | cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  rsrp  mcs  brate   ok  nok  (%)    bsr    ta  phr
   1 4601 |  11   4   14    28M  306    0   0%    961 |  25.7 -18.7   28   353k   52    3   5%      0   0us   11
   1 4602 |  12   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  11   4   14    44M  653    0   0%      0 |  25.0 -18.9   27   491k   76    3   3%      0   0us   11
   1 4602 |  11   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  12   3   16   4.1M  197    0   0%      0 |  25.8 -19.2   28   380k   48    1   2%    384   0us   11
   1 4602 |  11   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  13   3   14    75M  549    3   0%      0 |  23.4 -20.4   26   395k   50   10  16%      0   0us   11
   1 4602 |  11   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  10   4   15    68M  602    0   0%   669k |  25.9 -19.2   28   387k   72    0   0%  1.45k   0us   11
   1 4602 |  11   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  11   4   16    92M  350    0   0%      0 |  25.9 -19.2   28   213k   42    0   0%      0   0us   10
   1 4602 |  11   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  11   4   17    36M  342    0   0%      0 |  25.5 -18.2   27   241k   53    3   5%      0   0us   10
   1 4602 |  11   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  11   4   18    47M  393    0   0%      0 |  26.1 -18.6   27   254k   50    4   7%      0   0us    7
   1 4602 |  11   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  11   4   18   4.0M  164    2   1%      0 |  25.7 -18.7   27   138k   29    0   0%      0   0us   11
   1 4602 |  11   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  11   4   18   1.6M  157    2   1%      0 |  25.5 -19.5   28   215k   51    0   0%      0   0us   11
   1 4602 |  12   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  12   4   18    33M  374    8   2%      0 |  25.7 -19.3   28   289k   58    1   1%      0   0us   10
   1 4602 |  12   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5

          |--------------------DL---------------------|-------------------------UL------------------------------
 pci rnti | cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  rsrp  mcs  brate   ok  nok  (%)    bsr    ta  phr
   1 4601 |  11   4   18    18k   11    2  15%      0 |  25.7 -19.8   28    22k    5    0   0%      0   0us   10
   1 4602 |  12   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  11   4   19   1.5k    1    0   0%      0 |  26.4 -19.3   28   4.4k    1    0   0%      0   0us   10
   1 4602 |  12   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  13   2    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us   10
   1 4602 |  12   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  11   4   20   3.4k    3    0   0%      0 |  27.0 -19.0   28   8.7k    2    0   0%      0   0us   10
   1 4602 |  12   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    5
   1 4601 |  12   2    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us   10
   1 4602 |  12   3   20   2.2k    2    0   0%      0 |  28.3  -8.2   28   8.7k    2    0   0%      0   0us    3
   1 4601 |  11   4   19    848    1    0   0%      0 |  21.7 -19.0   28   4.4k    1    1  50%      0   0us   10
   1 4602 |  11   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    3
   1 4601 |  11   4   18    12M  259    0   0%      0 |  26.0 -18.5   27   345k   45    3   6%      0   0us   12
   1 4602 |  12   3    0      0    0    0   0%      0 |  35.7  -8.4   28   4.4k    1    0   0%      0   0us    4
   1 4601 |  11   4   18   7.8M  220    1   0%      0 |  25.5 -19.7   28   180k   45    1   2%      0   0us   12
   1 4602 |  12   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    4
   1 4601 |  11   4   18    36k   20    0   0%      0 |  21.9 -20.8   25   752k   15    6  28%      0   0us   11
   1 4602 |  12   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    4
   1 4601 |  13   2    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us   11
   1 4602 |  12   3    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0   0us    4

[hakegard]

The 'is_dl_broadcast_enebled` line is commented out, and according to the config file in the docs the default value is 0. So yes.

When changing these parameters, I only stopped and restarted the gnb with the new parameter settings. I did not restart the CN and the UEs nor rebooted the RU. I don't know if this has an impact on the results...

Using the UDP option of iperf3, I get 820 Mbps DL.