Unable to establish RRC connection between gNB and UE

[tilldroemmer]

Issue Description

Using srsRAN_Project and this Tutorial we want to establish a connection between our gNB (X310) and UE (B200mini).
In the past we have always been able to get at least a RRC connection, at one time even a PDU session.
However we had to move labs and had to change our hardware, so we set up our new testbed. This time, we are not able to create a RRC connection.

Setup Details

Our UE and gNB are both connected to the Leo-Bodnar 10MHz Clock
Core, UE and gNB are running on different physical machines.

UE (running on Raspberry Pi 4 Model B)

• b200mini
• uhd 3.15
• srsRAN_4G Built in Release mode using commit eea87b1d8 on branch master

gNB:

• X310
• uhd 3.15
• srsRAN_Project Built in Release mode using commit 0b2702c on branch main

Core

• Open5Gs v2.7.0

Expected Behavior

[What you expect to happen]
The UE should connect to the gNB, the RRC connection should be established and later on the PDU session should be established as well.

Actual Behaviour

[What happens instead e.g. error message]

Active RF plugins: libsrsran_rf_uhd.so libsrsran_rf_zmq.so
Inactive RF plugins:
Reading configuration file 0102.conf...

Built in Release mode using commit eea87b1d8 on branch master.

Opening 1 channels in RF device=uhd with args=type = b200, clock=external
Supported RF device list: UHD zmq file
[INFO] [UHD] linux; GNU C++ version 10.2.0; Boost_107100; UHD_3.15.0.0-4
[INFO] [LOGGING] Fastpath logging disabled at runtime.
Opening USRP channels=1, args: type=b200,master_clock_rate=23.04e6
[INFO] [UHD RF] RF UHD Generic instance constructed
[INFO] [B200] Detected Device: B200mini
[INFO] [B200] Operating over USB 3.
[INFO] [B200] Initialize CODEC control...
[INFO] [B200] Initialize Radio control...
[INFO] [B200] Performing register loopback test...
[INFO] [B200] Register loopback test passed
[INFO] [B200] Asking for clock rate 23.040000 MHz...
[INFO] [B200] Actually got clock rate 23.040000 MHz.
Setting manual TX/RX offset to 300 samples
Waiting PHY to initialize ... done!
Attaching UE...
RF status: O=4, U=0, L=0
[INFO] [UHD RF] Tx while waiting for EOB, timed out... 2.35479 >= 0. Starting new burst...
Random Access Transmission: prach_occasion=0, preamble_index=0, ra-rnti=0xf, tti=9461
RF status: O=24, U=4, L=106
Random Access Transmission: prach_occasion=0, preamble_index=0, ra-rnti=0x39, tti=9614
RF status: O=34, U=0, L=0
[INFO] [UHD RF] Tx while waiting for EOB, timed out... 4.57974 >= 4.57644. Starting new burst...
RF status: O=35, U=1, L=216
RF status: O=15, U=0, L=90
RF status: O=29, U=1, L=0
RF status: O=32, U=0, L=0
[INFO] [UHD RF] Tx while waiting for EOB, timed out... 8.10993 >= 8.01963. Starting new burst...
RF status: O=19, U=5, L=79
RF status: O=33, U=0, L=0
[INFO] [UHD RF] Tx while waiting for EOB, timed out... 10.86 >= 10.8363. Starting new burst...
[INFO] [UHD RF] Tx while waiting for EOB, timed out... 10.8896 >= 10.8363. Starting new burst...
RF status: O=33, U=2, L=20
^CStopping ..
Saving MAC PCAP (DLT=149) to /tmp/ue_mac.pcap
--- exiting ---

We noticed that in the UE.log the MIB seems to be problematic.
As well as there is always a RAR timeout
RAR Timer expired. RA response not received within the response window
And a Handling Timeout
Handling Timer Expired

Steps to reproduce the problem

UE.zip
gnb.zip

Additional Information

[Any additional information, configuration or data that might be necessary to reproduce the issue]

We played around with the Gains and time_adv_nsample for a while. We even tried all time_adv_nsample from 0 to 600 in steps of 25

[dhiaboujebha]

@andrepuschmann could you please help us through this issue?

[pgawlowicz]

could you check the cpu load of the host running srsUE? for example with htop and post a screenshot here

[tilldroemmer]

sorry for the confusion, the cpu load was from the wrong testbed. I will upload the load of the correct one as fast as possible

[tilldroemmer]

Hi @pgawlowicz I am now at the respective setup and meassured the load.
I also attached the specs of the Computers


specsUE.txt
specsgNB.txt

[pgawlowicz]

hmm, RPI might be too slow for 10MHz bandwidth, could you try with 5MHz?

[tilldroemmer]

Thank you so much for the fast response

gNB

UE

Here is a screenshot of the UE while starting

[pgawlowicz]

could you share your configs?

[tilldroemmer]

2401_srsue.conf.txt
gnbv2.yaml.txt

[tilldroemmer]

i was unsure about the coreset parameter in the 5MHz conifg

[pgawlowicz]

the configs are correct. Did you try running it with an external clock? also you might try to tune tx/rx gains

[tilldroemmer]

We did run it with an external clock, but only with the 10MHz or 20MHz config.
We will try it with different gains.

[mohmd-abzd]

I have a similar setup as yours @tilldroemmer .

Try using the following config for the gNodeB/x310:

  device_driver: uhd                                              # The RF driver name.
  device_args: type=x300,master_clock_rate=184.32e6,send_frame_size=8000,recv_frame_size=8000                                          
  #sync: external                                                
  srate: 30.72

By the way, I am using 10G interface with the x310.

[tilldroemmer]

Thank you for the recommendation @mohmd-abzd, I will try it soon. you are also using a raspberry Pi? Or do you mean similar regarding the USRPs?

[mohmd-abzd]

No I am using a laptop with 4-cores CPU.
For the gNodeB I have x300 and the UE I am using a b210. I noticed that we are using almost the same configuration parameters but the only difference is the sampling rate. Hope it works for you.

[pgawlowicz]

@tilldroemmer any updates on your issue? Could you try tuning TX/RX gains on both gnb and srsUE?

[tilldroemmer]

no updates yet, I will remove to the setup tomorrow. we already tried different gains in steps of 5, but I can try again.
I will also grab more logs and traces tomorrow.

[tilldroemmer]

I am trying the different gains on the devices and generating logs right now.
I just checked the first logs and I do not understand the way the gains on the UE side are handled. I changed the gains in the config (I checked it is the correct config file) but I can even configure for 120, although that is definiteley out of spec.

For understanding, the notation of the logs is UE_X_Y_Z.log and gNB_X_Y_Z.log
X is the gnb gain
Y is the UE gain
Z is the number of the measurement (for each config 3)

gNB_logs1.zip
UE_logs1.zip
UE_logs2.zip

[pgawlowicz]

@tilldroemmer do you run the experiments over an RF cable?

In the UE logs you need to filter for PRACH:|PDSCH: with grep -rE "PRACH:|PDSCH:" ./ then you can see that the UE tries to RACH and receives only a single PDSCH with high SNR for example:

PDSCH: cc=0 pid=0 si-rnti=0xffff prb=(1,8) symb=(2,13) CW0: mod=QPSK tbs=80 R=0.380 rv=0 CRC=OK iter=1.0 evm=0.02 t_us=610 epre=+30.8 snr=+37.6 cfo=-25.9 delay=+0.0  cfo=-25.9

In GNB logs you need to filter for PRACH:|PUSCH: with grep -rE "PRACH:|PUSCH:" ./ :
then you can see that gnb detected PRACH only in gnb_5_35_2.log and gnb_5_65_1.log and SNR of PUSCH was really low.

Could you share your current configs?

[tilldroemmer]

@pgawlowicz I saw that as well, i also noticed that in the UE there are a lot of late packets. Could this be a problem, or is the setup procedure robust enough?

I will also continue with higher gains on the gNB side

I am transmitting via cable in the moment.
X310 as gNB with srsRAN_Project v23.10.1
B200mini as UE with srsRAN v23.4.0
open5gs core v2.7.0

gnb.yaml.txt
ue.conf.txt

[pgawlowicz]

Do you have any attenuator in between?

[tilldroemmer]

yes a 30db antenuator on tx and rx as I believe national instrument stated on their website

[pgawlowicz]

you mean you are not adding any external attenuators, just using the built-in ones?

[tilldroemmer]

sorry for the confusion, I got two external 30db antennuators. I think these https://www.minicircuits.com/WebStore/dashboard.html?model=VAT-30%2B

[pgawlowicz]

ok, that is good. In the logs you have provided, do X and Y stands for gnb tx_gain and srsUE rx_gain? did you try other values for gnb tx_gain than 5?

[tilldroemmer]

Yes X is for gNB and Y is for UE
I am currently still running different gains, below are the logs of gains 10 and 15 from the gNB.
I am working on more, but it takes time, as I want to provide 3 logs of each gain to provide some redundancy.

gNB_logs2.zip
gNB_logs3.zip

UE_logs3.zip
UE_logs4.zip

[pgawlowicz]

you do not need to repeat it 3 times. If the parameters are good, it should work out of the box.
what about the time_adv_nsamples = 300 in srsUE config?

[tilldroemmer]

in my current config I uploaded before I have time_adv_nsamples = auto

gnb.yaml.txt ue.conf.txt

[pgawlowicz]

did you try with time_adv_nsamples = 300 ?

[tilldroemmer]

I can try it now, what should I use for time_alignment_calibration on the gNB side?

[pgawlowicz]

Please leave auto for gnb

[tilldroemmer]

UE with 300 nsample ue.log

gNB with auto nsample gnb.log

[pgawlowicz]

do you run the srsUE on RPI4? if so, could you reduce BW to 5 mhz

[tilldroemmer]

Yes I am running the UE on the RPI4, I tried with 5MHz:
ue5MHz.log
ue_5MHz.conf.txt

gnb_5MHz.log
gnb_5MHz.yaml.txt

[pgawlowicz]

could you check the load on rpi with htop when runing srsue?

[tilldroemmer]

HTop before running srsUE

HTop while running srsUE

[pgawlowicz]

this is with 5 or 10mhz bw?

Could you experiment with the following parameters in srsue config:

phy.pdsch_max_its = 4 -> reduce number of PDSCH decoding iterations (should decrease load)
phy.sync_cpu_affinity = 0 -> pin sync thread to a specific CPU core

phy.nof_phy_threads = 1, 2 or 3 -> set the number of decoding threads
phy.worker_cpu_mask = 2,4,8 -> set the cpu mask so that phy threads are allocated to a different CPU core then sync thread

[tilldroemmer]

this is with 5 MHz.

thank you for the great help today, I don't know If I can fit these tests in this week.

[tilldroemmer]

I just produced this two logs for the recommended configuration. The used configuration is also in the zip.
I don't understand, why there are so many out of sync packets although I use an external Clock.

log.zip

[pgawlowicz]

It seems that the UE did not even detect the cell, and did not even try to attach.
If you are using LEO Bodnar could you set clock=external and sync=default ?

[tilldroemmer]

I changed to a different computer on UE side to see where the problem is.
With a Fujitsu mini pc with 4 Cores and x86 architecture the config I used before works.

I transmitted via antenna and used a 10 MHz setting and was able to get a PDU session establishment.

We will now continue to look for the fault, our idea is to use a different RP4 this RP is a plain installation, we hope this might solve the problem for now, I will come back to the issue as soon as I have further intel