Segmentation Fault

[aoweis]

Hi,

I opened dvbs2_tx_rx.grc simulation file, choose a ts file for the file source and ran it successfully.
I then split the flow into two separate files: one for transmitting and one for receiving, as I wanted to test it by actually transmitting over the air using two bladeRF 2.0 boards.

For the transmitter, I fed the virtual sink directly to the bladeRF sink (and removed the throttle block), and deleted all the other blocks. I setup the RF parameters, etc.

On another machine running Ubuntu 20.04, I setup the receiver by removing the transmitting blocks and connecting the bladeRF source to the DVB-S2 Rx Hier block. The RF parameters were also set up to match the frequency on the Tx side.

Both bladeRF boards are equipment with suitable antennas and bias tees and are located within 30 cm of each other.

I ran the receiver and I can see the frequency spectrum normally. Once I start the transmitter, after a few seconds the gnuradio on the Rx side just stops. I tried to run the gnuradio python file from the command line and I found that a segmentation fault is returned possibly once the receiver detects a specific signal coming from the transmitter. I ran the python file from gdb and got the following output:

hread 21 "symbol_sync_cc8" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd48e9700 (LWP 3933)]
0x00007ffff1454da0 in ?? () from /lib/x86_64-linux-gnu/libvolk.so.2.4
(gdb) backtrace
#0 0x00007ffff1454da0 in () at /lib/x86_64-linux-gnu/libvolk.so.2.4
#1 0x00007fffe7b8a65e in gr::dvbs2rx::symbol_sync_cc_impl::loop(std::complex const*, std::complex, int, int) ()
at /usr/local/lib/x86_64-linux-gnu/libgnuradio-dvbs2rx.so.1.0.0
#2 0x00007fffe7b8b0a1 in gr::dvbs2rx::symbol_sync_cc_impl::general_work(int, std::vector<int, std::allocator >&, std::vector<void const, std::allocator<void const*> >&, std::vector<void*, std::allocator<void*> >&) ()
at /usr/local/lib/x86_64-linux-gnu/libgnuradio-dvbs2rx.so.1.0.0
#3 0x00007ffff18188cd in gr::block_executor::run_one_iteration() ()
at /lib/x86_64-linux-gnu/libgnuradio-runtime.so.3.10.5
#4 0x00007ffff18879b7 in gr::tpb_thread_body::tpb_thread_body(std::shared_ptrgr::block, std::shared_ptrboost::barrier, int) ()
at /lib/x86_64-linux-gnu/libgnuradio-runtime.so.3.10.5
#5 0x00007ffff186ec7d in ()
at /lib/x86_64-linux-gnu/libgnuradio-runtime.so.3.10.5
#6 0x00007ffff186fa57 in ()
at /lib/x86_64-linux-gnu/libgnuradio-runtime.so.3.10.5
#7 0x00007ffff10da43b in ()
at /lib/x86_64-linux-gnu/libboost_thread.so.1.71.0
#8 0x00007ffff7d97609 in start_thread (arg=)
at pthread_create.c:477
#9 0x00007ffff7ed1133 in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

_

[aoweis]

Forgot to mention, I'm running gnuradio version 3.10.5.1 (Python 3.8.10).

[aoweis]

another note, I just noticed that I have the throttle block in the transmitter side, actually I did remove the throttle block while testing and I got the same segmentation fault. Thanks!

[igorauad]

Hi @aoweis ,

Thanks for your interest in the project and for reporting the issue.

So, first of all, for your use case, I would recommend running dvbs2-tx and dvbs2-rx directly, given that these have more command-line options. These apps are available globally after you install the project. Alternatively, you can run them directly from the apps/ directory.

Currently, the dvbs2-tx/rx apps don't have the bladeRF source and sink options, but you could create simple source/sink flow graphs and pipe into/from dvbs2-tx/rx as explained in #16 (comment).

Regarding the segfault, I am aware that the symbol synchronizer implementation is not perfect. I have experienced problems before, especially when using high oversampling ratios (the sps parameter). It would be nice if you could share an IQ recording so that I can reproduce the problem and work on a fix. In the meantime, you can switch to the GNU Radio's default in-tree symbol synchronizer, which should work more reliably. To do so, run dvbs2-rx with option --sym-sync-impl in-tree.

The difference between the two symbol synchronizer implementations has to do with a design tradeoff. The in-tree symbol sync is more flexible/configurable but slower. The OOT implementation (within gr-dvbs2rx) is less flexible but faster for the dvb-s2 use case.

Regarding the throttle block, indeed, it won't make a difference for the problem you are seeing. It seems the problem happens at the Rx side.

I hope this helps.

Thanks

[igorauad]

Hi again, @aoweis ,

I have tried to add support for the bladeRF on the dvbs2-tx and dvbs2-rx applications. However, the implementation requires some experimental validation, and I don't have a bladeRF to test with. So, in case you find a chance, could you give it a go? I would appreciate that.

The code is on a branch named bladerf. So, first of all, you will need to git checkout bladerf and rebuild/reinstall the project.

Next, based on the screenshots you sent, I think you need to run the Tx side with the following options:

tsp -I craft --pid 100 | dvbs2-tx --gui --freq 2.5e9 --sink bladeRF --bladerf-rf-gain 50 --bladerf-bias-tee

Similarly, you can run the Rx side with the following options:

dvbs2-rx --gui --freq 2.5e9 --source bladeRF --bladerf-rf-gain 50 --bladerf-bias-tee --sink file --out-file /dev/null

Note:

• The Tx side transmits random TS packets with PID 100 generated by the tsp -I craft plugin. Alternatively, you can feed a TS file by dropping the tsp command and running dvbs2-tx with option --source file --in-file [path-to-ts-file]. That is:

dvbs2-tx --gui --freq 2.5e9 --source file --in-file [path-to-ts-file] --sink bladeRF --bladerf-rf-gain 50 --bladerf-bias-tee

• Also, if you don't have tsp and would like to use it, you can try these installation instructions: https://igorauad.github.io/gr-dvbs2rx/docs/usage.html#further-information.
• Both apps are in GUI mode so that you can tune the RF gain in runtime using the range widget if necessary.

[aoweis]

Hi @igorauad,

It will be my pleasure to help in this.

I tried the command, but it seems there's something missing with the gui:

dvbs2-tx --gui --freq 2.5e9 --source file --in-file thefuryclip.ts --sink bladeRF --bladerf-rf-gain 50 --bladerf-bias-tee

and I get this error:

Traceback (most recent call last):
File "/usr/local/bin/dvbs2-tx", line 1010, in
main()
File "/usr/local/bin/dvbs2-tx", line 980, in main
tb = dvbs2_tx(options)
File "/usr/local/bin/dvbs2-tx", line 174, in init
self.setup_gui()
File "/usr/local/bin/dvbs2-tx", line 372, in setup_gui
initial_gain == self.blade_rf['rf_gain']
UnboundLocalError: local variable 'initial_gain' referenced before assignment

Similarly, on the rx side:
dvbs2-rx --gui --freq 2.5e9 --source bladeRF --bladerf-rf-gain 50 --bladerf-bias-tee --sink file --out-file /dev/null

gives this error:

Traceback (most recent call last):
File "/usr/local/bin/dvbs2-rx", line 1495, in
main()
File "/usr/local/bin/dvbs2-rx", line 1451, in main
tb = DVBS2RxTopBlock(options)
File "/usr/local/bin/dvbs2-rx", line 158, in init
self.setup_gui()
File "/usr/local/bin/dvbs2-rx", line 515, in setup_gui
initial_gain == self.blade_rf['rf_gain']
UnboundLocalError: local variable 'initial_gain' referenced before assignment

I then tried to run without the --gui.

On the Tx side
dvbs2-tx --freq 2.5e9 --source file --in-file thefuryclip.ts --sink bladeRF --bladerf-rf-gain 50 --bladerf-bias-tee

The bladeRF seems to initialize properly but then I get a message saying "buffer submission requested, but no transfers are currently available". Here is the last couple of messages:

[bladeRF sink] init: Buffers: 512, samples per buffer: 4096, active transfers: 32
[bladeRF sink] bladerf_sink_c: DEBUG: initialization complete
[DEBUG @ host/libraries/libbladeRF/src/board/bladerf2/bladerf2.c:1062] bladerf2_set_sample_rate: enabling 4x decimation/interpolation filters
[VERBOSE @ host/libraries/libbladeRF/src/board/bladerf2/common.c:343] check_total_sample_rate: active_channels=0, rate_accum=0, maximum=80000000
[bladeRF sink] start: DEBUG: starting sink
[VERBOSE @ host/libraries/libbladeRF/src/backend/usb/nios_access.c:429] nios_config_read: Read 0x00000801
[VERBOSE @ host/libraries/libbladeRF/src/backend/usb/nios_access.c:440] nios_config_write: Wrote 0x00000801
[VERBOSE @ host/libraries/libbladeRF/src/board/bladerf2/common.c:343] check_total_sample_rate: active_channels=1, rate_accum=2000000, maximum=80000000
[DEBUG @ host/libraries/libbladeRF/src/streaming/sync.c:1001] sync_tx: Worker is now running.
[DEBUG @ host/libraries/libbladeRF/src/backend/usb/libusb.c:1426] Non-blocking buffer submission requested, but no transfers are currently available.

If I run a spectrum analyzer using the other bladeRF, I can see some signal being transmitted:

On the Rx side
Note: I'm starting the receiver before the transmitter to make sure I'm capturing the entire file.

The receiver seems to be working fine, waiting for data. I modified the output file to received.ts

dvbs2-rx --freq 2.5e9 --source bladeRF --bladerf-rf-gain 50 --bladerf-bias-tee --sink file --out-file received.ts

Here is the output of the bladeRF:

[bladeRF source] start: DEBUG: starting source
[VERBOSE @ host/libraries/libbladeRF/src/backend/usb/nios_access.c:429] nios_config_read: Read 0x00000801
[VERBOSE @ host/libraries/libbladeRF/src/backend/usb/nios_access.c:440] nios_config_write: Wrote 0x00000801
[VERBOSE @ host/libraries/libbladeRF/src/board/bladerf2/common.c:343] check_total_sample_rate: active_channels=1, rate_accum=2000000, maximum=80000000
[DEBUG @ host/libraries/libbladeRF/src/streaming/sync.c:433] sync_rx: Worker is idle. Going to reset buf mgmt.
[DEBUG @ host/libraries/libbladeRF/src/streaming/sync.c:454] sync_rx: Reset buf_mgmt consumer index
[DEBUG @ host/libraries/libbladeRF/src/streaming/sync.c:469] sync_rx: Worker is now running.

I check the size of the received.ts file and it remains zero. Nothing seems to be received.

[igorauad]

Hi @aoweis ,

Thanks for the feedback. My bad, silly mistakes on the GUI implementation. I've pushed a fix to the bladerf branch. Could you pull those and try again?

Once you run the Rx in GUI mode, could you share a screenshot? Also, you can run the rx with option -d5 so that it prints lots of debug logs.

[aoweis]

debug.zip
Hi @igorauad,
I did three trials as detailed below, in summary, the standard symbol synchronizer gives a segfault and the in tree one did not work for me, as I got several buffer over run messages in the beginning (is it because it's slow as you mentioned?). I tried to start the Tx first and the the Rx so I miss the initial synch word and the receiver seems to be synchronized.

Here are the details. Note, the attached zip file contains the output debug message for all trials.

Trial 1: with standard OOT symbol synchronizer (start Rx then Tx)

Command:
dvbs2-rx --gui -d5 --freq 2.52e9 --source bladeRF --bladerf-rf-gain 13 --bladerf-bias-tee --sink file --out-file received.ts > debug_trial1.txt

Result: segfault – core dumped

Trial 2: with standard OOT symbol synchronizer – start Tx then Rx

Command:
dvbs2-rx --gui -d5 --freq 2.52e9 --source bladeRF --bladerf-rf-gain 13 --bladerf-bias-tee --sink file --out-file received.ts > debug_trial2.txt

Result: seems to be synchronized but file size is still zero

Trial 3: with Gnu Radio in tree symbol sycnhronizer

Command:
dvbs2-rx --gui -d5 --freq 2.52e9 --source bladeRF --bladerf-rf-gain 13 --bladerf-bias-tee --sink file --out-file received.ts --sym-sync-impl in-tree > debug_trial3.txt

Result: several Rx over run messages from bladeRF in the beginning only. Rx file size is still zero.

[igorauad]

Hi @aoweis ,

Cool, thanks for sending further info. I have a couple of comments:

1. Your trial 2 achieved frame sync lock. It wasn't a stable lock, but it did lock. Whenever you see a PL Sync Output on the GUI, the PL Sync block has achieved frame lock. The debug logs also confirm that with logs like Peak after: 32490; Timing Metric: 57.600128; Locked: 1. However, the FEC stages could not decode the FECFRAME (check with cat debug_trial2.txt | grep -e ldpc -e bch), so you didn't obtain any data on the output ts file.
2. Please make sure to put the Tx data in repeat so that the Tx application does not run out of data to send. Sorry, I should have told you that earlier. By default, dvbs2-tx does not repeat the input file. You need to run it with the --in-repeat option.
3. You are not changing the symbol rate and rolloff, which means you are transmitting with the default 1 Mbaud symbol rate and 0.2 rolloff factor. Hence, we should see a 1.2 MHz bandwidth on the spectrum plot. Your trial 1 shows ~2 MHz (more on that below) and some small DC offset. I need to familiarize myself with how the bladeRF addresses DC offset. I will need to take a look and see if there is anything to adapt on how the dvbs2-rx launches the bladeRF source block. With USRPs, the trick is to tune the RF frontend sufficiently off the nominal RF center frequency so that the DC component falls out of band. The RF downconversion puts the signal on an intermediate frequency, and the DSP does the remaining frequency translation to put the signal in baseband. See the notes about two-stage tuning and comments on dvbs2-rx regarding how it launches the USRP source block.
4. Your trials 2 and 3 show the expected 1.2 MHz band, but they also show significant spectral shoulders. There are various reasons these shoulders can arise, but a common one that could be your case is nonlinear distortion causing spectral dispersion. That could happen, for instance, if the SDR board clips the transmitted IQ samples due to having a magnitude beyond the maximum supported one. However, although I'm not familiar with the max amplitude of the bladeRF, I'd think the IQ samples can oscillate within +-1, and dvbs2-tx guarantees that (run it with --gui --gui-all to check). So, the second and more likely possibility is that your Tx gain may be too high or your external Tx amp (you are using one, right?) is saturating. Try running with less Tx gain and see if the shoulders go away.
5. For an initial experiment, after fixing the above, it may still be helpful to activate PL pilots on transmission. On the Tx app, add the --pilots flag. On the Rx app, add the --pilots on option. I would recommend trying without pilots only after the transmission with pilots works well.

[aoweis]

Hi @igorauad,

It worked! Not 100% sure which one fixed it, but I was finally able to watch the movie on the Rx machine after doing all the above corrections. I was using Gnuradio's in tree symbol synch with pilots. If I remove the pilots, I'll have high bit error rate.

Regarding the clipping at the Tx side, using --gui-all, I found that the I/Q amplitudes were +/- 0.7. I did reduce the gain from 50 to 30 but I don't think this had any effect on the overall performance. I think that the Tx gain works on the RF signal, so the only problem with too much gain could be to saturate the Tx amplifier. However, I'm using a Bias-Tee PA from Nuand which should be compatible with BladeRF. I'd assume if a high gain could saturate the PA, Nuand would have warned the users, which is not the case.

As for the DC offset, I built a BPSK on Gnuradio and I had to manually follow the same procedure as the USRP. I tuned the RF freq at (f_rf - offset), and used a frequency shift block with frequency = -offset. The bladeRF block in GnuRadio has an option to automatically correct for DC offset, but I couldn't confirm it's working properly.
debug_trial4.zip

Thanks for your help!

[igorauad]

@aoweis ,

Glad to know it worked!

If I remove the pilots, I'll have high bit error rate.

Indeed, the current implementation needs more work to improve the pilotless mode. The current limitation is documented here: https://github.com/igorauad/gr-dvbs2rx/blob/master/docs/support.md. This task has been on my backlog for some time, but I still plan on working on it soon.

Regarding the clipping at the Tx side, using --gui-all, I found that the I/Q amplitudes were +/- 0.7.

That is the expected range. However, I thought more about it and realized the scaling was still wrong, although good enough to prevent clipping issues. The previous implementation scaled the samples so that the complex IQ magnitude would be within +-1.0. However, what we want is for the independent I and Q amplitudes to range within +-1.0. So I have updated that on the master branch, which now includes the preliminary support for the bladeRF.

I did reduce the gain from 50 to 30 but I don't think this had any effect on the overall performance. I think that the Tx gain works on the RF signal, so the only problem with too much gain could be to saturate the Tx amplifier.

Right. Perhaps you could try the Tx gain change on-the-fly by running dvbs2-tx in GUI mode. See if the Rx can lock either with 30 or 50 dB gain, or if it finds trouble locking as you increase the Tx gain.

As for the DC offset, I built a BPSK on Gnuradio and I had to manually follow the same procedure as the USRP. I tuned the RF freq at (f_rf - offset), and used a frequency shift block with frequency = -offset. The bladeRF block in GnuRadio has an option to automatically correct for DC offset, but I couldn't confirm it's working properly.

I had a quick look at the bladeRF source block. As far as I can tell, it does not do anything when the DC offset mode is set: https://github.com/Nuand/gr-bladeRF/blob/main/lib/bladerf/bladerf_source_c.cc#L434. I believe you may need to calibrate the DC offset as instructed in https://github.com/Nuand/bladeRF/wiki/DC-offset-and-IQ-Imbalance-Correction.

Your approach based on DSP also works at the expense of CPU usage. The USRP does that on the FPGA. However, for experimental purposes, or when there are sufficient CPU resources, the approach you adopted shouldn't be a problem.

Let me know if there is anything more I can help with.

[aoweis]

Hi again @igorauad,

Is there a way to reduce the end to end delay when running dvbs2-tx and dvbs2-rx over an RF link? I did some experiments based on our discussion above using bladeRF and two computers, and I calculated a delay of approx 11 seconds end to end. I was piping the webcam in the Tx machine using G-Streamer to dvbs2-tx as follwoing:

mkfifo video1.pipe
gst-launch-1.0 v4l2src ! videoconvert ! "video/x-raw,format=I420,width=640,height=480,framerate=30/1" ! x264enc tune=zerolatency ! h264parse ! mpegtsmux ! queue ! filesink location="video1.pipe"

and in another terminal:
dvbs2-tx --gui --freq 2.52e9 --source file --in-file "video1.pipe" --sink bladeRF --bladerf-rf-gain 30 --bladerf-bias-tee --pilots

and using this command in the Rx machine:

dvbs2-rx --gui --freq 2.52e9 --source bladeRF --bladerf-rf-gain 10 --bladerf-bias-tee --sym-sync-impl in-tree --pilots on --out-fd 3 3>&1 1>&2 | tsp --realtime --buffer-size-mb 1 -O play

Note that the Gstreamer itself causes around 1 sec delay maximum (I figured this out by receiving the stream in another terminal on the same Tx machine and playing it).

Is this delay caused by slow hardware? can we achieve lower delay?

Thank you

[igorauad]

Hi @aoweis ,

Thanks for the question. That is an interesting investigation.

Let's try to understand some possible sources of delay.

RRC Filtering on Tx

The root-raised cosine (RRC) filter used for pulse shaping on Tx is deliberately designed with a relatively significant delay. The motivation is due to the RRC response's dependence on the truncation length. In summary, the longer the filter, the lower the "truncation effects, " resulting in a sharper stopband attenuation with lower sidelobes. This is generally more important on the Tx (uplink) side than on the Rx side so that the uplink transmitter minimizes interference to adjacent carriers.

That being said, the default value of the --rrc-delay parameter is 50 on dvbs2-tx and only 5 on dvbs2-rx. This is because, on the latter, we prioritize a shorter filter to reduce the computational cost.

Now, a delay of 50 symbols is not a lot. With 1 Msps, that is only 5 microseconds. Hence, even though you can tune --rrc-delay from the command line, I'd not expect it to make any significant difference in latency at the level you are seeing (in the order of seconds).

FEC

The inner LDPC decoder block waits until a few frames are collected before processing them all at once. By doing so, it can benefit from SIMD speed gains. The number of frames processed in one go depends on the SIMD architecture available in your host. The longest wait is with AVX2, which processes 32 frames at a time. So the worst-case latency introduced at this stage is that of 32 frames. In contrast, the outer BCH decoder currently processes one codeword/frame at a time, so it does not impose a latency bottleneck.

So, next, let's figure out how much latency the LDPC decoder can add. I see you are transmitting with the default frame parameters, which are:

• MODCOD: QPSK 1/4.
• FECFRAME size: Normal.
• Pilots: disabled.

Hence, each PLFRAME has a length of 32490 symbols, which you can check in Python with the following:

from gnuradio import dvbs2rx
dvbs2rx.pl_info(constellation='qpsk', code='1/4', frame_size='normal', pilots=False)

With the default 1 Msps baud rate, each PLFRAME has a duration of roughly 32.5 ms. Hence, the FEC layer waiting for 32 frames may introduce a latency of up to ~1 second (or 1040 ms). Also, in this analysis, I'm only considering the minimum time to accumulate 32 frames, which is sample-rate-dependent, and I'm excluding the time to process such frames, which is CPU-dependent.

You can very loosely validate this delay by running a loopback simulation with real-time flows, like so:

cat example.ts | timeout -k 1s 4s dvbs2-tx --out-real-time | dvbs2-rx --in-real-time --sink file --out-file /tmp/decoded.ts

In this case, the timeout command kills the Tx app after around 5s running.

Each frame carries Kbch bytes (BCH message length) and Kbch=16008 bits (2001 bytes) with QPSK 1/4 normal FECFRAMES. So, if I'm not missing anything, by simulating for 5 seconds, at best, it would process 153 frames, which is enough to convey 153 * 2001 = 306153 bytes. In reality, you should see some decoded data with stat /tmp/decoded.ts but certainly not 306153 bytes given that the applications take some time to launch. You can play with the timeout command and see that you can get some data by running for shorter durations.

NOTE: the size of decoded.ts will always be a multiple of 188 bytes (MPEG TS packet length), not 2001 bytes.

TSP

Finally, I'd recommend checking the various tsp options to control buffering. It has options like:

• --realtime.
• --buffer-size-mb (which you are already using).
• --max-flushed-packets.
• --max-input-packets.

See, e.g., this thread. That may make a more significant difference in latency.

I'd be curious to know how this investigation goes.

Regards

[igorauad]

Hi @aoweis , I will close this issue for now. Please feel free to reopen if you have more questions.