Freeze in Gqrx with hackRF 2021-03-1 driver version

[Potomac]

Steps to reproduce

1. Install Gqrx software, 2.14.4 version
2. Use hackRF device (february 2014 PCB version)
3. Update firmware to 2021-03 version
4. Update hackRF driver to 2021-03-1 version
5. Use linux (archlinux 64 bits)
6. Run Gqrx
7. Switch several times FM to AM (and vice-versa)
8. Randomly Gqrx will freeze

Expected behaviour

Gqrx should no randomly freeze during operation

Actual behaviour

Randomly, when switching frequency mode (AM, FM, LSB etc...) then Gqrx can freeze, I hear no sound from hackRF, it can happen 30% of the time when switching frequency mode,

if I use 2018 hackrf driver version then there is no freeze, no bugs, all is Ok with Gqrx,
my hackRF is a 2014 PCB version, firmware has been updated to 2021-03

Version information

archlinux 64 bits

hackrf_info version: 2021.03.1
libhackrf version: 2021.03.1 (0.6)
Board ID Number: 2 (HackRF One)
Firmware Version: 2021.03.1 (API:1.04)

Gqrx 2.14.4

[Potomac]

If I use the official appimage of Gqrx (last version, same as archlinux package : 2.14.4) then there is no freeze, no bug with hackRF 2021-03-1 driver :
https://github.com/csete/gqrx/releases

The bug occurs with the archlinux package of Gqrx.

[Potomac]

But the appimage can also contain hackRF lib in 2018-01 version, that can explain why there is no freeze.

[straithe]

Have you tried your HackRF with other software on Arch?

[Potomac]

Yes, I tried 2 softwares : gnu radio and hackTV, both work with hackRF 2021-03-01.

Gqrx freezes randomly with hackRF 2021-03-01 when switching modes (AM, FM, LSB), and sometimes when I start/stop hackRF with the menu "file -> start dsp", and "File -> stop dsp", it seems that the driver in 2021-03 version may hang the device in some conditions.

[Potomac]

@straithe : do you have a test script/binary that can check if all features, functions of hackRF driver 2021-03-01 are Ok ?

I use 2 commands for checking transmission and reception and it seems ok :
hackrf_transfer -t /dev/zero
hackrf_transfer -r /dev/null -s 21500000

But I feel these commands are not enough for testing hackRF driver, if Gqrx 2.14.4 manages to crash the 2021-03 driver (and not with the 2018 version) then it is worth to investigate, there is perhaps something wrong in the 2021-03 driver version about some functions, a regression.

For tracking the faulty commit that introduced the bug then a solution would be to use the git-bisect feature of git.

Gnu-radio companion seems to work fine, but I don't know if it was built with libhackrf 2021 version or libhackrf 2018 version.

[miek]

I haven't had any luck reproducing this so far (running Ubuntu 20.04 & pybombs GR/GQRX).

Could you run gqrx in a terminal, reproduce the issue, and paste the terminal output here?

[Potomac]

@miek : Don't forget that ubuntu 20.04 is not a rolling release distro, some libraries, and linux kernel may not be at the last version,
Archlinux is a rolling release (all packages are up to date), then bugs/regression can occur more easily,

I ran gqrx in a terminal, when the freeze occurs then the output doesn't show unusual information :

$ gqrx
gr-osmosdr 0.2.0.0 (0.2.0) gnuradio 3.8.2.0
built-in source types: file fcd rtl rtl_tcp uhd hackrf bladerf rfspace airspy soapy redpitaya 
gr::log :WARN: file_source0 - file size is not a multiple of item size
Resampling audio 96000 -> 48000
gr::fft: can't import wisdom from /home/cesar/.gr_fftw_wisdom
BandPlanFile is /home/cesar/.config/gqrx/bandplan.csv
BookmarksFile is /home/cesar/.config/gqrx/bookmarks.csv
[INFO] [UHD] linux; GNU C++ version 10.2.0; Boost_107500; UHD_4.0.0.0-0-unknown
[ERROR] SoapySDR::loadModule(/usr/lib/SoapySDR/modules0.7/libhackrfSupport.so)
  duplicate entry for hackrf (/usr/lib/SoapySDR/modules0.7/libHackRFSupport.so)
ATTENTION: default value of option vblank_mode overridden by environment.
gr-osmosdr 0.2.0.0 (0.2.0) gnuradio 3.8.2.0
built-in source types: file fcd rtl rtl_tcp uhd hackrf bladerf rfspace airspy soapy redpitaya 
Using HackRF One with firmware 2021.03.1

perhaps there is a debug mode for qgrx in order to have more output at the terminal ?

I don't know what is pybomb, I use the regular gqrx package from archlinux, there is no pybomb used by gqrx,

my python version is 3.9.4,

• gnu-radio : 3.8.2
• soapysdr 0.7.2
• kernel linux : 5.11.16
• boost-libs 1.75.0
• Qt5 version : 5.15.2
• hackRF lib : 2021.03.1

my hackRF is a chinese version bought very recently, no problem with other SDR applications (gnu-radio companion, hackTV),
if I downgrade hackRF lib to 2018 version then gqrx doesn't have freezes.

I feel that the bug occurs when gqrx tries to start/stop the hackRF device, randomly it can create freeze, perhaps the way you use the hackRF lib is not fully appropriate, a more "conservative way" by adding additionnal time (when calling hackRF functions ?) may avoid the freeze ?

[Potomac]

I made a git bisect and I found the faulty commit :

1442014a80b5ce33f16ef665538c2881b37d3a68 is the first bad commit
commit 1442014a80b5ce33f16ef665538c2881b37d3a68
Author: Matioupi <mathieu.peyrega@gmail.com>
Date:   Wed Oct 16 12:03:08 2019 +0200

    Modified hackrf_stop_tx and hackrf_stop_rx to first join the transfer thread
    before setting the receiver to OFF mode (cf. issue #650)

 host/libhackrf/src/hackrf.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

1442014

We should revert this faulty commit in order to fix the crash about stop_tx/tx commands that might occur on recent hackrf one chinese models, when using 2021-03-01 hackrf driver version.

The git bisect log :

git bisect start
# good: [cc7f599677e222a19a2ba4336a23e1fb507cf9f6] Pin OS X builds to Xcode 7.3 for now
git bisect good cc7f599677e222a19a2ba4336a23e1fb507cf9f6
# bad: [e6eb4ba29bbe5dc2fcd092e394188bc10a8bad54] Merge pull request #848 from mossmann/pre-release
git bisect bad e6eb4ba29bbe5dc2fcd092e394188bc10a8bad54
# good: [56b3bd0bed544cf40c9287c6d2b1c811608eda57] Merge pull request #546 from dominicgs/operacake_gpio
git bisect good 56b3bd0bed544cf40c9287c6d2b1c811608eda57
# good: [1c7bf39bda81accb67d04c46aa08808d863180b8] Merge branch 'develop'
git bisect good 1c7bf39bda81accb67d04c46aa08808d863180b8
# bad: [92ee1447143aed8d6b7e291e98e4139190206aac] Merge pull request #699 from miek/ui_disable
git bisect bad 92ee1447143aed8d6b7e291e98e4139190206aac
# good: [923f9fe617c9dcf00c97c18f5aa79e51720b9e59] Merge remote-tracking branch 'mossmann/master'
git bisect good 923f9fe617c9dcf00c97c18f5aa79e51720b9e59
# bad: [058276d010d437486c4e0ee4e0b7fe3f3e52f836] Merge pull request #671 from yhetti/acrylic_case
git bisect bad 058276d010d437486c4e0ee4e0b7fe3f3e52f836
# good: [1569737109283ef944c3ae4d803bbcbb3a523fbf] Merge pull request #607 from dominicgs/portapack_ui_opera_cake_coexistence
git bisect good 1569737109283ef944c3ae4d803bbcbb3a523fbf
# bad: [8ff56c615fa8835e8a0db36ed754e2793ada9a9b] Merge pull request #661 from mgesteiro/hackrf_transfer-fix
git bisect bad 8ff56c615fa8835e8a0db36ed754e2793ada9a9b
# bad: [1442014a80b5ce33f16ef665538c2881b37d3a68] Modified hackrf_stop_tx and hackrf_stop_rx to first join the transfer thread before setting the receiver to OFF mode (cf. issue #650)
git bisect bad 1442014a80b5ce33f16ef665538c2881b37d3a68
# good: [208fae75389e1888c6118f12b79c4ac7522fe372] Merge pull request #645 from jboone/master
git bisect good 208fae75389e1888c6118f12b79c4ac7522fe372
# first bad commit: [1442014a80b5ce33f16ef665538c2881b37d3a68] Modified hackrf_stop_tx and hackrf_stop_rx to first join the transfer thread before setting the receiver to OFF mode (cf. issue #650)

[Potomac]

Reverting this commit is not enough for solving freezes in Gqrx, as recent commits in hackrf change also start/stop rx, tx functions,
1442014 is just the first bad commit, the problem may be more deep, related to thread management, usb transfert, it will need more debugging for solving this bug.

[straithe]

Since we can't reproduce your issue, can you please upload a screen capture of you reproducing this issue?

[Potomac]

Here is the video in attachment, I used the last version of gqrx (2.14.5) and libhackrf 2021.03, with archlinux,
at the end of video I managed to trigger the freeze, by changing the mode (WFM to AM, then WFM).

The bug doesn't occur if I use the appimage version of qgrx (the official appimage that we can find on gqrx's github), but I think the appimage may contain hackRF lib in 2018-01 version, that can explain why there is no freeze when using appimage.

VID_20211028_223011_.mp4

[straithe]

Thank you for the video. That helps! Can you give me a screen shot of your right hand menu before and after you change from WFM to AM, then to WFM? I am updating my setup and trying to follow along with your steps exactly and I can't quite see what buttons you are pressing there.

[Potomac]

Hello,

here is a new screen capture, this time the freeze occurs after the change WFM to AM.

output.mp4

[straithe]

This issue may be related to #916. We are going to try to recreate that issue, address it, and hopefully provide a solution that you might be able to use as well. I'll update you as we have more information.

[martinling]

Now that I've figured out how #916 happens, I suspect that this issue has the same cause.

Could you test with my fix in #1029 and see if that solves this issue for you?

[Potomac]

@martinling : Can you provide a patch file compatible with the 2021-03 version of hackrf (last release version) ?

Your current commit is not compatible with the last release version of hackrf (date of 2021-03), because I tried to apply your modifications to hackrf.c file and I get a lot of errors during compilation, about undeclared functions (HACKRF_OPERACAKE_MAX_BOARDS, and unknow types like hackrf_operacake_freq_range).

[martinling]

@Potomac sure, here's a patch against 2021.03.01: bug-916-rebase-2021.03.01.patch.txt

You can also get the same result by cherry-picking my commit b346790e. It applies cleanly on top of the v2021.03.01 tag.

[Potomac]

@martinling : I tried your patch, I see no real difference, the bug is still here, gqrx freezes randomly after changing settings like frequency mode (AM, FM etc...), activating/deactivating RDS mode for example.

I suggest you to test with gqrx (not your test source code), and try to change settings like frequency mode, RDS, you have to test for a long period (sometimes the freeze doesn't occur immediately, you have to retry sometimes 20 times before the freeze occurs).

[martinling]

I'm afraid I haven't been able to reproduce this problem here. I'm running gqrx 2.14.4 on Debian. With firmware and libhackrf from 2021.03.01, I can switch back and forth between FM and AM over 100 times without it hanging.

[gozu42]

i am somewhat able to repro what is probably the same issue, even though i trigger it without any retuning.
fedora-35 vm, hackrf passed in through usbip.

gqrx starts up fine, i can rx some fm.
but if i repeatedly stop/start i am seeing ...

• after a repeat start gqrx again will "receive" unchanging data (wiggly line not wiggling, unchanging pattern in waterfall)
• after a stop the rx led stays on
• the next attempt to start will lock up gqrx
• if i kill gqrx in this state, and start it again, it will lock up right away (rx led stays on)
• if i kill gqrx, then run something like hackrf_sweep -1, the sweep will work, the rx led goes dark, gqrx works again (for a while)

wireshark usb capture of a gqrx session during the issue.
gqrx-stall-usb.zip

• 0sec start wireshark
• 9sec start gqrx
• 16.5sec start rx (working)
• 18.3sec stop rx
• 19.5sec start rx (not working)
• 29sec killed gqrx

[martinling]

@Potomac could you perhaps try running the test program (hackrf_loop.c) from #916?

If this is a hackrf bug rather than a gqrx one, it seems like it must involve some failure during a quick stop & start of the hackrf - which is exactly what that test program exercises, over and over again.

You should be able to build it against your installed libhackrf with:
gcc `pkg-config --cflags libhackrf` hackrf_loop.c `pkg-config --libs libhackrf` -o hackrf_loop

Then if you run ./hackrf_loop you should get output like the following, repeating indefinitely:

Start
Not started yet
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Finally started

If you see anything else (e.g. errors, hanging, or Not started yet repeating), then post it here - what you see may give us a clue as to what's going wrong in the gqrx scenario too.

[martinling]

@gozu42, was that result with or without my fix from #1029?

[gozu42]

@gozu42, was that result with or without my fix from #1029?

libhackrf version: git-b346790e (0.6)
Firmware Version: git-a83bfc4b (API:1.05)

so libhackrf with the #1029 changes, firmware from m0/buffer branch which "shouldnt" matter for this.
(and according to ldd, gqrx is using the right libhackrf version)

[martinling]

Aha! I've reproduced it now - using the same commits as @gozu42.

I wasn't able to get it to happen by switching AM/FM or the start/stop button, but using the keyboard shortcut (Ctrl-D) for start/stop and just holding it down, I triggered it after a few seconds. Terminal output shows:

Using HackRF One with firmware git-a83bfc4
Failed to start RX streaming (-1000)
Failed to stop RX streaming (-9999)
OOOOFailed to start RX streaming (-1000)
Failed to stop RX streaming (-9999)
Failed to start RX streaming (-1000)
Failed to stop RX streaming (-9999)
Failed to start RX streaming (-1000)
Failed to stop RX streaming (-9999)
Failed to start RX streaming (-1000)
Failed to stop RX streaming (-9999)
Failed to start RX streaming (-1000)
Failed to stop RX streaming (-9999)

I'll investigate further.

[Potomac]

@martinling : thanks for your commit, I will test it soon as possible.

[Potomac]

@martinling : Like the last time : can you provide a patch file compatible with the 2021-03 version of hackrf (last release version) ?
I don't master the git cherry-pick command.
Thanks.

[martinling]

@Potomac sure, here you go - this should apply to the 2021.03.01 release and includes both fixes from #1029:
bug-916-v2-rebase-2021.03.01.patch.txt

[Potomac]

@martinling : I made the test of your patch with Qqrx, and I still have the bug, no change.

Then I tried the test program "hackrf_loop" and I see the message "not started yet" after few seconds :

Finally started
Start
Not started yet
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Valid length: 262144
Finally started
Start
Not started yet
Not started yet
Not started yet
Not started yet
Not started yet
Not started yet
Not started yet
Not started yet
Not started yet
Not started yet
Not started yet
Not started yet

I tested on 2 different USB controllers (internal USB 2.0 ports of my motherboard, and a USB 3.0 controller PCI-E card) and the result is the same.
The motherboard I use is old : from 2007, a gigabyte GA-P35-DS3L, CPU is intel core 2 quad core Q9650 (3 Ghz) :
https://www.gigabyte.com/Motherboard/GA-P35-DS3L-rev-20#ov

OS is archlinux, a rolling release distro (like gentoo) where all libraries and linux kernel are at the latest release version (it may have an importance, as bugs can occur more easily on rolling release distros than classic distros like ubuntu, where libraries are not always in their latest versions).

[martinling]

Thanks for testing this @Potomac. In my case I'm no longer getting problems in gqrx, but I've seen that pattern happening in hackrf_loop, and @gozu42 noted it in #916 too.

I don't yet understand what's happening in the case where it just gets stuck at Not started yet without any errors being reported. I've poked at it with GDB and can't see what the problem is. The shutdown seems to have worked correctly, the restart seems to have successfully resubmitted the transfers, the transfer thread is looping and isn't hung, but the transfer callback never gets called.

I'm going to get the USB analyzer out next and check whether data is actually showing up on the wire in this case.

[Potomac]

@martinling : what is sure is that the previous hackrf driver version (2018) doesn't have the bug, hackrf_start_rx() and hackrf_stop_rx() run without problems with 2018 version on my hackRF.

So a solution would be to track the difference in source code between 2018 and 2021-03 versions, I tried with a git-bisect command but the results were inconclusive, the first bad commit found by git is :
1442014

but reverting 1442014 is not possible, as commits made after 1442014 need this faulty commit.

[martinling]

Unfortunately because this involves a race condition, the problem can come and go as a result of completely unrelated changes if they affect the timing of the code execution. We can't rely on testing and bisecting - we need to understand what the actual issue is to make sure it's solved properly.

1442014 made the stop functions kill the transfer thread, but that behaviour was removed later - the transfer thread now persists as long as the device is open, just as it did before that commit. So it's already effectively reverted anyway.

[Potomac]

Perhaps a log could help to understand better the race condition, if hackrf (or one of its dependencies) can be modified in order to generate debug log.

You can also try the tests with an old PC (or a raspberry pi 4), as the execution of the code will be slower on an old PC, it may trigger more easily the race condition.

[martinling]

I don't see a great way to attack this with more logging, because the symptom is that something doesn't happen. As far as I've been able to tell so far, everything is being started up correctly but no data comes back.

Setting an environment variable of LIBUSB_DEBUG=4 allows to see what's happening at the libusb level, but I see no further clues there. The attached log (edit: see next comment) is from a Windows machine on which I can reproduce this quite consistently. The interesting part starts at line 2248. The control transfer to set the transceiver mode to RX is submitted, and completed successfully. Then the four bulk transfers are submitted, also successfully.

And then we wait for the transfer callbacks, which just never come. We can see that the transfer thread hasn't hung, because every 500ms it times out and starts libusb_handle_events_timeout_completed again.

It looks like there may be a device-side issue involved, so I'll be looking at that side of things next.

[martinling]

The attached log is from Windows 10 running hackrf_loop against libhackrf with the #1029 fixes. The failure to start occurs at line 2248.
failure-to-start.txt

[martinling]

I have confirmed that there is a firmware-side bug which can cause RX stop/start to appear to succeed, but stall after the first 16KB of data. I've started a separate issue for this in #1042.

[Potomac]

Thanks Martinling for continuing the investigations,
you think it's a firmware bug, do you think that bug is present since the beginning of the project ? (from the first version of the firmware ? 2014 ?)

[martinling]

I'm certain that it's a firmware bug, because I have captured the USB traffic directly, and we can see that the host has correctly told the device to stop & start, but the device then stops sending data after the first 16KB. The host is continuing to ask for more data and the device doesn't send any more. That puts the blame clearly on the firmware.

At the moment I don't know how that bug happens on the firmware side, so I can't tell how far back it goes in the project history. And because it's intermittent and seems to be affected by timing, bisecting can't be relied on to find out. But it's interesting that you've had no problems with the 2018 version, so I'll look at the differences between there and 2021.

My next step will be to get a JTAG/SWD debugger hooked up to the HackRF's MCU and look at the firmware state when this happens. That should hopefully reveal something about the cause.

Let's keep further discussion about the firmware bug on #1042.

In terms of the original topic of this issue - the freeze in gqrx - my conclusion at the moment is that the same symptom can happen for three different causes. Two of those causes are host-side bugs that I believe to be fixed in #1029 (but that needs a little more work, plus review and merge). The third cause is the firmware bug which I'm now tracking in #1042.

[martinling]

@Potomac I believe I've now fixed the firmware bug as well. If you're able to test #1029 now I think you should have no more freezes in gqrx. But you will need to update the firmware as well as the host side. Let me know if you need any help.

[Potomac]

Can you provide a patch file compatible with 2021-03 hackrf version ?
The firmware file is built automatically when building hackrf ?

[martinling]

Sure, here you go:

bug-916-v3-rebase-2021.03.01.patch.txt

To build and flash the firmware you can follow the instructions here.

[Potomac]

@martinling : I tried your patch and I flashed the firmware : the bug is still here, I can easily trigger the bug with Gqrx,

I tried also the hackrf_loop program : I get also the bug with the message :

Start
Not started yet
Not started yet

Perhaps you should test with a configuration similar to mine :

• old PC (old CPU intel core2 quad Q9650 3 Ghz)
• a rolling release distro like archlinux, in order to be sure to use the last release version of libraries like libusb, and the last version of linux kernel

If I test with the appimage version of Gqrx (2.14.5) then there is no bug.

[gozu42]

can you please post output from hackrf_info on your current configuration?
and perhaps doublecheck that both test program and gqrx are using the right libhackrf?
something like "ldd hackrf_loop | grep hack" and "ldd which gqrx | grep hack".

i was able to repro the problem on both test program and gqrx before, but with the combination of the host+fw fixes the hangs are now gone here, on both high powered PC and low-powered raspi.

[Potomac]

Well I think I made a mistake about the firmware file, I think I still use the 2021.3 version of firmware,

I didn't manage to build the new firmware file, the compilation fails, can you send me a compiled version of the new firmware file ?

[gozu42]

can you send me a compiled version of the new firmware file

hackrf-fw-7057235a.zip

[Potomac]

@martinling @gozu42 : I finally manage to flash the firmware with the correct version sent by gozu42,

the bug seems gone when I use hackrf_loop program,
and also no bug when I use Gqrx,

Thank you very much Martin for this excellent job.

The output of hackrf_info :

$ hackrf_info 
hackrf_info version: 2021.03.1
libhackrf version: 2021.03.1 (0.6)
Found HackRF
Index: 0
Serial number: 0000000000000000048866dc377e48c3
Board ID Number: 2 (HackRF One)
Firmware Version: git-7057235a (API:1.05)
Part ID Number: 0xa000cb3c 0x00464753
Operacake found, address: 0xff
Operacake found, address: 0xff
Operacake found, address: 0xff
Operacake found, address: 0xff
Operacake found, address: 0xff
Operacake found, address: 0xff
Operacake found, address: 0xff
Operacake found, address: 0xff

[martinling]

That's fantastic news @Potomac, looks like we've finally solved this one. It's been quite an adventure!

[gozu42]

the reported version there for the lib might actualy be an artifact of the take-release-and-manually-patch process.

my flow for the build was ...

• going to the PR linked Fixes for concurrency bugs in start/stop operations #1029
• at the top, find the "x want to merge to y from z", the z part https://github.com/martinling/hackrf/tree/bug-916
• git clone -b bug-916 https://github.com/martinling/hackrf hackrf-martling
• then build as usual

the result is that all versions in hackrf_info output directly give the git commit used:

hackrf_info version: git-7057235a
libhackrf version: git-7057235a (0.6)
Firmware Version: git-7057235a (API:1.05)

[Potomac]

It would be interesting to have a nightly build channel for hackrf (like firefox), in order to download and test a beta/alpha version of hackRF binary (including the firmware).

Thanks again Martinling for the resolution of the bug, I hope these commits will be included soon in the master branch of hackrf.

[martinling]

That's a good idea but I have some worries about it. We would need to provide quite a lot of different binaries - we have three supported boards for the firmware, and several supported operating systems for the host tools. Some users will always download the wrong thing without understanding, and that can create a lot of support headaches.

A simple change we could make is to retain the build products of our existing Github Actions builds, which can be done. We run these builds every time commits are pushed to master or a PR, so if we kept the results you could just look at the latest CI run for any branch, and download prebuilt binaries for libhackrf, the host tools and the firmware. But it would depend on the user knowing how to correctly install these in the right places I expect.

I'll discuss it with the team.

Thanks also to you @Potomac for all your work in reporting this problem, experimenting with things, and testing fixes over the last few months. I don't think we would have tracked down all the different bugs involved without your persistent efforts.