Test 11 (qa_plsync_cc) Failed

[razed11]

Hi,

I'm running Ubuntu 20 as a Parallels VM guest on MacOS Monterey 12.3.1 (Apple Silicon M1 Macbook Pro).

The build went pretty smoothly after I installed libsndfile-dev but CTest showed one test failure.

11/12 Test #11: qa_plsync_cc .....................***Failed    2.78 sec
      Start 12: qa_symbol_sync_cc
12/12 Test #12: qa_symbol_sync_cc ................   Passed    0.18 sec

92% tests passed, 1 tests failed out of 12

Total Test time (real) =   3.68 sec

The following tests FAILED:
	 11 - qa_plsync_cc (Failed)

[igorauad]

Hi @razed11 ,

Thanks for the question. Do you get the error consistently? Or do the tests pass if you try again?

As it is now, the PL Sync QA includes performance tests that are a bit optimistic and prone to errors. I plan to separate performance tests from basic unit tests in the future. However, for now, this is my easy way of detecting when some change makes performance worse (if tests executed on CI start failing consistently).

I believe the error you are seeing happens on the test_detect_noisy_plheader_closed_loop test case. This is the one that most frequently fails at the moment. It consists of sending two consecutive PLFRAMEs at 5 dB SNR and detecting both with the correct decoding of the PLSC information. This is a bit tricky because the performance normally improves after several frames are received, i.e., once the estimates are averaged out over more frames. Detecting the correct information based on only two frames is a relatively high expectation, but which I've been keeping within the tests to ensure it still works (it does most of the time) and does not regress.

So I would recommend running the tests again. If the test case fails occasionally, but not most of the time, then we are probably seeing the same performance.

Also, regarding running on a VM, I happen to be using an M1 too. I have been developing everything GR-related inside docker containers and maintaining these handy container images here: https://hub.docker.com/r/igorfreire/gnuradio-oot-dev. Perhaps that would be of interest to avoid using VMs. The gr-dvbs2rx Docker image is also based on this gnuradio-oot-dev image, see here: https://github.com/igorauad/gr-dvbs2rx/blob/master/Dockerfile.

[razed11]

Hi @igorauad,

Thanks for responding and for the work you put into this.

I ran it a handful of times when I first installed it and it failed each time but I just ran it several times and more often it passed. This is fine now that I understand your performance v. regression testing goals.

Yes! I would prefer not to use the VM. I saw mention of the Docker image but it didn't occur to me to use it for running the code.

I'm about to start testing this with a physical DVB-S2 transmitter and an Ettus USRP N210.

[igorauad]

Thanks, @razed11

I'm about to start testing this with a physical DVB-S2 transmitter and an Ettus USRP N210.

Nice. FYI, I have pushed some changes adding more USRP options for both Tx and Rx. I have been doing some tests with USRP devices lately and organized the changes I was using. The basic commands are presented in Example 6 and Example 7 of the docs, but there are more options that you can find on the help menu (repeated below).

Tx

USRP Options:
  --usrp-args USRP_ARGS
                        USRP device address arguments as a comma-delimited string with key=value pairs (default: None)
  --usrp-antenna USRP_ANTENNA
                        USRP antenna (default: TX/RX)
  --usrp-clock-source {internal,external,mimo,gpsdo}
                        USRP clock source (default: None)
  --usrp-gain USRP_GAIN
                        USRP Tx gain (default: 0)
  --usrp-otw-format {sc16,sc12,sc8}
                        USRP over-the-wire data format (default: sc16)
  --usrp-stream-args USRP_STREAM_ARGS
                        USRP Tx streaming arguments as a comma-delimited string with key=value pairs (default: None)
  --usrp-subdev USRP_SUBDEV
                        USRP subdevice specification (default: None)
  --usrp-time-source {external,mimo,gpsdo}
                        USRP time source (default: None)

Rx

USRP Options:
  --usrp-args USRP_ARGS
                        USRP device address arguments as a comma-delimited string with key=value pairs (default: None)
  --usrp-antenna USRP_ANTENNA
                        USRP antenna (default: TX/RX)
  --usrp-clock-source {internal,external,mimo,gpsdo}
                        USRP clock source (default: None)
  --usrp-gain USRP_GAIN
                        USRP Rx gain (default: 0)
  --usrp-lo-offset USRP_LO_OFFSET
                        USRP's tune request LO offset frequency in Hz (default: None)
  --usrp-otw-format {sc16,sc12,sc8}
                        USRP over-the-wire data format (default: sc16)
  --usrp-stream-args USRP_STREAM_ARGS
                        USRP Rx streaming arguments as a comma-delimited string with key=value pairs (default: None)
  --usrp-subdev USRP_SUBDEV
                        USRP subdevice specification (default: None)
  --usrp-time-source {external,mimo,gpsdo}
                        USRP time source (default: None)