OSX / Raspberry Pi issues

[cjcliffe]

Just trying out SoapyRTLSDR and SoapyRemote on a Raspberry Pi here, few items to note:

• -std-c++11 in CMakeLists.txt isn't valid for GCC 4.6.3 on current Raspbian, -std-c++0x seems to work everywhere
• SoapySDRServer --bind doesn't seem to work alone; just outputs the following with no reason after the 'FAIL: ':

pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

upid://raspberrypi?pid=5198&hid=8323329
Launching the server... tcp://[::]:55132
Server socket bind FAIL: 

Running it with a specific bind parameter seems to work:

pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind="0.0.0.0:55132"
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

upid://raspberrypi?pid=5202&hid=8323329
Launching the server... tcp://0.0.0.0:55132
Server bound to 0.0.0.0:55132
Press Ctrl+C to stop the server

[guruofquality]

I'm aware of the bind issue, probably its on platforms which are missing ipv6 support in the kernel. You can also use --bind="0.0.0.0" to get the default port. I think it needs to test and fallback to ipv4 automatically... seems more graceful that way.

Regarding the flag, I think it need a test and fallback as well. I dont want a newer gcc to bomb out either just in case they finally take -std-c++0x away.

[cjcliffe]

Aye, it looks IPV6 related for sure; not a huge issue in that regard. Having a fallback for c++0x would be a good idea but I'm guessing it'll be around for awhile to support the many projects that haven't updated yet.

On another note might as well keep this issue going with the following:

pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind="0.0.0.0:55132"
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

upid://raspberrypi?pid=5241&hid=8323329
Launching the server... tcp://0.0.0.0:55132
Server bound to 0.0.0.0:55132
Press Ctrl+C to stop the server
SoapyServerListener::accept(192.168.1.101:63177)
Segmentation fault

Seems to crash right away once I connect from desktop; I'll do some more debugging on the Pi to figure out what's going on.

[guruofquality]

Might be the same problem, not sure since he said the module wasn't all filled out: https://groups.google.com/forum/#!topic/pothos-users/w_ABDy-xOU0

But arm to/from amd64 should be OK in principal. So I guess a backtrace will be very telling.

[guruofquality]

Hopefully the first issues have been fixed now in master, fixed flags, strerror, and ipv6 bind

[guruofquality]

This may be the fix for the crash: 2fbb9ac

[cjcliffe]

Excellent, was just about to do some more testing on the Pi

[cjcliffe]

Just a typo found so far:

        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std-c++0x")

should be

        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x")

[guruofquality]

doh! and i copied that around quite a bit -- fixed now

[cjcliffe]

Another quick typo, looks like you didn't mean to return here, compiles when removed:

SoapyRemote/common/SoapyRPCSocket.cpp:214:12: error: cannot initialize return object of type 'const char *' with an rvalue of type 'int'
    return strerror_r(errno, buff, sizeof(buff));
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

(when compiled on OSX)

[guruofquality]

Unfortunately there are two versions of strerror_r, and one of them returns a char pointer, which I have to return, the other doesn't. The ifdefs were supposed to be the way to figure it out: http://linux.die.net/man/3/strerror_r

It looks like (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE should become ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE) || __APPLE__

Thats a mess :-)

[cjcliffe]

Seems to be working properly with SoapySDRUtil; probes, finds, makes remotely; but when I actually try to start a stream:

cjmacbook:build ccliffe$ SoapySDRServer --bind
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

upid://cjmacbook.local?pid=23325&hid=0
Launching the server... tcp://[::]:55132
Server bound to [::]:55132
Press Ctrl+C to stop the server
SoapyServerListener::accept([::ffff:127.0.0.1]:53537)
SoapyServerListener::accept([::ffff:127.0.0.1]:53538)
Found Rafael Micro R820T tuner
SoapyServerListener::accept([::ffff:127.0.0.1]:53539)
SoapyServerListener::close()
SoapyServerListener::close()
SoapyServerListener::accept([::ffff:127.0.0.1]:53540)
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Floating point exception: 8

This isn't specific to the Pi; that's output from my OSX console connecting to localhost.

On the CubicSDR side:

[DEBUG] Found RTL-SDR Device using device index parameter 'rtl' = 0
[DEBUG] RTL-SDR opening device 0
[DEBUG] RTL-SDR Using buffer length 16384
[DEBUG] RTL-SDR Using 6 buffers
[DEBUG] RTL-SDR direct sampling mode 0
[DEBUG] RTL-SDR I/Q swap: No
[DEBUG] RTL-SDR offset_tune mode: No
[DEBUG] RTL-SDR PPM: 0
[DEBUG] RTL-SDR Tuner type: Rafael Micro R820T
[DEBUG] Setting sample rate: 2400000
[DEBUG] Setting center freq: 106072873
[DEBUG] Setting RTL-SDR AGC: Automatic
[INFO] SoapyRemote::setupRxStream(remoteFormat=CF32, localFormat=CF32, scaleFactor=32768, mtu=1500, window=16384)
[INFO] Client side stream bound to 127.0.0.1:60019
[INFO] Client side status bound to 127.0.0.1:60019
[DEBUG] Generating RTL-SDR lookup tables
[INFO] Using format CF32.
[INFO] Server side stream bound to [::ffff:127.0.0.1]:63781
[INFO] Server side stream connected to [::ffff:127.0.0.1]:60019
[INFO] Server side status connected to [::ffff:127.0.0.1]:63736

Connect seems fine, but as soon as I call:

    SoapySDR::Stream *stream = device->setupStream(SOAPY_SDR_RX,"CF32");

Immediately throws a runtime error:

�[1m�[31m[ERROR] SoapyLogAcceptor::handlerLoop() �[0m
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: SoapyRPCUnpacker::recv(header) FAIL: Undefined error: 0

Will do some debugging to try and narrow it down.

[cjcliffe]

Also ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE) || __APPLE__ does the trick on OSX for the strerror_r issue.

[cjcliffe]

gdb reveals:

Found Rafael Micro R820T tuner
[R82XX] PLL not locked!

Program received signal EXC_ARITHMETIC, Arithmetic exception.
[Switching to process 23811 thread 0xe07]
0x0000000100030790 in SoapyStreamEndpoint::SoapyStreamEndpoint (this=0x1002012b0, streamSock=@0x100200214, statusSock=@0x100200218, isRecv=false, numChans=0, elemSize=8, mtu=1500, window=16384) at /Users/ccliffe/Documents/GitHub/SoapyRemote/common/SoapyStreamEndpoint.cpp:41
41      _buffSize(((_xferSize-HEADER_SIZE)/numChans)/elemSize),

looks like numChans is causing divide-by-zero?

[guruofquality]

I think i see...

1. The error is forwarded over the rpc and then not caught, the log acceptor should at least catch here
2. The channels vector is empty, which we support, but its also used to set numChan, so that needs to bumped to a minimum of 1

[guruofquality]

Well, i pushed some fixes, I hope that does the trick

[cjcliffe]

Just pulled the update; seems to be working now -- unfortunately the number of elements returned per call is a consistent 178; far lower than the 40448 I am requesting which causes problems on the CubicSDR side -- should I be doing some additional buffering on readStream input when it's a remote connection or is there a buffer missing in SoapyRemote module on the client side?

[DEBUG] Setting RTL-SDR AGC: Automatic
[INFO] SoapyRemote::setupRxStream(remoteFormat=CF32, localFormat=CF32, scaleFactor=32768, mtu=1500, window=16384)
[INFO] Client side stream bound to 192.168.1.101:55699
[INFO] Client side status bound to 192.168.1.101:55699
[DEBUG] Generating RTL-SDR lookup tables
Loaded font 'Bitstream Vera Sans Mono' from 'vera_sans_mono12.fnt', parsed 167 characters.
[INFO] Using format CF32.
[INFO] Server side stream bound to 192.168.1.107:38872
[INFO] Server side stream connected to 192.168.1.101:55699
[INFO] Server side status connected to 192.168.1.101:61467
�[1m�[33m[WARNING] StreamEndpoint resize socket buffer: set 44040192 bytes, got 163840 bytes�[0m
[INFO] Configured sender endpoint: dgram=1452 bytes, 178 elements @ 8 bytes, window=160 KiB
[INFO] Client side stream connected to 192.168.1.107:38872
[INFO] Configured receiver endpoint: dgram=1452 bytes, 178 elements @ 8 bytes, window=16 KiB
calculated optimal element count of 40448

[guruofquality]

So, you are seeing fragmentation for the ethernet MTU. The SoapyRemote client side read/writeStream doesnt attempt to loop and continue on the same buffer, it just returns after the first datagram. I can make the read/writeStream implement the loop, but its worth noting that not all modules have this loop and repeat until the buffer is exhausted (or timeout) behaviour. Although convenient, if you want to be robust against lazy implementations, you might need CubicSDR to have a loop anyway. Thoughts?

FYI, if you set the "remote:format=CS16" stream arg, then we will get shorts over the ethernet and get double the number of samples per packet. SoapyRemote can convert CS16 to CF32 on the client side. So it might also be worthwhile handling CS8 conversions and CS8 as an RTL output.

[cjcliffe]

That works; I just wasn't sure if readStream on the client side was handling that buffer loop or I should expect to -- I think it's perfectly reasonable to do this on the CubicSDR side and I will make the adjustments.

I'll add the CS8 format to SoapyRTLSDR too; that makes sense to have available for this.

[cjcliffe]

SoapySDRServer is working now on localhost with OSX and remotely on Raspberry Pi. CS8 is also now implemented for SoapyRTLSDR, but the remote driver doesn't seem to acknowledge the "remote:format=CS8" as I still see:

[INFO] SoapyRemote::setupRxStream(remoteFormat=CF32, localFormat=CF32, scaleFactor=32768, mtu=1500, window=16384)

I also seem to be dropping a lot of packets and the audio demodulation is stuttering regularly as a result -- even with localhost at 250KHz bandwidth it's skipping; Pi is much much worse but Pi is wired to the network at 100Mbit and I'm running on 802.11a Wifi so at least I'd expect some loss there.

[guruofquality]

So I was working on the CS8 as we speak. I just checked it in. Needs testing though.

There is something regarding the socket buffering on OSX that needs to be figured out. From my experience, increasing the SO_RCVBUF beyond a few kb causes a crash. The current window default for OSX is actually way smaller (window=16384 in the snippet above). For linux, I set this to 10s of megabytes.

Can you mess around with this? either pass in "remote:window=something_big" to the stream args or change the default in SoapyRemoteDefs.hpp I think the bigger buffer is critical, just didnt want to crash OSX as the default option.

[cjcliffe]

Hmm, I tried the CS16 remote format and it seems to be returning I/I or Q/Q instead of I/Q so it's mirrored on the spectrum.

CS8 seems to be working with RTLSDR once I fixed the conversion; but both CS16 and CS8 remote format seem to make the stuttering worse (constant).

I've taken the window size as high as 4MB so far to localhost and there doesn't seem to be any improvement.

[cjcliffe]

Hmm, actually I don't think the window is working:

[1m�[31m[ERROR] StreamEndpoint resize socket buffer FAIL: No buffer space available�[0m
[INFO] Configured sender endpoint: dgram=1452 bytes, 714 elements @ 2 bytes, window=4000 KiB

~2MB seems to be about the limit for me:

[INFO] Configured sender endpoint: dgram=1452 bytes, 178 elements @ 8 bytes, window=2000 KiB
[INFO] Client side stream connected to 127.0.0.1:59736
[INFO] Configured receiver endpoint: dgram=1452 bytes, 178 elements @ 8 bytes, window=2000 KiB
calculated optimal element count of 40448

No change in performance though; and CS16 is still funky and CS8 stutters like crazy.

[guruofquality]

I just found a ridiculous typo that explains the conversion issues... how does it look now?
(I tested cs16 w/ the bladerf, so I don't even)

[cjcliffe]

CS8 seems to be working slightly better and the stuttering is now on-par with CF32; CS16 still seems to be mirrored and duplicating I/I or Q/Q though.

[guruofquality]

More typos from when I added the CS8, works over here now...

Is there an OSX sysctl equivalent setting to get the buffer larger than 2M?

I'm curious about the role the flow control is playing in the stutters. We might need a different strategy when the buffer is small. This will disable waiting on flow control (ACK) packets.

diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
index c3bfa78..f26c320 100644
--- a/common/SoapyStreamEndpoint.cpp
+++ b/common/SoapyStreamEndpoint.cpp
@@ -252,6 +252,7 @@ void SoapyStreamEndpoint::releaseRecv(const size_t handle)
  **********************************************************************/
 bool SoapyStreamEndpoint::waitSend(const long timeoutUs)
 {
+    return true;
     //are we within the allowed number of sequences in flight?
     while (not _receiveInitial or
         uint32_t(_lastRecvSequence+_maxInFlightSeqs) < uint32_t(_lastSendSequence))

Just for some rough maths here, if there is a 2MB window, then roughly 1 million samples (at CS8) can be held in that window. At 1 Msps (sample rate), the remote endpoint has about 1 second before it will block on flow control. And it should see a flow control update every eight of a window. It seems like plenty of time and that 2MB should not be an issue for lower speeds.

[cjcliffe]

CS16 is fixed; seems a little bit smoother with the patch but still stuttering -- though it improves slightly when moving from CF32 to CS16 and from CS16 to CS8, but stutter seems to be of similar consistency whether it's running at 250KHz or 2.56Mhz -- the stuttering increases with rate (i.e. 2.048Mhz seems to have twice as much stuttering as 1.024Mhz)

Window seems to have a larger margin of improvement now, default window vs. 2MB window is much better; but still stuttering quickly such that it sounds "grainy"

[cjcliffe]

Doubt it's related; but I do get this warning several times from a fresh compile:

In file included from /Users/ccliffe/Documents/GitHub/SoapyRemote/common/SoapyURLUtils.cpp:4:
/Users/ccliffe/Documents/GitHub/SoapyRemote/build/common/SoapySocketDefs.hpp:71:13: warning: 'htonll' macro redefined [-Wmacro-redefined]
    #define htonll(x) __builtin_bswap64(x)
            ^
/usr/include/sys/_endian.h:141:9: note: previous definition is here
#define htonll(x)       __DARWIN_OSSwapInt64(x)
        ^

[guruofquality]

Oh, thats nice. I didn't get htonll on my platform. I fixed up the ifdefs, should be ok now.

[guruofquality]

Recap

To recap, you are streaming from the Pi to the OSX around 2 Msps. Increasing the socket buffering helps, increasing the samples per packet helps. But you are still seeing some kind of overflow or loss through the link.

Some thoughts

The OSX side is probably keeping up just fine pulling samples out of the socket. So its probably an issue of the Pi keeping up on the send() side. I guess we are seeing overflow, but the RTL doesn't have an indicator when this happens. Some things worth investigating here

• Is there a problem feeding the socket on the Pi at ~2 Msps?
  • Is it a intermittent issue vs an actual bottleneck?
• Also consider packet loss itself in the UDP layer
  • At least detecting and printing when this happens would be helpful: See handle sequence errors and drops for acquireRecv() #2
• If the Pi is keeping up, this could also be a overflow in the recv socket buffer, this would happen if the OS has overhead tracking in the socket buffer that the SoapyRemote didn't account for
  • or similarly, the Pi is keeping up, but its the one overflowing the socket buffer (SO_SNDBUF), I dont think this is the case because linux doubles this buffer just for overhead management, and ethernet stack should be able to drain this buffer faster than RTL @ 2 Msps can fill it
  • It may also be worthwhile to try a larger MTU over Ethernet, if the Pi and OSX support it: https://github.com/pothosware/SoapyRemote/wiki#remote-mtu

Replicating the issue

I don't have exactly the same hardware, but I think I will give RTL on a Novena to my linux desktop a shot and mess around with the window sizes and some of the ideas mentioned here to see what happens.

Diff for printing S when there is a packet drop or overflow in socket buffer -- packet loss in network layer

diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
index c3bfa78..5e3ab59 100644
--- a/common/SoapyStreamEndpoint.cpp
+++ b/common/SoapyStreamEndpoint.cpp
@@ -208,6 +208,10 @@ int SoapyStreamEndpoint::acquireRecv(size_t &handle, const void **buffs, int &fl
     const int numElemsOrErr = int(ntohl(header->elems));

     //TODO sequence out of order or skip failures
+    if (uint32_t(_lastRecvSequence+1) != uint32_t(ntohl(header->sequence)))
+    {
+        SoapySDR::log(SOAPY_SDR_SSI, "S");
+    }

     //update flow control
     _lastRecvSequence = ntohl(header->sequence);

Maybe we are overflowing on the OSX side and the socket buffer has unaccounted for overhead:

diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
index c3bfa78..1efbf46 100644
--- a/common/SoapyStreamEndpoint.cpp
+++ b/common/SoapyStreamEndpoint.cpp
@@ -100,6 +100,7 @@ SoapyStreamEndpoint::SoapyStreamEndpoint(
     {
         //calculate maximum in-flight sequences allowed
         _maxInFlightSeqs = actualWindow/mtu;
+        _maxInFlightSeqs /= 2;

         //calculate the flow control ACK conditions
         _triggerAckWindow = _maxInFlightSeqs/_numBuffs;

[cjcliffe]

Ok, I'll give these a go -- note that most of my testing since the arithmetic error fix has actually just been to localhost (running SoapySDRServer and CubicSDR on my macbook) since the Pi introduced it's own issues testing over Wifi.

The MTU on lo0 I've been able to push up to 65535; but no actual network device seems to let me go over 1500 on the OSX side. And pushing lo0 to 65535 doesn't seem to affect the 178 element limit connecting to localhost anyways.

Will get back to you soon with results.

[guruofquality]

I should add that for the mtu stuff, you need to set both "remote:mtu=4096" as well as ifconfig lo mtu 4096 -- for example.

[cjcliffe]

Ok, so current status:

First patch:
Printing 'S' only really reveals it showing an 'S' when I move the center frequency or do other things that actually interrupt the stream.

Second patch:
Overflow patch seems to have no effect.

But Removing all the patches and going back to master I have managed to get it reasonably stable using:

   streamArgs["remote:mtu"] = "8192";
   streamArgs["remote:format"] = "CS8";
   streamArgs["remote:window"] = "16384000";

remote:mtu at 8192 seems to be the maximum; going beyond this it just connects successfully and then no data is sent at all. It looks like remote:window might be capable of going higher though.

Though it complains in the log about "No buffer space available"; cranking the mtu and the window to these levels seems to actually work and changes the endpoint init values in the log.

I had been stopping when I hit the error previously since I assumed they weren't being applied but it appears they are; and they work:

[INFO] SoapyRemote::setupRxStream(remoteFormat=CS8, localFormat=CF32, scaleFactor=128, mtu=8192, window=16384000)
[INFO] Client side stream bound to 127.0.0.1:56215
[INFO] Client side status bound to 127.0.0.1:56215
[INFO] Using format CS8.
[INFO] Server side stream bound to [::ffff:127.0.0.1]:59978
[INFO] Server side stream connected to [::ffff:127.0.0.1]:56215
[INFO] Server side status connected to [::ffff:127.0.0.1]:50924
�[1m�[31m[ERROR] StreamEndpoint resize socket buffer FAIL: No buffer space available�[0m
[INFO] Configured sender endpoint: dgram=8144 bytes, 4060 elements @ 2 bytes, window=16000 KiB
[INFO] Client side stream connected to 127.0.0.1:59978
�[1m�[31m[ERROR] StreamEndpoint resize socket buffer FAIL: No buffer space available�[0m
[INFO] Configured receiver endpoint: dgram=8144 bytes, 4060 elements @ 2 bytes, window=16000 KiB
calculated optimal element count of 40448

Perhaps OSX just doesn't let you tweak the socket buffer manually and is handling buffering behind the scenes in a dynamic way?

At these settings I can also now use SoapyRemote with localhost at up to 3.2Msps somewhat stable -- which is strange since it stutters at 3.2M using just the SoapyRTLSDR module directly and has never really worked properly on my MacBook at that rate..

I do still seem to notice the odd glitch but they're several seconds apart at minimum and sound more like just a skipped sample instead of a constant grain like before. CF32 format is almost usable as well but has more frequent stuttering.

Edit:
Tried again with the Raspberry Pi, with the above streamArgs I can almost use 250khz over the hard-wired network but any higher it gets choppy. remote:mtu seems to be Ok with 65535 and I've had remote:window up to 32MB with the Pi remotely, neither settings improves much upon the 8192/16384000.

[guruofquality]

I gave my FM demod demo a shot, and the SoapyRTLSDR is very choppy and causing audio underflows compared to the RTL support from SoapyOsmo. I'm going to take a closer look, but I think there is an issue in SoapyRTLSDR readStream and buffer length somewhere. Just wanted to report this first...

[guruofquality]

I switched to the RTL async API on this branch, and my issue goes away: https://github.com/pothosware/SoapyRTLSDR/tree/async_stream

Just curious if it also fixes any of the stuttering mentioned here.

[cjcliffe]

Yeah, the async version works 100% locally with SoapyRemote and any stream options I throw at it now; I'm going to update the Pi and see how it goes.

I've been using the sync api for quite some time without issue so I'm surprised it's the apparent cause; perhaps because we were requesting buffer sizes that weren't in multiples of 512 or whatever the requirement is during SoapyRemote streaming?

[cjcliffe]

OK, nice work; Pi is awesome now!

I can use it stable at about 1Mhz at default settings; but with the stream args of:

   streamArgs["remote:mtu"] = "8192";
   streamArgs["remote:format"] = "CS8";
   streamArgs["remote:window"] = "16384000";

3.2Mhz now appears to be quite stable! And that's over Wifi :)

Edit: yeah this is much better than rtl_tcp 👍
Edit2: been running it stable for an 1/2 hour now on the Pi, don't recall hearing a single glitch so far

[guruofquality]

The sync api was always using bulk transfers of 16384*16. I did try smaller sizes like 16384 as well. Actually, I wasnt even using SoapyRemote when I mentioned this issue. So it really may be a sync api bug in rtl. I guess we can close this one out at least.