[RFC] Python API

[mbr0wn]

I'd like to use this issue to collect feedback on the Python API. It is available here: https://github.com/EttusResearch/uhd/tree/python-api

[awelkie]

First of all, I love having a python API and I can't wait until it's shipped by default.

One suggestion: Put the documentation in a docstring below the method/class instead of in comments above the method/class. That way, the documentation will be available when calling help() on an object or when using ipython's ? operator. Also see some standardized docstring formats here.

Also, this API isn't sufficient for me since I need to transmit and receive at specific times, but I assume that mirroring more of the C++ API is still in progress.

I also really appreciate having python 3 support. If supporting both python 2 and python 3 becomes too burdensome, I would rather have python 3 support and wouldn't mind the lack of python 2.

[mbr0wn]

@awelkie Thanks for the feedback. As for documentation, I'd like even more improvements (such as pulling all the docs from Doxygen, maybe similar to how GNU Radio does it), so that'll definitely get some attention (at some point in the future... ahem).

I'd be curious if more people feel like you do about 2 + 3. We fully intend to support both, but I would expect the majority of users to prefer 2 over 3 (because they're locked to some distro or whatnot), but I'd be happy to learn otherwise.

As for timed streaming, it was my impression this is already supported. Let me check that.

[storborg]

This is really cool!

I'd second Python 3 support being more important than Python 2. At the moment, I'm stuck in Python 2 land for most things related to UHD because of GNU Radio's lack of support for Python 3. As soon as that lands, I'm looking forward to switching to Python 3 exclusively and never look back. I think the same is true for most people that I work with. Supporting both is great, though.

It would also be cool if there was support for timed commands (beyond timed streaming), or even crafting / decoding CHDR packets directly.

[mbr0wn]

@bhilburn Do you have thoughts/opinions on the Py2k vs. Py3k issue? Note that it is academic at this point. It seems like we can support both.

[awelkie]

Some more feedback:

1. I think the duration argument to send_waveform is unnecessary. All it does is truncate my signal, and I'd prefer to do that outside of this method. I end up just calculating it based on the length and the rate anyways:

usrp.send_waveform(signal, len(signal) / rate, freq, rate, gain=10)

2. send_waveform implicitly assumes that the data is numpy.complex64. If instead it's numpy.complex256, then the actual signal transmitted over-the-air is incorrect. You need to either change the datatype of the signal or change the stream arguments based on the datatype.

[mbr0wn]

@awelkie reports problems with Py3k in #111.

[mbr0wn]

@awelkie Thanks for the feedback -- that's exactly what I was hoping for with this thread. Some comments:

I think the duration argument to send_waveform is unnecessary. All it does is truncate my signal, and I'd prefer to do that outside of this method. I end up just calculating it based on the length and the rate anyways:

I fully agree with that. This change will go into future versions of the API.

send_waveform implicitly assumes that the data is numpy.complex64. If instead it's numpy.complex256, then the actual signal transmitted over-the-air is incorrect. You need to either change the datatype of the signal or change the stream arguments based on the datatype.

Under the hood, we directly interpret pointers and thus assume complex64. We won't add support for complex256 (I didn't even know this was a valid type until you posted this). In send_waveform(), we could add a check though.

[awelkie]

It looks like TimeSpec objects don't support addition:

In [19]: uhd.types.TimeSpec(2) + uhd.types.TimeSpec(2)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-19-a4c3bd89743a> in <module>()
----> 1 uhd.types.TimeSpec(2) + uhd.types.TimeSpec(2)

TypeError: unsupported operand type(s) for +: 'TimeSpec' and 'TimeSpec'

It's still possible to add two TimeSpec objects by converting to seconds by the get_real_secs method, so it's not a big deal, but it would be nice to mirror the C++ API by being able to directly add two TimeSpec objects.

[awelkie]

I don't think it's possible to specify the timeout in RXStreamer.recv or TXStreamer.send, which makes receiving large buffers very difficult. Also, there's no way of getting the asynchronous messages from a TXStreamer, so I'm not able to tell when a transmission is actually finished.

[mbr0wn]

@awelkie It is our intention to mirror the C++ API as much as possible, so we'll look into the addition feature.

[adray000]

Hello.How to interrupt the transmission of a signal on a button and how to generate a constant and not a specified amount of time ??

[mbr0wn]

@adray000 That's a general programming question; I recommend asking on usrp-users.

[adray000]

Tell me how the get_rx_stream function works and what arguments are passed to it. I do not want to open the session every time.

[dkozel]

@adray000 Please ask your question on the mailing list.
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

[oogieoogieful]

I am happy to begin using this package. Currently my RF testing team uses python 2.7 but we are in the process of moving to Python3.

Also, I would like to help with improving the documentation.

[mbr0wn]

@oogieoogieful Thanks, we'll appreciate any input. We need to figure out how to auto-improve documentation, but if you have specific comments, that'd be great to hear.

[oogieoogieful]

I would prefer to use pip to install the python-api uhd package.

[mbr0wn]

Using pip is on our radar, but it's very far out. For now, we've just made sure that Python packages show up in pip after you've installed it, so that other modules can list uhd as a dependency. Since the Python package depends on libuhd, which would probably not be a good candidate to make available via pip, one challenge that we'll face is ripping the UHD-Python code out of the repository for packaging on PyPI.

[noc0lour]

If you switch to building with setup.py and build_ext instead of CMake this could enable buliding the python module during the pip install.
This would require users to have Boost.Python and libuhd present (in LDPATH) and a working compiler and stuff. (And producing meaningful errors if this is not the case) That's how other python projects like numpy and Twisted are installed through pip. Ripping out the glue code to a separate project might be unavoidable though (since setup.py might has to be at the root of the project).

[stmharry]

Would it be possible to use threading or multiprocessing with module uhd to achieve parallel transmission and reception?
I want to accomplish this because we are sweeping through a list of frequencies and we have to coordinate TX frequency with RX frequency (not necessarily synchronized, but has to be the same across a few seconds), and however using Python threading would result in a GIL lock leading to a different behavior from that of uhd/host/examples/txrx_loopback_to_file.cpp. On the other hand, using multiprocessing currently leads to a behavior of seemingly sabotaging non-copyable resources such as multi_usrp, and ultimately fails to TX/RX.
Hence the question again, is it possible to achieve something similar to txrx_loopback_to_file.cpp but in a fully parallel, multi-CPU fashion?
I would be thankful and happy if developers can offer some insight into the underlying C driver and how we should use Python on top of it. :)

[mbr0wn]

That's a really interesting question, and would require some more research. I can see how mirroring the C++ approach in Python would fail (for the reasons you mention).
Some random thoughts:

• We don't really need a multi_usrp in both the TX and RX threads; all they need is a streamer. Maybe we can move that to a different process, although that would still require moving a bunch of shared ptrs (under the hood) into a different process, which is probably not possible.
• Maybe mirroring the C++ approach is not the right thing. We can't move multi_usrps across process boundaries, but we can share numpy arrays. Maybe we need a central thread to dispatch send() and recv() calls, and then process elsewhere.

...none of this is an answer to your question, but it's also a difficult one. The answer may also be that Python is not the right thing for this kind of problem (hopefully not).

[noc0lour]

From what I know using the threading module in Python could actually do the right thing. If I'm not wrong the tx_streamer and rx_streamer already use C++ threads to send()/recv(). And the GIL is only an issue if you try to do things in parrallel in Python land. Since the recv()/send() call are blocking in Python until they have the data available and the GIL doesn't stop the C++ threads you should be fine. During the time the Python interpreter waits for I/O to happen it is running threads which are not waiting.

If you then try to do a lot of postprocessing in Python this approach probably won't work since then you would run into issues with the GIL jumping around in different Python threads (and they don't block waiting on I/O).

This is solely based on what I've read up/learned about the GIL and might not apply in this case because of special circumstances.

[guruofquality]

Hey @mbr0wn, long time lurker here. For soapysdr we use the -threads option with swig to generate thread state save/release for every call. This is automatically part of the generated module source. But if you are manually writing python modules in C, you should add PyEval_SaveThread() and PyEval_RestoreThread() to every API call, especially the blocking ones with long timeouts.

This has been absolutely useful for some transmit and receive use cases. But its not just a matter of streaming performance; if you happen to be running a python-based GUI you must save the thread state or the GUI will become very laggy and stall while blocking for samples.

So python is never going to give you the best performance, but you basically have to address this, or it will cause strange problems down the road when someone's app behave kind of funny -- they might not realize the GIL is at play or how to fix it readily.

[noc0lour]

@guruofquality, good point, basically you have to tell Python that you want to release the GIL during C++ computation. I found this: https://stackoverflow.com/a/8011153 which is a nice example of how to use the release mechanism.

[mbr0wn]

Good points. As @noc0lour is hinting at, we use Boost.Python and don't write any C code for Python by hand, but that snippet is useful. I'm still very hesitant to do any of this until we have good test cases set up and good performance metrics to optimize for.

[arthurscholz]

Just did a build on Ubuntu 17.10 with Python3.6.

Only major issue was I had to create a simlink from libboost_python3.so -> libboost_python-py36.so in /usr/lib/x86_64-linux-gnu/ in order for CMake to find the the Boost Python libraries.

It also seems like the initial build step still depends on Python 2.7 - I had to have python-setuptools installed before make would finish without errors.

The default python install directory was /usr/local/lib/python3 which isn't in the default Python 3.6 path. I moved the uhd directory over to /usr/local/lib/python3.6 and everything worked great.

Looking forward to more Python support!

[applebull]

Could anyone provide a complete guidance of installation of UHD Python API on Ubuntu?

I cloned the python api repo, and followed this installation of UHD from source

Then I installed Boost.Python followed here, with these commands

cd path/to/boost_1_66_0
./bootstrap.sh --help
./bootstrap.sh --prefix=/usr/local/lib/boost --with-libraries=python 

However, when I run 'import uhd' in python, it says 'no module named uhd'. The OS is Ubuntu running on a Mac Mini.
What could go wrong? Thanks!

[brentstapleton]

Hi @applebull,
There a number of things that could be going wrong; we're going to need more information in order to know what the issue is. However, this Github issue was created to track bugs and such, more so than providing direct customer support. Could you send an email to our mailing list usrp-users@lists.ettus.com?
In the mean time, some common issues include:

• Did you checkout the python-api branch of UHD? (Python is not supported on our master branch)
• Was your UHD build successful? If not, what error messages did you see?
• Is your UHD install directory on your PYTHONPATH?

[jaykuhn]

Hi,

I am having trouble compiling uhd-python-api for python. It seems to be related to boost versions.

1. Followed instructions at https://github.com/EttusResearch/uhd/tree/python-api
2. Build instructions at http://files.ettus.com/manual/page_build_guide.html
3. It specified minimum boost version of 1.53. So I installed latest 1.66.
4. This had Visual C error because "native" had been depreciated and should be native_handle.
5. I backed up to the minimum 1.53.
6. This produced a new error about not finding and include file. I'm sorry I can not remember the exact file and I am off trying other boost version 1.58 because I found a mention of that.

I'm hopping someone can give me a boost version recommendation before I have to continue searching.

Thanks,

Jay

I found the same problem with 1.58.

uhd-python-api\host\lib\transport\tcp_zero_copy.cpp(25): fatal error C1083: Cannot open include file: 'boost/make_shaBOOST_CONFIG_SUPPRESS_OUTDATED_MESSAGnatred.hpp': No such file or directory

[mbr0wn]

@jaykuhn 1.58 is what I'm using, so I'm pretty confident it'll work. We need rebase python-api to latest master; the native issue has been fixed there. (@brentstapleton FYI).

[jaykuhn]

Thanks,
I think I am having problems with configuration. I clean git checkout of uhd from from the python branch. I download boost_1_58_0 (There is also a 32b, should I have used?)

I get the following error.
Error LNK1104 cannot open file 'libboost_regex-vc140-mt-1_58.lib'

The file that is there is 120, not 140.

Maybe you could give me your exact configuration:

1. Which Git branch
2. Which boost
3. What variables did you set to what in Cmake (Can you show me full path since I am not sure if they just stop at root or have to point to low ever level directories.

Jay

[mbr0wn]

@jaykuhn Are you on Windows?

[jaykuhn]

Yes. Also, which vs are you? I've been community 17 which was giving me some problems building boost. I am now building 1_58_0 with vs 2012. I will see what that gives me. By the way: 1_58_0 or 1_58_0_32b?

Jay

[jaykuhn]

VS 2012 Did much better but still failed. First failure was usb_dummy_impl.obj

usb_dummy_impl.obj : error LNK2019: unresolved external symbol "public: virtual __thiscall uhd::transport::usb_zero_copy::~usb_zero_copy(void)" (??1usb_zero_copy@transport@uhd@@UAE@XZ) referenced in function "public: void __thiscall uhd::transport::usb_zero_copy::`vbase destructor'(void)" (??_Dusb_zero_copy@transport@uhd@@QAEXXZ)
2>C:\Users\Jay\Ettus\git_python_api\uhd\build\lib\Debug\uhd.dll : fatal error LNK1120: 2 unresolved externals

Next failure was linking to unit tests:

10>LINK : fatal error LNK1104: cannot open file 'boost_unit_test_framework-vc110-mt-gd-1_58.lib'

Strange..
Jay

[jaykuhn]

Martin,

Are you able to support me on Windows? Is this the right place to do it? Are you with Ettus or just a community supporter?

Jay

[mbr0wn]

@jaykuhn Yeah I'm with Ettus. The best place would be to take this to the mailing list. We're short on people with Windows experience.

[jaykuhn]

What/Where is the mailing list? From: Martin Braun <notifications@github.com> Sent: Thursday, April 26, 2018 2:49 PM To: EttusResearch/uhd <uhd@noreply.github.com> Cc: Jay Kuhn <jkuhn@impinj.com>; Mention <mention@noreply.github.com> Subject: Re: [EttusResearch/uhd] [RFC] Python API (#105) @jaykuhn<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjaykuhn&data=02%7C01%7Cjkuhn%40impinj.com%7C20bf74c4638c4538a37f08d5abbf7f71%7C6de70f0f73574529a415d8cbb7e93e5e%7C0%7C0%7C636603761317083268&sdata=wMuBi1uaKDiV%2FWKx5D1lE4xfkk5fLgTLAC6YOAnDFPU%3D&reserved=0> Yeah I'm with Ettus. The best place would be to take this to the mailing list. We're short on people with Windows experience. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEttusResearch%2Fuhd%2Fissues%2F105%23issuecomment-384799655&data=02%7C01%7Cjkuhn%40impinj.com%7C20bf74c4638c4538a37f08d5abbf7f71%7C6de70f0f73574529a415d8cbb7e93e5e%7C0%7C0%7C636603761317083268&sdata=f%2FrSgMVBg7yppHs8rjE387RnLFhuG25k775DB70RdL4%3D&reserved=0>, or mute the thread<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAk4Dq0BDQTRl3I-e70OXaFKBKcZhSmNAks5tskDCgaJpZM4OIm-Y&data=02%7C01%7Cjkuhn%40impinj.com%7C20bf74c4638c4538a37f08d5abbf7f71%7C6de70f0f73574529a415d8cbb7e93e5e%7C0%7C0%7C636603761317083268&sdata=dcXq0T8HxMJBt0B204Vbikr0avK5tWX9IJtIGpy%2F0CQ%3D&reserved=0>.

[dkozel]

@jaykuhn The USRP mailing list is located here: http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

I'll catch you on the mailing list and try helping with the linking errors. Have you seen the app note on building on windows?
https://kb.ettus.com/Building_and_Installing_the_USRP_Open_Source_Toolchain_(UHD_and_GNU_Radio)_on_Windows

[jaykuhn]

I'm waiting to get authorized on the listinfo. In the meantime..
Yes, I have now followed those instructions. It built just fine. But those are not the python instructions, right?

I then used those instructions on the git git_python_api branch.

1. In CMake BOOST_PYTHON_FOUND is blank and i don't get PYTHON API enabled. I cannot tells what is missing. Just for kicks I set BOOST_PYTHON_FOUND TRUE in Cmake and Cmake runs fine.
2. I run Visual Studio. Virst error is what I think is a c++ version error:

C:\Users\Jay\Ettus\git_python_api\uhd\host\lib\transport\libusb1_base.cpp(394):
error C2600: 'uhd::transport::usb_device_handle::~usb_device_handle' :
cannot define a compiler-generated special member function
(must be declared in the class first)

That is just the first one. I get 100's of similar error... usb_handle ...

I assume I am using wrong version of VS for the version of C++.

Jay

[jaykuhn]

Oh yes, when I did try a latter version of VS I needed later version of boost. When I did that I found that UHD code was calling socket.native but the boost code said this was depreciated and the correct code was was socket.native_handle..

But somewhere I've got all my versioning messed up...

[jaykuhn]

Martin,

I still have no acess to mailing list. I sent the following e-mail but it is rejected.

[jaykuhn]

The last thing that happened when I applied is it said I would receive email but I didn't.

Hi,

I subscribed to mailing list Friday morning and still have not been allowed access. I have pressing need for support. We purchased three boxes that arrived 10 days ago and we are still unable to program them on windows with Python. I have been having some discussion on the github threads with Martin but he suggests the mailist for better support. I have easily built the standard interface on Windows but I get syntax errors when I use the git branch. Typical ones are socket.native was depreciated in boost and now wants native_handle. Other error seem to be around virual methods which is again a syntax that I believe has changed in C++ standard recently.

Jay

[mbr0wn]

@jaykuhn If you have a specific issue, can you please open a new thread on github. This one is supposed to be a general complaint bucket for the Python API. The mailing list does not have any kind of access control, all you need is to sign up and click a confirmation link. Please check your spam messages.

[mbr0wn]

Python API is now merged -- closing this. Please submit new issues if something comes up with the Python API. Thanks all for all your valuable comments!