(Question) drone_id_test.grc failed to output file

[alphafox02]

Very interesting work and write-up. I'd love to test against a Mavic 3, however, after installing w/ GR 3.8 and attempting to run w/ a B205mini I get the following when running the drone_id_test.

FFT SIZE: 1024, sample rate: 1.536e+07
Traceback (most recent call last):
  File "/home/dragon/dji_droneid/gnuradio/drone_id_demod_test.py", line 350, in <module>
    main()
  File "/home/dragon/dji_droneid/gnuradio/drone_id_demod_test.py", line 328, in main
    tb = top_block_cls()
  File "/home/dragon/dji_droneid/gnuradio/drone_id_demod_test.py", line 218, in __init__
    self.droneid_demodulation_0 = droneid.demodulation(samp_rate / decimation)
  File "/usr/local/lib/python3/dist-packages/droneid/droneid_swig.py", line 845, in make
    return _droneid_swig.demodulation_make(sample_rate)
RuntimeError: Failed to open output file

I suspect it's something to do with the note about 1024 FFT and maybe my use of the 205. Although the failed to open output file has me confused. I'm not super familiar with how the droneid_swig.py is generated besides what I quickly researched after reading the top of the file saying that it's automatically generated.

The correlation_test flow graph has no problems running.

[proto17]

Thanks for the ticket! The issue you're seeing is that I added debugging logic to the various modules that outputs files to /tmp/bursts. If that directory doesn't exist it'll throw an exception. An example of that is https://github.com/proto17/dji_droneid/blob/gr-droneid/gnuradio/gr-droneid/lib/demodulation_impl.cc#L88. You can either create that directory or comment out the following lines:

https://github.com/proto17/dji_droneid/blob/gr-droneid/gnuradio/gr-droneid/lib/demodulation_impl.cc#L88
https://github.com/proto17/dji_droneid/blob/gr-droneid/gnuradio/gr-droneid/lib/demodulation_impl.cc#L162

If you choose not to comment out the lines be aware that you might run out of disk space if you leave the graph running for hours and hours. The files are very small and what's getting written is just the burst itself for debugging purposes (I throw them into MATLAB and plot things)

Just to reiterate, this is a WIP and I might end up breaking things here and there. Feel free to post tickets like this to bring it to my attention :)

Oh, and the code has only been tested against the DJI Mini 2. I know that some drones only send 8 OFDM symbols while my drone sends 9. Mine sends one additional symbol at the beginning, but that symbol zeros out after the scrambler is removed, so it's not actually used. The logic was updated to not look at the first symbol for frequency offset correction.

[alphafox02]

Thank you, works great now. I'll close the ticket and try this out on as soon as I can. There's an aerial photographer person around here, I think they've got a Mini 2 as well as the Mavic 3. Have to see if they'll let me experiment again! This was an easy test on DragonOS as it already had everything needed to run.

[alphafox02]

Sorry one other question, based on the flow graph defaults for sample rate and my limited knowledge at the moment, is it safe to assume you need an SDR that’s at least capable of doing a sample rate of 30msps or greater?

[proto17]

DroneID is 10 MHz of occupied bandwidth, so in theory you could sample at ~ 10 MSPS (assuming you have a really good filter with sharp roll-off) and resample to 15.36 MSPS to do demod. I was using 30.72 as it gave me the ability to offset tune to get the DC spike out of the way, and made it easier to hunt for the signal (since it hops around).

You should be able to change samp_rate to 15.36e6 and then set decimation to 1. I haven't tried it recently, so not 100% sure everything will play nice.

[alphafox02]

Thank you. I’ll try it as is and also change to what you explained above and see if the hackrf is sufficient. I also have a lime/blade to try out as well.

[alphafox02]

I should be able to try this again tomorrow but while trying to capture drone ID I would occasionally have the flow graph exit out. I went as far away from anything on 2.4 as possible, checked and made sure nothing was in the spectrum and forced the controller to 5Ghz.

on the default freq of the flow graph I could see a spike and would occasionally get activity talking about burst in the bottom left gnuradio window along with activity in the QT graphs. Picture attached. I’m not certain If that’s what I should experience.

[proto17]

So there are going to be two main problems:

1. I don't think the GNU Radio OOT module was updated to ignore the first OFDM symbol (note that in your time domain display the first symbol is absent which happens for some drones). This means that the constellations are going to be hot garbage since the frequency offset correction logic is going to fail in a major way
2. The MATLAB/Octave and GNU Radio OOT module both have a flaw in that they cannot deal with large (> a few KHz) of frequency offset. The coarse estimate right now uses the cyclic prefix of one of the OFDM symbols (this is where MATLAB/Octave and the OOT module differ) which cannot detect larger frequency offsets. The other problem here is that the ZC sequence correlation will move around in time if there is a frequency offset. I remember stumbling across this when tinkering with detecting bursts at any frequency offset. The ZC will do a great job of correlating for and detecting a burst, but if the burst is way off in frequency (tens of KHz or higher) then there will not only be a degradation in the correlation score, but the spike will show up before or after the actual start time of the ZC sequence in the recording. This time offset causes a phase offset in the frequency domain which means that the constellation will not stay still (keeps rotating from sample to sample or FFT bin to bin).

I'm looking at a way to fix the second issue by autocorrelating for the true ZC sequence start. That logic will be in the MATLAB/Octave before the OOT module though as the OOT module is secondary for me at the moment.

I'll take a look at the first issue tonight and see if it's an easy fix. For drones that send 9 OFDM symbols, the first one is just a constant value that ends up as all zeros after removing the scrambler. So it's not all that useful anyway.

[proto17]

Quick correction: The OOT modules does do CFO estimation and correction based on the second (your first) OFDM symbol. Working on adding in some debugging to show extra information to help with troubleshooting.

I just have the DJI Mini 2 here to test with, so these kinds of issues will keep happening =\

[proto17]

Oh that's perfect! Thank you!

The signal you're seeing in the first part is likely just Bluetooth. They are usually pretty narrow and look like little cones.

Around 2:00 is where you start to see Drone ID bursts. I'm not sure why you're not getting a constellation plot that makes any kind of sense. Usually if there is just a freq offset you'll see a ring, not a blob.

As for the rampant crashing, well, poop. I haven't given this branch any love in a while due to trying to make the MATLAB/Octave work reliably. I'll try running for longer here to see if I can recreate that crashing.

[proto17]

Additionally, the signal does jump around between frequencies. It doesn't stay put for long which is why I made the GNU Radio graph in the first place. Taking really long collects just to get a frame or two wasn't working out well. I've been told that it might be moving between 2.4 and 5.8 GHz vs the usual stay put in one band that most systems seem to use. You only get to see a burst every ~ 640 milliseconds and I don't normally see more than ~ 15 bursts at a single frequency before it moves on.

[proto17]

One possible thing that's causing you to not see constellations: The amplitude of the constellation is too high. Try setting the X/Y max/min values to something like -10 and 10 instead of -2 and 2 that are set by default. Alternatively, enable auto scaling. I think that what you are seeing in the constellation plot is actually the FFT bins that are not carrying data. Those will all usually be fairly low amplitude which means they cluster together in the center of the constellation plot.

The graph is exploding regularly for me too :( Will be looking at what's causing the problem

[alphafox02]

I’ll take all this info and try again tomorrow evening. I’ll look into what else may have been transmitting and go even further away from stuff and/or not just disable but power off anything additional I have on me. Exciting stuff, not to mention the first time I’ve actually messed with a DJI.

[proto17]

The crashing issue you were seeing was due (at least in part) to walking off the edge of an array. There is a commit in the gr-droneid branch [1] that should fix it.

967ba42

No real need to get a completely RF quiet area. It's rather unlikely to get a false trigger. The real thing to stay away from is WiFi. Try not to use frequencies that are being used by your WiFi AP as those transmit wide bandwidths and very frequently.

[alphafox02]

Getting a fatal error on extractor_impl.cc of droneid/misc_utils.h No such file or directory after doing the pull. I’ll retrace my steps.

[proto17]

Man... I really should have just gone to bed. I goofed twice in a row with that fix. The local copy of the repo that I am using has some changes to the utilities that don't exist in the remote just yet. Gimmie a sec to fix that. Maybe now that I'm awake it won't all go pear shaped.

[proto17]

Give the new main a go and see if that helps. Apologies for the busted code!

[alphafox02]

Pulled main, switched to gr-droneid and did another pull. Builds and installs fine now. I’m probably not helping with the sleeping part haha! Thanks for the update. I’ll give it another go this evening.

[alphafox02]

Flow graph was so much more stable, in fact I only got it to close out by cranking the gain next to some WiFi networks. It took me a little to realize I was seeing (I think) the burst.

[proto17]

You're definitely seeing bursts. If you zoom out on the constellation plot (bottom plot) you'll likely see some constellations :)
The equalizer is disabled, so the constellation points are going to be rough and rotating.

[alphafox02]

I didn’t even realize I could zoom out or just didn’t think of that, but good to hear I’m looking at the right thing. The last record to file sink I did in the video came out to about 2.3GB so I’ll see if I can work with that and use your other tools. Hopefully I can borrow the drone again some time. Safe to assume this ticket can probably be closed. Although it’s been helpful to share info and learn.

[alphafox02]

To do anything with the smaller capture, is that where I'd need to start learning about Matlab/Octave? I compiled the turbo decoder and was reading the notes at the top of it, doesn't seem like I can just feed it a 2.3GB file and expect it to do something with it.

[proto17]

There is a very brief and vague explanation at https://github.com/proto17/dji_droneid/#how-to-process-samples. There is a script called processing_file.m that you edit the top of to have it read your file. You will need to change the sample rate, file name, file type (usually just 'single' if you're using floating point values), and any frequency offset in your recording. My code cannot deal with large offsets (greater than one carrier spacing - or 15 KHz), so you have to make sure your center frequency is good.

You don't need to process the full file if there were bursts. Check your /tmp/bursts directory after running the flow graph. That directory will contain one file per burst detected by the flow graph. This problem of processing giant files is why I created the graph and partially why it doesn't do full equalization and demod. I just needed it to pull bursts out instead of having to wait for minutes and accumulating GBs of samples. You can feed the individual bursts to the process_file.m script just as you can with the large file. If you want you could probably go one step further and merge all of the bursts into one file using cat and process them all at once.

[alphafox02]

Now I get what the tmp part is for. I wasn't connecting the dots. I went in and disabled that part of the code. I'll put it back, then use my file capture to run back through GNU Radio and get the bursts this time in /tmp, hopefully.

[alphafox02]

Last question since this ticket has grown to more then initially asked, ha! I ran the captured file back through the flow graph after rebuilding with comments in code removed. I see both channel and zc in the bursts directory, each about 8k. Opening zc in inspectrum looks like the attachment. I think I’m getting somewhere and that this is what I’d have to use within matlab?

edit: you actually already answered my question above, you mentioned what the tmp directory would contain.

[proto17]

Whoops, it looks like I commented out the saving of the bursts. Uncomment line 85 in time_sync_impl.cc (https://github.com/proto17/dji_droneid/blob/gr-droneid/gnuradio/gr-droneid/lib/time_sync_impl.cc#L85) to have each burst saved off.

The file you're looking at now is the golden reference for the ZC sequence. Though as this thread has been going, I'm wondering if my ZC sequence is wrong. I think it's supposed to be one sweep, not two O.o Might need to remove the call to fftshift.

Sorry about that!

[alphafox02]

Oh I gotcha, I’ll go do that. That way I’ll get more bursts I suppose.

[alphafox02]

Should it be pdu_element_count at the end of line 85 in time_sync_impl?

[alphafox02]

It would seem I’m close, I ended up having to process the entire IQ file as for some reason it wouldn’t detect individual bursts I generated by processing the file back through the flow graph. Maybe because of what I did above to get it to compile to save the multiple bursts. Here’s the result of processing the entire file. Images 1-9 all basically look exactly the same, image 10 is the one that looks messed up. I also noticed the /tmp/bits is empty.

[proto17]

Were you by chance tuning around during the recording? It's very strange to me that you found a good ZC sequence, but didn't have any symbols before, and the symbols after were rather garbage except for number 7.

The version of the MATLAB/Octave code in the GNU Radio branch is a little bit behind. You can comment out these lines in the process_file.m script https://github.com/proto17/dji_droneid/blob/gr-droneid/matlab/updated_scripts/process_file.m#L151-L153. Then run it again and you should see frames.

In the latest main that error was changed to a warning https://github.com/proto17/dji_droneid/blob/main/matlab/updated_scripts/process_file.m#L206-L209

Also, the MATLAB/Octave code in gr-droneid doesn't use normalized cross correlations, so the threshold for detecting a burst varies a lot. You might have better luck using the MATLAB/Octave from main to do the post-processing

[alphafox02]

I reran again as I walked away to bed on a much larger capture, woke up thirsty just now and couldn’t resist checking. I’m looking at frames printed to screen with a good looking png. I think we have success!

edit: I was using the gnuradio branch for its one piece, but main for all the new decoding stuff. However, I was tuning around in some of the captured. I’ll make better notes when I’m awake again.

[alphafox02]

Messing with the resulting frames and attempting to line up the info against an image on Twitter of how the drone ID breaks down. It seems like some parts of the beginning and end match up, still working on the middle.

[proto17]

The structure of the DroneID frame is slightly different on the "RF" (I don't have a better name) link vs the WiFi. The beginning and end are slightly different. The middle does match up if memory serves. See https://github.com/proto17/dji_droneid/blob/main/matlab/updated_scripts/transmit/create_frame_bytes.m for a full breakdown of the fields. It's not documented *at all* but the order of the fields can be seen at line 185

…

On Fri, May 6, 2022 at 9:41 AM alphafox02 ***@***.***> wrote: Messing with the resulting frames and attempting to line up the info against an image on Twitter of how the drone ID breaks down. It seems like some parts of the beginning and end match up, but I’m curious if the matlab/decoder script blanks out any portion of the dji packet. [image: 0A2FC076-8313-4AC4-9AF8-1D114182B082] <https://user-images.githubusercontent.com/44436101/167144074-0efb42ce-7a87-4f73-8c55-1f873b37ecea.jpeg> — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABALJH3MTC5PIGKJOF3VPGDVIUOOXANCNFSM5UDYNUDQ> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[alphafox02]

Thank you for the help along the way! I think this ticket is filled with valuable info. At this point everything seems to be working perfectly. I’ll go ahead and close and continue to test and refine on my end. Super awesome, thanks sharing!

[alphafox02]

I ran the entire IQ file again (same one) through the octave script and the frames were exactly like I got back 4 days or so ago, except, after the last number in my previous attempt there was lots of zeros and then another string of numbers this time. Basically the frame was longer, but everything up till the ten or so zeros was exactly the same as before.

I'm just wondering could this have been do to code changes in main and the decoding process? I'm going to attempt again and see if the frame is still the longer length. I failed to keep track of my git pulls, so it's possible I was running older code in main last week. Just thought this was odd and wondered if you've ran into this on your end.

edit: The frame the first time around was roughly 182 characters long

[proto17]

There was an update to remove_turbo.cc that started printing out all of the received bytes, and not just the DroneID related bytes. As I understand it there are a bunch of random bytes from memory stuffed in with the DroneID payload. These bytes will be constant until the drone is power cycled. Is that what you're seeing? It should be 100 or so bytes of DroneID, a lot of zeros, some completely random garbage that always looks the same frame to frame, and then the CRC

The relevant commit is 000d76e

[alphafox02]

I did multiple IQ files on different days, so power cycles happened. I'll look closer now at all of them and see what remains the same using the latest code in main.

Without sharing the frame from last week online, what I'm seeing is about 68 characters before hitting more or less 50 zeros (there's a few non zeros) and then back to roughly 64 various numbers. Sorry for being vague. I'll study all results closer and attempt to decode them again and see if my lat/long and other information is correct.

[alphafox02]

I checked the remove_turbo.cc file on the machine I decoded the burst with initially and it did not have the commit you referenced above. This explains why my output is different now that I'm decoded it with the newer code in main.

[alphafox02]

I’m somewhat stuck on converting the lat/long - the hex portion seems to make make sense but I’m not sure how to convert the float into a readable lat/long that matches up with where I was standing.

[proto17]

Check out [1] and [2] to see how to convert the value. This is the arithmetic to convert from a floating point lat/long to the packed integer representation that is used in the DroneID frame.

[1] https://github.com/proto17/dji_droneid/blob/main/matlab/updated_scripts/transmit/create_frame_bytes.m#L158
[2] https://github.com/proto17/dji_droneid/blob/main/matlab/updated_scripts/transmit/create_frame_bytes.m#L173

[alphafox02]

Got it and it comes out spot on. Had a little extra help with the conversation but all good now. Thank you.