IndexError - Can't run Hackrf_Sweep through Mossmann's Heatmap.py

[mh4it]

Merry Christmas!!!
Update Below!!!***
I ran a Hackrf_sweep driving around the NRQZ and I’m trying to run the CSVs through Mossmann’s Heatmap.py inside rtl-sdr-misc.
https://github.com/mossmann/rtl-sdr-misc
I split the csv up smaller, to only about 100MB, or 1,000,000 rows, then I reduced a test CSV down to only 9 rows and I'm still getting this error. I'm not sure if my scanning range is part of the issue, I ran the sweep from the upper minimum 1MHz to the upper maximum 7.25GHz. Additionally I also tried erasing the date, time, and redundant column fields, and the error changed to a decimal value within a int(). I'm getting an error because hack.hackrf_sweep records decimal values but heatmap.py uses int()

user@host:~/Downloads/rtl-sdr-misc-master/heatmap$ ./heatmap.py ~/Documents/StoGB1a_NO.csv img_test.png
loading
Traceback (most recent call last):
  File "./heatmap.py", line 618, in <module>
    summarize_pass(args)
  File "./heatmap.py", line 246, in summarize_pass
    low  = int(line[2]) + args.offset_freq
ValueError: invalid literal for int() with base 10: '-60.56'
user@host:~/Downloads/rtl-sdr-misc-master/heatmap$ ./heatmap.py ~/Documents/test.csv img_test.png
loading
x: 45, y: 2, z: (-80.680000, -58.780000)
drawing
Traceback (most recent call last):
  File "./heatmap.py", line 621, in <module>
    img = push_pixels(args)
  File "./heatmap.py", line 422, in push_pixels
    pix[x,y+tape_height] = rgb(zs[x])
  File "./heatmap.py", line 362, in rgb_inner
    return palette[tone_scaled]
IndexError: list index out of range
user@host:~/Downloads/rtl-sdr-misc-master/heatmap$ 

Here are 4 rows from a CSV of only 9 rows testing the Mossmann/Heatmap.py

2020-12-03, 12:53:40, 3946000000, 3951000000, 1000000.00, 20, -73.41, -68.06, -60.90, -62.13, -66.51
2020-12-03, 12:53:40, 3956000000, 3961000000, 1000000.00, 20, -64.63, -62.12, -61.49, -66.87, -70.58
2020-12-03, 12:53:41, 3961000000, 3966000000, 1000000.00, 20, -73.80, -68.74, -66.31, -75.62, -62.97
2020-12-03, 12:53:41, 3971000000, 3976000000, 1000000.00, 20, -62.04, -74.01, -67.02, -68.43, -74.63

Steps to reproduce

1. Run HackRF_Sweep from 1 MHz to 7.25GHz
2. Take small segment of CSV and run through Mossmann Heatmap.py
3. IndexError or Int() space erroring on a decimal value.

Expected behaviour

Output of HeatMap.PNG!

Actual behaviour

Stop on IndexError fault

Version information

Operating system:
Devuan BeoWulf

[mh4it]

The problem is actually coming from very low points in the sweep data.

I have negative spikes or pits that jump from -60 to -90dB, and then it drops to -136 or -145.

These pits are what is causing an IndexError.

Mossman's Heatmap.py is trying to push_pixels but rgb_inner where the IndexError occurs on large negative spikes. Is it division by zero or something? The large negative value breaks the array index math computation perhaps but this is my first dive into Python and I'm slow any help is appreciated!

line 621

img = push_pixels(args)

File "./heatmap.py", line 422, in push_pixels

pix[x,y+tape_height] = rgb(zs[x])

File "./heatmap.py", line 362, in rgb_inner

return palette[tone_scaled]

rgb_inner is defined:

def rgb_fn(palette, min_z, max_z):

    "palette is a list of tuples, returns a function of z"

    def rgb_inner(z):

        tone = (z - min_z) / (max_z - min_z)

        tone_scaled = int(tone * (len(palette)-1))

        return palette[tone_scaled]

    return rgb_inner

rgb_inner is being called by rgb_fn by the function push_pixels:

def push_pixels(args):

    "returns PIL img"

    width = len(args.freqs)

    rgb = rgb_fn(args.palette(), args.db_limit[0], args.db_limit[1])

    img = Image.new("RGB", (width, tape_height + args.pix_height))

    pix = img.load()

    x_size = img.size[0]

    average = [0.0] * width

    tally = 0

    old_y = None

    height = len(args.times)

    for t, zs in collate_row(x_size):

        y = args.times.index(t)

        if not args.compress:

            for x in range(len(zs)):

                pix[x,y+tape_height] = rgb(zs[x])

            continue

        # ugh

        y = args.pix_height - time_compression(height - y, args.compress)

        if old_y is None:

            old_y = y

        if old_y != y:

            for x in range(len(average)):

                pix[x,old_y+tape_height] = rgb(average[x]/tally)

            tally = 0

            average = [0.0] * width

        old_y = y

        for x in range(len(zs)):

            average[x] += zs[x]

        tally += 1

    return img

[mossmann]

I had completely forgotten that I forked heatmap.py long ago. My pull request was never merged, and the upstream project appeared abandoned, so I never made any attempt to keep it up to date.

I recommend trying @miek's new heatmap. It is very new, but it worked for me when I tried it after installing rust and then using cargo to build heatmap. I did:

hackrf_sweep >/tmp/out
heatmap /tmp/heatmap.png </tmp/out

[straithe]

I'm going to close this as there hasn't been a response in a while, but please re-open this issue or open a new one if you still need assistance.