hdfWaterfall with LWA-NA

[ctaylor-physics]

hdfWaterfall.py seems to behave poorly with North Arm data. It appears to have trouble handling stale data at the start of a file, either denying to process the data because it thinks the observation is at the wrong start time, requested offsets not cooperating with the code, or something else.

[jaycedowell]

In addition to the pulsar test this morning I also ran a two hour zenith pointing yesterday evening to look for packet loss/missing data/time tag things. I only see a couple of problems in that run so I'm hoping that that will solve this issue. Although we may still need to do additional tweaking in NDP to better meter what we send to the data recorders.

[jaycedowell]

That also reminds me - checkTimetags.py is very much a product of the strict data order that we have with DP at LWA1. It needs to be updated to be more flexible with that frame structure that ADP and NDP have. I'll make a new issue for that.

[ctaylor-physics]

Last few observation files that I have been reading, after the changes to the command time handling + buffer, have worked completely fine. Those changes, and a few others noted here, probably solved this issue by proxy.

[ctaylor-physics]

Just kidding, looks like some of the recent changes to /lsl/reader/ldp.py are causing some fault. Made it half way through the file before losing the cFrames variable somehow. Going to look at some of these inputs to see if something weird is going on.

... Found #1 at 2024-04-26 15:29:59.678640 with sample rate 19600000.0 Hz
Actual integration time is 49.9 ms
Traceback (most recent call last):                            ]  54.0%    0m28s
  File "/data/local/ctaylor/Commissioning/DRX/HDF5/hdfWaterfall.py", line 625, in <module>
    main(args)
  File "/data/local/ctaylor/Commissioning/DRX/HDF5/hdfWaterfall.py", line 567, in main
    processDataBatch(idf, antennas, obsList[o][0], obsList[o][2], obsList[o][3], args, ds, obsID=o, clip1=clip1, clip2=clip2)
  File "/data/local/ctaylor/Commissioning/DRX/HDF5/hdfWaterfall.py", line 147, in processDataBatchLinear
    tInt, cTime, data = idf.read(args.average)
  File "/home/ctaylor/.local/lib/python3.8/site-packages/lsl/reader/ldp.py", line 1189, in read
    baseframe = copy.deepcopy(cFrames[0])
UnboundLocalError: local variable 'cFrames' referenced before assignment

[ctaylor-physics]

Using the -m flag with the metadata does not cause this error to present. (sometimes)

[jaycedowell]

What file/command signature threw the UnboundLocalError: local variable 'cFrames' referenced before assignment exception?

[ctaylor-physics]

Oh for this it would have been this:
$LEXTS/Commissioning/DRX/HDF5/hdfWaterfall.py /data/network/recent_data/ctaylor/060426_000776222 -a 0.05
I did notice that longer averaging times occasionally will go all the way through, but I don't think that is super surprising given the size of the packet loss events at NA.

[jaycedowell]

This exception should be fixed now in main/lwana.