Verify DBBC3 multicast data support with independent capture

[wehimwich]

There should be a comparison with the multicast capture by @varenius to verify agreement.

This should be done with a version >= 125 so that the timestamps in the data can be used to make sure the correct data is being compared.

[dehorsley]

These should help: https://github.com/dehorsley/dbbc3mcastc or https://github.com/dehorsley/dbbc3mcast

[wehimwich]

Yes. But what the FS is using is based on what you wrote for those. I am looking for an even more independent comparison. :)

As part of looking into #97, I compared the BBCnnn output to the multicast. I include part of a post from that issue here:

It looks like the TP values from the BBCnnn commands are the same as from the multicast:

2021.069.13:22:05.56#dbtcn#tpcont/ 041l,15418,15341, 041u,15534,15453, 042l,15367,15298, 042u,15474,15395, if, 56072521, 55926651
2021.069.13:22:05.66/bbc041/ 827.600000,f,32, 1,agc,228,231,15534,15418,15453,15341
2021.069.13:22:05.68/bbc042/ 859.600000,f,32, 1,agc,232,246,15474,15367,15395,15298

(The values match once the differing orders are taken into account.)

Since the BBCnnn output agrees with multicast output, I am inclined to believe this shows the net result of the whole chain of multicast handling (capture, unpacking, logging) is correct. You might have already verified your part, but I wanted to verify my part as well. Of course, this only checked two BBCs (selected at random).

We could still compare to @varenius, but I am inclined to think this issue could be closed once we do a multicast to all BBCnnn comparison.

[varenius]

I'm inclined to agree that this could be closed pending the multicast to BBCnnn comparison. After all, there are a limited number of ways one can parse these data. What I did was just to capture the multicast by reading the packets with python, based on the sample script bundled with the DBBC3 firmware (thanks again Sven). Once the numbers look reasonable, and appear in the reasonable order, there are not so many ways to go wrong.

One additional possible nugget of validity is that I have this week managed to convert my TPI-values to Tsys vs time for actual experiments. Applying the resulting calibration values to the correlated visibilities does produce reasonable (!) flux densities for sources on the sky. So, if the multicast TPI values make their way into the FS logs via the standard RXG conversions, I'm inclined to say that it works.
Of course, there may be subtle problems left to investigate (such as the USB/LSB TPI differences), and maybe unknown unknowns. But for the bulk part, it seem to make sense - all the way from the multicast to the sky.

[wehimwich]

I have, I think, verified that the multicast counts in the FS (derived from @dehorsley's muticast packet capture routines) agree with the output of BBCnnn commands. This is for DDC_U v125. This was a little more complicated than I expected, primarily because it takes about 3 seconds to read all 128 BBCs, so up to five multicast packets can be required to fully span the BBCnnn output. It would be possible to arrange this in smaller sets that would only require one packet to compare to and have a truly unique comparison. However, this would have require several runs with careful timing, which is made more complicated with the differences in the BBC command "update epochs" as described in #97.

Instead, I settled for any match between the values for on and off (together) from the BBC commands and the corresponding counts from one of the two directly adjacent multicast packets. My test case is in bbc128b.txt. Out of the 512 TP values in the BBC output lines, there were 19 duplicate values. Of 256 pairs of on and off counts on the BBC output lines there were two duplicates. So the check isn't completely exclusive, but fairly close. I made a script, tpcheck (with .txt added to allow it to upload here), for the comparison that I will leave in the directory for the DBBC3 multicast receiving program, dbtcn. It can be used for more comparisons later.

I did two runs like this. The second, discussed above, successful. The first, bbc128.txt, failed. Two sidebands did not match between the BBC command and multicast data, 038l and 038u:

2021.080.18:38:16.57#dbtcn#tpcont/ 038l,11942,11900, 038u, 9319, 9293, 039l,11153,11124, 039u,11317,11275, 040l, 9904, 9879
2021.080.18:38:17.35/bbc038/1243.600000,e,32, 1,agc,255,255, 9319,11942, 9313,11929
2021.080.18:38:17.59#dbtcn#tpcont/ 038l,11943,11929, 038u, 9325, 9313, 039l,11164,11151, 039u,11319,11301, 040l, 9919, 9904

Interestingly in both cases, the on value corresponds to the multicast before and the off value to the multicast after. It is as though it was part way through updating the BBC output when it was sampled. The sampling was close to what would be the "update epoch" for that BBC. The on minus off differences are on the order of 33-50%. It should be noted that the telescope was looking at the ground and surprisingly the gains are maxed out at 255. Hopefully, the differences would be less extreme for the sky. It would be an unlucky shot, but this could impact USB/LSB differences in #97.

[wehimwich]

As a final step, I spot checked a few Tsys calculations by hand (well, with a spreadsheet). Continuing with bbc128b.txt, I used the .rxg file ottn.txt and the LO setup lo.txt and produced DBBC3 Tsys validation.xlsx.

This was not done by script, but copy-and-paste, which made it fairly easy to complete using a simple repetitive method. I calculated Tsys for USB and LSB for 2 BBCs on each IF. For each IF, one BBC was below position 65, the other above 64. I did an extra BBC on IF H below 65 because the one that I picked initially (BBC059) had a sideband with a Tsys that did print in the log because it was over 1000 K. All the BBC frequencies were different except for the two of the three for IF H, for which I happened to pick as the extra BBC one with the same frequency as the one above 64, i.e., 951.6 MHz. All valid values from the log (33 of them) agreed with the "hand" calculation to better that ±0.05 K (the values were written in the log to 0.1 K precision). I think this verifies both the Tcal look-up and the Tsys calculation.

Doing it by "hand" revealed a couple things (it is often a good idea to look at the raw data :):

1. Although only two BBCs sampled with the BBC commands had exactly the same four TP values, many from the multicast did. Out of 640 total sets of four TP values per BBC (five multicasts) there were 274 duplicates. This would appear to be primarily due to the fact that the frequencies of BBCs 65-128 were the same as for BBCs, 1-64, offset by 64, e.g., BBC 65 was the same as BBC 1. One might expect that to result in 320 duplicates. That is fairly close, but the values for matching frequencies might have small differences in some cases. This also doesn't quite agree with the fact that for the TP values from the BBC commands there were only two duplicates. However, those were sampled over a period of four seconds, so for the commands, the second BBC (offset by 64) should have been sampled in a different second.
2. Most of the channel center frequencies fell on Tcal tabular points, but a few did not. Those few help verify the linear interpolation of the Tcal table. There is only one .rxg for all four bands. In one case, bbc041, a center frequency fell between tabular points for two different bands (C and D). This may argue for using one .rxg file per band. It may be difficult to make a truly spanning table for each IF if there is one Tcal table for all bands with adjustable first LOs. However, even one per band may have some "edge" issues.

I don't think the first point invalidates the previous count comparison. That comparison successfully matched with only the multicast closest before or after each corresponding BBC command sample. The offset by 64 BBC sampling generally occurred between completely different multicasts (in a 13 cases, one of the two multicasts was the same). However for any checks in the future, it would probably be better to have unique frequencies for each BBC (not hard to do). Having more center frequencies off tabular points would probably be a good idea too.

Lastly, I also checked the IF Tsys calculations for the multicast. There is no user interface command to return the same counts for comparison to the multicast. But I did verify the Tsys calculations for both polarizations of both an USB and a LSB first LO. For this system, the only USB LOs are for IFs A and B with an LO value of zero. This puts the center frequency well outside the Tcal table range. In this case the FS uses the closest value. For the LSB LOs used with IFs G and H, the center frequency is between the tabular points for bands C and D, so interpolation is used. The results from the "hand" calculation agreed to better that ±0.05 K with logged values with that approach. It is not clear what these Tsys values represent. Even if there were tabular points within the bands to interpolate with, it is not clear that this is better than perhaps averaging over the whole bandwidth, 4096 MHz. It is also is not clear that the full bandpass is in use or if the input is narrower. Using the BBCs for Tsys seems like a better strategy, but for what it is worth the IF calculations have been verified.