Excess correlation with different digital noise seeds

[AaronParsons]

There has been evidence brought up by @dstorer and confirmed by @david-macmahon that the correlated values with different noise seeds do not achieve the expected noise floor for some baselines. Per @david-macmahon 's slack post https://eoranalysis.slack.com/archives/CS3GR3U3C/p1646345061328679:

I integrated in time all the visibility data from quantity 399 uvh5 files (not to be confused with shorthand for JD 2459399) each containing two ~9 second integrations. At our channel width, this works out to an expected correlation coefficient for uncorrelated noise of about -44.87 dB. Most baselines in the first polarization fit that profile very nicely, with no structure in their phases. There are 35 baselines that have a maximum correlation coefficient that is greater than -35 dB in the first polarization.

These break down into two groups: one group of 19 baselines that are very nearly perfectly correlated (modulo the effect we are blaming on packet loss) and one group of 16 baselines where they are not identical, but have correlation coefficients in the -17 to -22 dB range, with a weird step at the end of the band. I suspect (but don't know for sure) that the first group really did have the same seed value even though we thought they were all unique. Maybe the supported seed space doesn't have enough unique seeds to allow all inputs to have unique seeds?

The second group of baselines is actually two groups of 8 baselines each. I'm not sure, but it may very will be the "first" 8 baselines and the "last" 8 baselines. This one feels like an X engine bug.Here is the list of baselines that exhibited higher than expected (dare I say "excess"?) correlation. The first 8 and the last 8 (i.e. the group of 16 described above) are separated from the middle 19 (the first group described above) by blank lines for clarity.

`(0, 14)
(0, 23)
(0, 100)
(0, 119)
(0, 178)
(0, 179)
(0, 184)
(0, 185)

(107, 110)
(90, 111)
(88, 189)
(105, 93)
(108, 94)
(91, 109)
(89, 92)
(106, 177)
(126, 178)
(125, 179)
(116, 135)
(45, 190)
(46, 191)
(73, 168)
(136, 18)
(155, 27)
(156, 28)
(157, 29)
(158, 30)

(189, 221)
(189, 222)
(7, 8)
(7, 9)
(9, 220)
(9, 221)
(21, 32)
(21, 33) `

[david-macmahon]

The second polarization has the same 19 baselines with correlation coefficient > -1 dB that the first polarization has. The second polarization also has 127 baseline that have a correlation coefficient between my cutoffs of -1 dB and -35 dB. The actual range of "excess correlation" was approximately -15.7 dB to -23.4 dB. Here are these 127 baselines for the second polarization that show "excess correlation":

(0, 2)
(0, 14)
(0, 23)
(0, 100)
(0, 119)
(0, 178)
(0, 179)
(0, 184)
(0, 185)
(2, 1)
(11, 12)
(12, 13)
(14, 23)
(23, 24)
(25, 26)
(26, 39)
(120, 104)
(104, 103)
(87, 101)
(101, 122)
(84, 123)
(123, 121)
(107, 90)
(107, 109)
(107, 111)
(90, 88)
(90, 110)
(90, 189)
(105, 108)
(105, 92)
(105, 94)
(108, 91)
(108, 93)
(108, 109)
(124, 89)
(124, 92)
(106, 126)
(106, 176)
(106, 178)
(126, 125)
(126, 177)
(126, 179)
(81, 82)
(82, 83)
(100, 119)
(119, 138)
(98, 99)
(99, 116)
(99, 135)
(72, 42)
(42, 54)
(56, 69)
(69, 40)
(55, 71)
(71, 70)
(41, 57)
(45, 46)
(45, 189)
(45, 191)
(46, 73)
(46, 168)
(46, 190)
(161, 162)
(162, 182)
(140, 141)
(141, 142)
(160, 180)
(180, 181)
(176, 177)
(176, 30)
(177, 178)
(135, 136)
(135, 18)
(136, 155)
(136, 27)
(155, 18)
(156, 157)
(156, 27)
(156, 29)
(157, 158)
(157, 30)
(158, 29)
(68, 37)
(37, 38)
(52, 36)
(36, 50)
(92, 93)
(93, 94)
(109, 110)
(110, 111)
(112, 127)
(127, 128)
(129, 130)
(165, 166)
(166, 184)
(185, 186)
(186, 187)
(163, 164)
(164, 143)
(144, 145)
(18, 27)
(27, 28)
(29, 30)
(3, 4)
(4, 5)
(15, 16)
(16, 17)
(150, 167)
(168, 169)
(169, 170)
(189, 190)
(189, 221)
(189, 222)
(190, 191)
(7, 8)
(7, 9)
(8, 9)
(9, 220)
(9, 221)
(19, 20)
(20, 10)
(21, 31)
(21, 32)
(21, 33)
(31, 32)
(220, 221)
(221, 222)

I resisted the urge to sort the baselines and instead left them in the same relative order that they appear in the UVH5 file. I suspect/hope that might offer clues as to where in the process things are going awry.