Clock recovery in Sinara/ARTIQ

[hartytp]

Moving from sinara-hw/sinara#515 (comment)

Putting out our thought process so far in detail for anyone who's interested:

Current situation in ARTIQ:

• we use the Si5324 for clock recovery
• That originally seemed like an excellent choice (and, in many ways, it was an excellent choice) as it is extremely flexible and easy to use and has impressively low noise for an IC of its type
• However, the Si5324 does not have deterministic input-output latency, and we've observed quite bad (around 100ps pk-pk) phase fluctuations in the output. This basically rules it out for any use-case that requires time distribution to much better than 1ns, such as generating data converter clocks

So, we are considering implementing something closer to the CDR element White Rabbit, using the DDMTD technique. Comments about this:

• because of the relatively high noise floor of the FPGA CDR circuits, the BW for the PLL needs to be quite low (<100Hz). This makes a traditional analog PLL basically impossible because component values are not practical.
• One approach would be to use an analog PFD + an ADC to digitize the signal, and some digitally-controlled crystal oscillator (DCXO) with a digital loop filter on an FPGA. Calculations suggest that would work fine, but it's a bit complex/expensive. Instead, we plan to follow WR and use the FPGA as the phase detector using the DDMTD approach developed by CERN. Measurements made by CERN and repeated by us suggest that the performance of the two techniques will be similar (although, we only really have data on noise for this, not on longer-term stability -- that's still to do)
• Then the question is "what kind of DCXO to use"?
• Traditional WR uses a VCXO + DAC. We didn't like that for two reasons:
  • firstly, VCXOs are generally much (20dB or more) worse than comparably-priced XOs. And, that's taking the data sheet values, which are measured using a battery to provide the control voltage and in a controlled environment. In practice, some degradation of this noise floor is inevitable once the VCXO is included in a closed-loop system
  • secondly, because we want to use the CDR PLL in a "hostile EMI environment" such as Kasli. Avoiding spurs/broad-band noise due to pickup on the tuning input is also quite challenging unless one adds screening etc
• From these considerations, the optimal approach is probably to use a high-quality XO at something like 1GHz to clock a DDS, then tune the DDS output frequency to lock the RF phase. However, this ends up being quite bulky and expensive, so we decided against it
• A close second is to use a 2-loop PLL DCXO architecture like the one in the SI5324. Here, a microwave VCO, typically running at 5GHz or so, is phase locked to an XO with a wide bandwidth. The fractional part of that phase lock is tuned to lock the output RF to the reference.
• Chips like the Si549 and LMK61e07 are great for this, as they offer the XO, VCO and PLL in a single screened package, including power filtering + LDOs. Moreover, (in principle at least) the IC designers have taken care to minimize digital cross-talk, making life easier for the circuit designer.
• The LMK61e07 looked like a better choice in terms of noise/availability, but we ended up ruling it out because it seems to have bad digital-RF cross-talk. The Si549 does not

So, our plan is:

• Try using DDMTD + Si549 for our clock recovery
• Measure phase stability and see if it's really good enough to distribute data converter clocks
• if the phase stability is good then integrate this into Kasli. NB probably worth keeping the Si5324 as well since it's easier to do things like change RTIO frequency using that chip (maybe population options to switch between the two, depending on cost)

[hartytp]

NB to do: dig out the WR closed-loop phase noise plots and compare to Weida's data. IIRC, WR is quite a bit worse (partially because all the WR VCXOs I've seen used are a lot noisier than the Si549 for the reasons discussed above).

[hartytp]

NB also:

• Mouser has some options for the Si549 in stock that would work fine AFAICT: https://www.mouser.co.uk/Silicon-Laboratories/Passive-Components/Frequency-Control-Timing-Devices/Oscillators/Programmable-Oscillators/Si549-Series/_/N-7jdx1?P=1y98pm2Z1z0zp1d
• cost in quantity is only £10-£20 depending on the version, so not that expensive compared with other options
• availability seems at least as good as Si571 etc, but it's a better solution AFAICT

@gkasprow

But adding little board with vcxo + DAC to Kasli may be simpler approach.

See my post above for the reasons I favor the DCXO to a VCXO + DAC.

[dtcallcock]

If the Si549 scheme continues to work out well, is the plan still to implement it in Kasli 2.0?

[hartytp]

@dtcallcock good question. I was putting off thinking about that until we knew how well it worked. But, I think we have enough data now to show that the performance of a WR PLL would be good enough to be useful for many applications.

At this point, I think we can say that there will be a version of Kasli with the WR PLL in. Whether that's (a) the "standard" version of kasli (b) a population variant (c) a fork of Kasli depends on interest from users.

We also have to decide what we want to do about the Si5324. My preference would be to keep it in Kasli as well but have some means of switching between them. The rational for this is that -- even if we do a good job of integrating WR with ARTIQ/misoc and producing design tools (e.g. a python loop filter design tool) -- the Si5324 is always going to be easier to get running, so it's good to have it there as a fallback option.

We need a fanout after the Si549 anyway, so there isn't much cost making it a (cheapish LVDS) mux and connecting the Si5324 to the second input (it can always be DNPd in any case).

[hartytp]

NB Weida has made some progress getting the TI chip working https://e2e.ti.com/support/clock_and_timing/f/48/t/723417# The noise for that chip should be the same as the Si549, but the documentation/design tools aren't as good so it's harder to find the optimal configuration.

So, it's a bit of a trade-off with the Si549 being less readily available, but much nicer to use. Thankfully, this isn't such a big deal since the two chips are close enough to being footprint + pin compatible that we should be able to use the same layout for both with some resistor jumpers on low-speed signals.

[hartytp]

Does anything say "preliminary data" more than a photograph of a plot on a monitor?

Anyway, here is a measurement of the stability of WR using the Si549. As above, we're using the recovered clocks from two independent KC705s. Now we're beating them together on a mixer as a phase comparator.

• normalization needs double checking
• we see < 1ps pk-pk over a few hours. Rms is <300fs (!) So, this is definitely good enough for quite a few applications
• the data was taken in "office" conditions with no active humidity or temperature control. However, the two KC705s were within a metre of each other so most environmental fluctuations will be common mode.
• to do: unplug the fan on one KC705 to measure the temp-co of the DDMTD
• this setup is a hack of multiple eval boards, baluns, clockers, flexible (not phase stable) coax, etc. Based on previous experience, I would expect to see something of the same order of magnitude even if the DDMTD phase detector were perfect. Once we have WR integrated with Kasli we'll do a more careful measurement and also measure the interferometer stability.

[hartytp]

@jordens what are your thoughts about Kasli integration? Any objections to (a) making this a population variant on the mainline Kasli branch (b) making it a "default" population option?

[hartytp]

NB this is on the basis that we will commit to

• integrating the WR PL with misoc/artiq and supporting it
• providing optimized configuration settings (LF config etc) for a handful of potentially relevant frequencies
• providing some basic python tools to help users pick settings for other frequencies

[jordens]

Is risk the only downside of making it the default? The let's do some more careful analysis, reviews and tests and go for it.
And the loss of convenient loop filter configuration...

[hartytp]

Is risk the only downside of making it the default?

By "default" I mean, populating it by default.

For the time being, my proposal would be to keep both the Si5324 and the WR-PLL. My thinking is that we will need a fanout after the DCXO, so there isn't really any cost in making that fanout a mux. At that point the selection between the two clocking options can be done in software.

So, the only downside is the extra cost associated with the PLL.

Eventually, one might want to DNP the Si5324 to save money, but I'd argue that we should keep it around until we have a decent amount of experience with the WR PLL (no matter how careful one is on the test bench, there is always room for unexpected things to come up in the field).

[hartytp]

And the loss of convenient loop filter configuration...

So long as we do a decent job of the design tool, that shouldn't be an issue. If one isn't trying to scrape the last few dB of noise out, and only wants a simple type-II loop filter then the design tool can be pretty simple and intuitive.

[jordens]

What about the recovered clock output connection from the FPGA GTs? Is that another fanout? And the external reference input? Another one? And we need to make sure that we can turn off both oscillators individually. Otherwise the mux isolation will be problematic.

[hartytp]

What about the recovered clock output connection from the FPGA GTs? Is that another fanout?

You mean CLK_REC on Kasli? I'd have to double check, but I don't think we alter that connection at all since it's not needed for the WR implementation (the connection to the DDMTD input is internal to the FPGA).

And the external reference input?

The easiest thing would be to route the external reference directly to a differential input on the FPGA. Then route an FPGA output to the Si5324 input. That way the FPGA can do the muxing as required. You'll pick up a bit of high-frequency jitter, but nothing the Si5324 can't clean up (i.e. a lot less than you get from the recovered clock). In any case, I don't see that being a problem since the Si5324 isn't really suitable for generating noise/drift-critical clocks anyway due to the known issues with it.

Having said that, one might want a clock buffer between the SMA and the FPGA to provide some protection against abuse from users. If we go down that path, it could be a 2-output LVDS fanout. But, that's optional.

And we need to make sure that we can turn off both oscillators individually.

IIRC (but it's something I'd need to double check before finalizing the designs) DCXO and the Si5324 can both be shutdown in software. Otherwise, we can add a FET on their power lines.

So, essentially yes, the points you raise are all good and there are definitely some implementation details that need to be thought through before we commit to anything.

[hartytp]

Well, probably the best thing is not to worry to much about "default" population options. So long as no one objects to having the pads for the WR circuitry and maybe having to add an extra mux, the best solution is probably to agree to add it to the design and make a decision once we've got ARTIQ up and running with it.

[hartytp]

Trying to wrap up measurements on our test setup this week. Here is some more data:

• more careful measurement of phase stability using a mixer and two independent KC7054 + Si549s
• yellow and red curves are measurements of our WR PLL setup. The differences are small variations in how we do the clocking, see block diagrams below. We made this measurement because we needed to take some baluns back for another measurement, so swapped them for power splitters. If the stability were limited by the DDMTD then the yellow and red curves should be identical
• blue curve is a null measurement checking the stability of the mixer. note that this setup does not include the mess of cabling, baluns etc in the DDMTD measurement, so it's not directly comparable
• RMS phase stability for the two DDMTD setups is 700fs and 400fs, reference measurement is 70fs
• the fact that one of the DDMTD measurements has significantly worse stability suggests that a significant proportion of the instability we measure is due to our measurement setup. Not surprising since we have several meters of standard (not phase-stable/t&m grade) coax and a bunch of baluns + splitters etc. Anyway, it hints that the DDMTD stability is likely good enough that it is non-trivial to make a lab setup that is limited by it! It also suggests that we can expect to measure better stability once we integrate this into Kasli/Sayma and do away with the rats nest
• also shown is the modified Allan deviation

Block.pdf

[hartytp]

Integrated jitter measurement. 1.5ps between 1Hz and 100MHz. NB this is a measurement of absolute jitter and so includes a significant contribution from the wenzel oscillator we use as a reference

[gkasprow]

That looks very good!. I'm just curious where the 170Hz peak comes from. next peak comes probably from SMPS.

[hartytp]

I'm just curious where the 170Hz peak comes from. next peak comes probably from SMPS.

@WeiDaZhang

FWIW this setup is a mess of eval boards with a large loop area for pickup and somewhat poor grounding. I expect some of those spurs to vanish once we put it all on a pcb...

[WeiDaZhang]

It seems the 170 is quite much induced by the beating setup (amp + mixer + filter), or more likely their cables and wires.
By removing them and leave the 2 x (KC705 + DDMTD + DCXO) run alone, the 170 disappears.

[WeiDaZhang]

Well, some other spikes come in.

[hartytp]

@gkasprow so, here is the Sinara-WR proposal in full:

1. Main DCXO. Layout should support Si570, Si549 and LMK61e07. These are all footprint compatible, but not pin compatible. From a quick look, it seems that a few 0R jumpers should allow the same pads to be used for all DCXO choices. NB these are the only DCXOs we've found with close to the correct performance; Si570 is cheapest but lowest performance option; Si549 should be default population option; we need to chase the TI engineers to see why the LMK chip doesn't meet data sheet spec (see above link); if we get the LMK IC to work as expected then that would be the top choice due to better availability and potentially lower noise.
2. Helper DCXO (clock for the DDMTD DFFs). Any of the above DCXOs, use whichever is cheapest (is it more cost effective to save a BOM line by using a Si549 as well or is is cheaper to use a Si570?)
3. Main DCXO feeds a 2-input mux, whose other input is the SI549. This mux must have low close in phase noise, probably an ADCLK948.
4. An output from this mux goes to a MGTREF to clock the transcievers
5. An output from this mux goes to a differential input for the DDMTD DFF
6. Other outputs go to the EEM connectors/BP/etc
7. External reference SMA goes to clock-capable differential input for the other DDMTD DFF when this is the WR master
8. Helper PLL DCXO goes to a cc pin. Try to keep the helper PLL clock and both DDMTD inputs as physically close (adjacent pins) as possible to minimize routing inside the FPGA (don't want to use global clock network if we can avoid it)
9. Recovered clock output routed to an Si5324 input (this can be internally routed to a DDMTD DFF when WR is used)
10. Other Si5324 input driven by an FPGA output. This can be used, for example, to route the SMA clock to the Si5324. NB this shouldn't affect the noise since the Si5324 acts as a jitter cleaner. And, in any case, the problems with the Si5324 prevent it being used when clock quality is absolutely crucial.
11. Need fully independent I2C buses for both DCXOs, which must update at 1MHz clock rate
12. @WeiDaZhang can you double check that the Si549 can be fully shutdown (no RF output) via software? If not, let's add a FET switch on the power line

@gkasprow @sbourdeauducq how does that sound? Any questions? Do we have enough pins to implement this on Kasli?

[hartytp]

LMK61e07 only seems to reach the specified noise performance for some fractional divider ratios. So, when used in a feedback loop the noise can be quite a bit worse than the data sheet indicates. As a result, we don't think it's suitable for WR, although we haven't conclusively ruled it out yet (need to chase TI to get them to confirm that the numbers on the data sheet are not representative).

[WeiDaZhang]

Regarding the Sinara-WR proposal.12 by @hartytp
Si549 has an OE pin on most of its packaging types (indicated with its suffix), and all the packaging types which we can buy from Mouser.

[gkasprow]

OK, I planned to release Sayma and Metlino schematics for review today, I need a few more days to implement these changes. I simply added WR oscillators as assembly option. Is that OK @jbqubit ?.
@hartytp I will look at it tomorrow.

[hartytp]

OK, I planned to release Sayma

Wow! Are all issues on Sayma fixed already?

[hartytp]

Is that OK @jbqubit ?.@hartytp I will look at it tomorrow.

Sorry, what's the question here? Is what okay?

[gkasprow]

@hartytp We agreed with Joe that I will release schematics today :)

[gkasprow]

@hartytp do you care that much about oscillator output type? The SI549 with CMOS output ( 549CBAC000112ABG)
is 3x cheaper than one with LVDS/LVPECL output (549BACB001937ABG). We don't care about initial stability, do we?

[gkasprow]

OK, in other WR designs I did it was routed to CC pin. I'm pretty sure that somebody from WR commmunity told mi that it is not strict requirement. I managed to shift lines and get two GC pins close to each other and GC for ref clock as well.

[hartytp]

@gkasprow GC is definitely not required.

Not sure if CC pin is required. The DDMTD noise doesn't depend strongly on the helper noise, but if it gets too bad this can be a problem. I'd like to keep the helper pin CC if possible. CC/GC is not needed for ref/main DCXO.

[jordens]

DDmtd is symmetric. You can sample the helper with the both the recovered and the main clock or (classic) sample both recovered and main with the helper. The first should work, may need fewer GC inputs in our case but might be one more internal CDC.

[hartytp]

@gkasprow see this comment. Please can we use the LVDS part instead of the LVCMOS. It's the same cost, but should have reduced sensitivity to supply noise etc.

[gkasprow]

The ones with LVDS outputs and same cost are not available. Available LVDS ones are 42$ per piece.

[hartytp]

Aah, you're right, I'd misread that. Okay, let's stick with LVCMOS for now. If the performance is bad then we can change the population option in a later revision.

If we produce a large enough batch of boards with Si549 WR then we should consider ordering a batch of LVDS DCXOs from SI.

[WeiDaZhang]

@gkasprow GC is definitely not required.

Not sure if CC pin is required. The DDMTD noise doesn't depend strongly on the helper noise, but if it gets too bad this can be a problem. I'd like to keep the helper pin CC if possible. CC/GC is not needed for ref/main DCXO.

The scenarios which have been tested by us, always route the HELPER to a CC pair.
As @gkasprow said, it is not strictly required, but there is a bit of WR logic runs on it, a BUFx is needed and CC is, therefore, say "recommended".
The other two, REF and MAIN doesn't require CC from the WR point of view. However, if there is some logic runs on the WR recovered clock - CDR_CLK_CLEANx - in the FPGA, which I assume that is normal, the clock should enter the FPGA in one of the CC/GC/MGTREFCLK ways.

[gkasprow]

Thanks. So in theory it should be sufficient to route the clock to MGTREFCLK only? No clock copy required to GC ?

[WeiDaZhang]

Though depends on FPGA types and applications that CPLL or QPLL is used, I think MGTREFCLK can always route to fabric and drive BUFG. I've tried this on KC705 before with at least one of the CPLL/QPLL.
So in THEORY, it should. I can't think of a use-case which we have to have the CDR_CLK_CLEANx enters the FPGA in more than:

• A normal LVDS pair which is close to HELPER entry and REF entry,
• one of CC/GC/MGTREFCLK ways, and MGTREFCLK is probably preferred as WR-ing the fibre is essential.
  @hartytp Right?

[WeiDaZhang]

This one is probably clearer.

[gkasprow]

I asked just for curiosity. I connected both MGTREFCLK and GC.

[hartytp]

Right @WeiDaZhang.

• the DXCO must go to MGTREF to drive transceivers . This is actually the only DCXO connection that is strictly required since the MGTREF can be routed to a DFF data or clock internally to the FPGA for the DDMTD
• It is useful/preferred to also connect the DCXO to a standard (not CC/GC) DIFF input to allow it to be directly connected to an IOB DFF for DDMTD. This allows us to minimise the amount of internal FPGA routing involved in getting the DCXO to the DDMTD DFF
• As Weida says, the MGTREF input can be routed to the global clock network using buffers inside the FPGA, so we do not need the DCXO routed to a CC pin. However, if there is one free, it's a "nice to have" I guess.

[hartytp]

@jordens @sbourdeauducq @gkasprow here is a BD of the WR test setup we used on the KC705.

@WeiDaZhang correct me if I'm wrong, but the configuration was:

• Transceiver MGTREF clocked at 100MHz from a Wenzel oscillator
• transceiver PLL used to generate 125MHz reference
• PLL output fed to one DDMTD input
• DCXO output goes to ADCLK948 fanout buffer
• one output of the DCXO fanout goes to phase noise meter/interferometer (for phase drift measurements) the other arm goes to the second DDMTD
• WR loop setup as shown (DDMTD->deglitch->IIR->I2C)
• for phase drift measurements, we use a power splitter (ADCLK948 IIRC) to split the reference and and send it to two identical copies of the above setup
• we then use an interferometer (mixer) to measure the drifts between the setups
• NB we didn't test phase determinism here (just drifts). e.g. we didn't implement transceiver resets in the way DRTIO does.

Sources of idiosyncrasy in this setup:

1. @WeiDaZhang used a shared I2C bus for both the helper and main DCXOs. This made the code a little messy and prevented us from updating the DCXOs as fast as we would have liked. In the Sinara implementation, we will have independent buses for each DCXO, simplifying things and allowing us to update the DCXOs more frequently.
2. As @WeiDaZhang didn't have a 125MHz wenzel, he used the transceiver PLLs to generate 125MHz from the 100MHz reference. This step won't be used in the misoc design

[hartytp]

Not shown in the above BD are a few baluns and fanout buffers.

Message @WeiDaZhang for a copy of his Vivado design.

[hartytp]

A few comments about the TI oscillators (@WeiDaZhang please correct any mistakes):

• TI offer two DCXOs LMK61e07 and LMK61e2
• currently we still cannot make the e07 perform as expected from the data sheet, apart from at particular frequencies
• the e2 does behave as expected at all output frequencies. However, it does not have the 4x reference divider that the e07 has. So, unless an external divider is used, the frequency resolution is 4x worse, resulting in 12dB worse 1/f2 phase noise
• The Si549 is generally about 10dB better noise-wise than the TI chips. If the PFD freq of the TI chip is lowered by using the dividers then the noise is broad-band noise (3dB for each factor of 2 in PFD frequency). If the PFD freq is increased then there is significant 1/f2 noise associated with DCXO quantisation. So, the closed-loop noise is always worse than the Si chip, but one has a degree of freedom as to the shape of the noise spectrum.
• The only motivation for the TI chips is availability. However, since we've found some Si549 models with decent availability, that doesn't seem to be such a strong argument any more.
• We'll continue talking to TI support about the issues with the e07, so that we do have a second option in case we need it in the future, but otherwise focus on the Si549

[WeiDaZhang]

• LMK61E07 (the TI chip we wanted to use) has 2x2 REF options (x2 or bypass) (div4 or bypass). Its sister chip LMK61E2 has 2 REF options (x2 or bypass).

• I've attached measurements of some settings of one or the other chip, as not all of them behaves.

• The curves are measured on free run DCXOs, the dash lines indicate estimations when DDMTD locked.

• The frequency resolution of the TI chip is not good enough if we want to obtain good "off-band" (1kHz-10kHz) performance (follow the red '.-' and then the red '--' on the attached figure).

• On the other hand, the "off-band" noise of the TI chip is too high, if we want a good close-in performance (follow the green '.-' and the '--').

• The 1/f^2 close-in noise lines ('.-') caused by the frequency resolution, are drawn for simplified and very much optimistic cases. The reality should be about 10dB worse but depends on specific settings.

• Si549 appears to be 10dB better than the TI chip noise-wise, whichever way we configure the TI, and on all the interested sideband.

• Some spurs of the TI chip, appearing at the sideband of 100Hz -10kHz, are not insignificant, where the measurement on SI549 is quite quiet.

• The off-band noise appears to be the main contributor to the difference between measurement and the datasheet. The datasheet shows the best-case scenario and mentioned the effect of altering the reference. I'm following the issue with TI support.

• The phase noise are ploted at 125MHz.

• I'm afraid the figure is quite big, apologies.
  

[hartytp]

Thanks Weida

[hartytp]

@WeiDaZhang that plot is really nice. Could you stick the Si549 noise on as well for clarity, then it really gives one all the info about the different chips.

One other comment about this: in these calculations we assume the DCXO is updated once per loop iteration, so around 4kHz. The DCXOs support update rates of twice this, so it's possible to update twice per loop iteration to get an effective 1bit increase in tuning resolution and a corresponding 6dB reduction in the 1/f2 noise. This is something we'll probably do in the final version, but we haven't implemented it in our tests.

[hartytp]

@WeiDaZhang: @sbourdeauducq is now using DDMTD in ARTIQ for Sayma SYSREF phase calibration. He's observed that the DDMTD jitter is significantly worse when the FPGA is heavily loaded.

IIRC you also spent quite a while testing DDMTD noise/stability on the KC705 under varying FPGA loads, but didn't see a significant effect. Can you remind me what measurements you made on this?

Also relevant here is @sbourdeauducq's comment

The DDMTD stability [I assume this means jitter, not long-term stability] is better when the core is not clocked from the GTH pins but from the GC clock input. Unfortunately, the GTH and GC clocks are driven by different outputs of the Si5324, and they are not synchronized, so using the GC clock makes the DDMTD clock work better but then it is not synchronized to RTIO anymore :(

@sbourdeauducq to check I understand what you mean by this:

• you're using DDMTD to compare the phase of the HMC7043 output with the FPGA 150MHz rtio clock on the AMC FPGA
• on Sayma AMC, the 150MHz rtio clock comes from the Si5324, which drives both a MGTREF input (CDR_CLK_CLEAN2/ si5324_clkout) and a GC input (CDR_CLK_CLEAN2 / si5324_clkout_fabric)
• the HMC7043 150MHz output comes to a GC input on the FPGA
• the DDMTD helper clock is generated using a MMCM, so the beat frequency is relatively high
• you've found that the DDMTD jitter is significantly worse when we get the rtio clock from the MGTREF (MGTREF->IBUFDS_GTE3->BUFG_GT, cf https://github.com/m-labs/artiq/blob/62d7c89c4841775285bbcdd9c47d00e5e0f52e13/artiq/gateware/drtio/transceiver/gth_ultrascale.py#L683 ) than using the GC pin directly (with an IOB FF?)

@WeiDaZhang how does that compare to your observations on the KC705/design for WR in Sinara?

[hartytp]

Also, @sbourdeauducq am I right in thinking that the deglitcher you've implemented is more of a debouncer

class DDMTDEdgeDetector(Module):
    def __init__(self, i):
        self.rising = Signal()


        history = Signal(4)
        deglitched = Signal()
        self.sync.helper += history.eq(Cat(history[1:], i))
        self.comb += deglitched.eq(i | history[0] | history[1] | history[2] | history[3])


        deglitched_r = Signal()
        self.sync.helper += [
            deglitched_r.eq(deglitched),
            self.rising.eq(deglitched & ~deglitched_r)
        ]

IIRC the Cern papers discuss a few more sophisticated ways of doing this that reduce jitter further (@WeiDaZhang is using one of the ways discussed in that paper)

[hartytp]

NB not trying to suggest you should do anything different here, just trying to keep track of the different work being done on related projects and pool info in one place.

[sbourdeauducq]

@sbourdeauducq to check I understand what you mean by this:

That's correct, except that it's not relevant if the HMC7043 output goes to a GC pin on the FPGA (it's connected to a DFF input).

The "jitter" is the peak-peak jitter measured by the first test of the firmware that operates on the raw (before averaging) results from the DDMTD core.

[sbourdeauducq]

Also, @sbourdeauducq am I right in thinking that the deglitcher you've implemented is more of a debouncer

That's the first deglitcher from the CERN paper, "First Edge selects the first positive edge as a good edge for the time differences counter".

[WeiDaZhang]

@hartytp @sbourdeauducq The attached is the stability test diagram and result.

[WeiDaZhang]

We tried:

• either to input the reference or the main DCXO through the MGTREFCLK pins (Setup A or B in the block diagram), and

• either to have the other input using or not using the IOB, and

• load the FPGA about 75% on either the reference clock domain or the main DCXO clock domain.

I'm afraid I have not found what did we conclude by the time through searching in the email chain. @hartytp
But looking at the curves, and as what you said, the difference seems <= 2 x jitter which we see in other cases.

[jordens]

Was that loaded by static logic or was the logic switching and doing something?

[WeiDaZhang]

LFSRs were feeding DSP48E slices or distributed slices to do multiplications, outputs are XORed to feed LED.
https://github.com/WeiDaZhang/FPGA_LOAD