ZCU208 DAC Clocking Issue

[jramsey123]

This might be on the edge of applicability for qick-specific issues, but I thought I'd ask just the same. I've both a ZCU216 and ZCU208 in my lab, both with CLK204 and XM655 adapters (both ZCU's are cabled to the CLK104 module the same, and the SMAs on the XM655 are connected the same) The intent is to run qick on the ZCU208 because of our requirements. I've taken the ZCU216 Vivado model and adapted it to the ZCU208 (it was mostly an issue of reducing the number of DACs and adding axis terminators to the tproc and switch IPs where needed). The issue is the DAC clocks aren't being detected as locked. I see the following output from debug that I added in qick.py for both ZCU's:

ZCU216
list_rf_blocks: configuring DACs and ADCs to list
Adding DAC 2
DAC 2 - 0 slice enabled
DAC 2 - 1 slice enabled
DAC 2 - 2 slice enabled
DAC 2 - 3 slice enabled
Adding DAC 3
DAC 3 - 0 slice enabled
DAC 3 - 1 slice enabled
DAC 3 - 2 slice enabled
Adding ADC 2
ADC 2 - 0 slice enabled
ADC 2 - 2 slice enabled
clocks_locked: dac_tiles= [2, 3] adc_tiles= [2]
iTile DAC: 2 PLLLockStatus= 1
iTile DAC: 3 PLLLockStatus= 1
iTile ADC: 2 PLLLockStatus= 1
dac_locked: [False, False]
adc_locked: [False]
set_all_clks: board= ZCU216
resetting clocks: 245.76 491.52
lmk: [{'spi_device': PosixPath('/dev/spidev1.1'), 'compatible': 'lmk04828', 'num_bytes': 3}]
lmx: [{'spi_device': PosixPath('/dev/spidev1.3'), 'compatible': 'lmx2594'}, {'spi_device': PosixPath('/dev/spidev1.2'), 'compatible': 'lmx2594'}]
clocks_locked: dac_tiles= [2, 3] adc_tiles= [2]
iTile DAC: 2 PLLLockStatus= 2
iTile DAC: 3 PLLLockStatus= 2
iTile ADC: 2 PLLLockStatus= 2
dac_locked: [True, True]
adc_locked: [True]

Note with the ZCU216 that the first call to clocks_locked shows both DACs and the ADC clock as not locked, and the second call to clocks_locks shows both DACs an the ADC clocks as locked (is the PLLLockStatus value from one of the status registers on the LMX2594?).

ZCU208
list_rf_blocks: configuring DACs and ADCs to list
Adding DAC 2
DAC 2 - 0 slice enabled
DAC 2 - 2 slice enabled
Adding DAC 3
DAC 3 - 0 slice enabled
DAC 3 - 2 slice enabled
Adding ADC 2
ADC 2 - 0 slice enabled
ADC 2 - 1 slice enabled
clocks_locked: dac_tiles= [2, 3] adc_tiles= [2]
iTile DAC: 2 PLLLockStatus= 1
iTile DAC: 3 PLLLockStatus= 1
iTile ADC: 2 PLLLockStatus= 2
dac_locked: [False, False]
adc_locked: [True]
set_all_clks: board= ZCU208
resetting clocks: 245.76 491.52
lmk: [{'spi_device': PosixPath('/dev/spidev1.1'), 'compatible': 'lmk04828', 'num_bytes': 3}]
lmx: [{'spi_device': PosixPath('/dev/spidev1.3'), 'compatible': 'lmx2594'}, {'spi_device': PosixPath('/dev/spidev1.2'), 'compatible': 'lmx2594'}]
clocks_locked: dac_tiles= [2, 3] adc_tiles= [2]
iTile DAC: 2 PLLLockStatus= 1
iTile DAC: 3 PLLLockStatus= 1
iTile ADC: 2 PLLLockStatus= 2
dac_locked: [False, False]
adc_locked: [True]

Note with the ZCU208 that the first call to clocks_locked shows both DAC clocks as not locked but the ADC as locked (this is odd), and the second call to clocks_locks shows both DACs still not locked but the ADC clock as locked. So the locked status doesn't change after the clocks are programmed (I do see the ZCU108 CLK104 PLL locked LEDs flash when the clocks are reset). Is there anything in the PetaLinux build that would cause this? I can run xrfclk.py and the Xilinx embeddedsw gitlab version, xrfclk.c, on both the ZCU216 and the ZCU208, and the CLK104 board PLL locked LEDs flash off, then on in both cases (for both the xrfclk.c built for Linux and the Python version). Can anyone think of something I've missed in the port that would cause this? Thank you!

[jramsey123]

After looking at the LMX2594 data sheet, there's a section on the register initialization order and timing requirements. It says this:

For the most reliable programming, TI recommends this procedure::

1. Apply power to device.
2. Program RESET = 1 to reset registers.
3. Program RESET = 0 to remove reset.
4. Program registers as shown in the register map in REVERSE order from highest to lowest.
5. Wait 10 ms.
6. Program register R0 one additional time with FCAL_EN = 1 to ensure that the VCO calibration runs from a
   stable state.

I changed xrfclk.py so that the _write_LMX_regs functions is now this:

with open(lmx['spi_device'], 'rb+', buffering=0) as f:
# Program RESET = 1 to reset registers.
reset = struct.pack('>I', 0x020000)
f.write(reset[1:])

    # Program RESET = 0 to remove reset.
    remove_reset = struct.pack('>I', 0)
    f.write(remove_reset[1:])
    
    # Program registers as shown in the register map in REVERSE order from highest to lowest
    lmx_reg_number = 112
    for v in reg_vals:
        if lmx_reg_number == 0:
            #print("disabling FCAL_EN until after sleeping 10ms")
            v = v & ~(1 << 3)
        #print("writing lmx R", lmx_reg_number, " value: ", (hex(v)[2:]).upper())
        data = struct.pack('>I', v)
        f.write(data[1:])
        lmx_reg_number -= 1
    
    # Wait 10ms
    time.sleep(0.01)
    
    # Program register R0 one additional time with FCAL_EN = 1 
    # to ensure that the VCO calibration runs from a stable state.
    # Note: the registers are programmed in reverse, and R0 is the last entry in the array (index 112 with 113 registers)
    #print("rewriting lmx R0 value: ", (hex(reg_vals[112])[2:]).upper())
    stable = struct.pack('>I', reg_vals[112])
    f.write(stable[1:])

It doesn't break the ZCU216, nor does it fix my DAC clock locking issue on the ZCU208. So I suppose this timing requirement was already met by xrfclk.py, or the timing requirement isn't really an issue; but 10ms seems like a long time.

[meeg]

Just to make sure - DAC tiles 2 and 3 are the ones you enabled in the firmware (it's possible our code for extracting the list of active tiles doesn't work on 208), and your reference clock frequencies are set to 245.76 MHz (that would of course make the DAC PLL lock fail)?

[jramsey123]

The TICS file I'm using came from the ZCU216 PYNQ port here: https://github.com/sarafs1926/ZCU216-PYNQ/tree/main/tics

Why would the LMK04828 set to 245.76MHz and the LMX2594 set to 491.52MHz cause the DAC PLL lock to fail?

Here is as much of the RF Data Converter configuration I can provide with screen shots. Both DAC 0 and DAC1 slices are enable for DACs 2 and 3 (tiles 230 and 231). And the clock configuration matches the ZCU216 design, as well. Thanks for your response; it does help.

[meeg]

Sorry for the delay, but maybe I have something here:

We're using the LMK-generated clocks that the CLK104 puts on its high-density connector (as opposed to the LMX outputs, which come out on SMP). Looking at the schematics, it seems that the ZCU208 and ZCU216 wire those clock lines to different RF-DAC banks:

ZCU208

ZCU216

If I'm thinking straight, you should therefore try selecting DAC228 for your DAC clock source. (The ADC clocks go to bank 226 in both boards, so probably that is fine.)

By the way, I notice that your clock config has 19.2 MHz ADC clock out - this should be the same as the fabric clock. Not the cause of your current problem, but I think it would be a stumbling block later.

[jramsey123]

I'll rearrange the DACs and use 228 instead of 230 for the clock generator. One thing, I don't see the ability to change the ADC clock to match the fabric clock (see the following). I can only get it to 1/2 the fabric clock. Is there something I'm missing? Thanks for your response; it helps.

[meeg]

That's odd. The Xilinx doc (PG269) doesn't say what the valid values are for the output clock divider. Maybe they differ between the Quad ADCs (as on the ZCU216) and the Dual ADCs (as on the ZCU208).

@leoeltipo what do you think? we need this output clock to be the AXI-S clock, right? is there a workaround, like using a clocking wizard to double this output? or is this telling us there is something else really different between the Quad and Dual RF-ADCs, which needs more work to deal with?

[leoeltipo]

Hi there I’m not sure what you are talked about but I guess it is the same difference between the zcu111 and zcu216. Yes for the 111 the output clock generated by the pll of the adc is half what you need at best. So there is a clock wizard to generate the required clock for the adc digital input stream and the related logic. If this ia what you mean it’s an easy fix.

…

On Jul 22, 2022, at 12:11 PM, Sho Uemura ***@***.***> wrote:  That's odd. The Xilinx doc (PG269) doesn't say what the valid values are for the output clock divider. Maybe they differ between the Quad ADCs (as on the ZCU216) and the Dual ADCs (as on the ZCU208). @leoeltipo what do you think? we need this output clock to be the AXI-S clock, right? is there a workaround, like using a clocking wizard to double this output? or is this telling us there is something else really different between the Quad and Dual RF-ADCs, which needs more work to deal with? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[jramsey123]

@leoeltipo what would you suggest the fix be? Thank you.

[meeg]

We think the correct fix is indeed to use a clock wizard to double that 153.6 to 307.2. The ZCU111 also won't let you set the ADC output clock to match the fabric clock; you can look at that firmware as an example.

[jramsey123]

Okay, I see. I imported/built the ZCU111 and see the Clocking Wizard. I'll add that to the ADC clock, double the clock rate so it matches the fabric clock and use that rate in the ZCU208 design for the AD paths. I'll let you both know how it goes. Thanks again for your help.

[leoeltipo]

Check the timing.xdc file. You’ll need to add one more clock definition.

…

On Jul 22, 2022, at 5:35 PM, Jeff Ramsey ***@***.***> wrote:  Okay, I see. I imported/built the ZCU111 and see the Clocking Wizard. I'll add that to the ADC clock, double the clock rate so it matches the fabric clock and use that rate in the ZCU208 design for the AD paths. I'll let you both know how it goes. Thanks again for your help. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[jramsey123]

Okay, the timing XDC file looks like this now:

set clk_axi [get_clocks -of_objects [get_nets -of_objects [get_pins d_1_i/axis_signal_gen_v4_0/s_axi_aclk]]]

# ADC/DAC
set clk_adc2 [get_clocks -of_objects [get_nets -of_objects [get_pins d_1_i/usp_rf_data_converter_0/clk_adc2]]]
set clk_dac0 [get_clocks -of_objects [get_nets -of_objects [get_pins d_1_i/axis_signal_gen_v4_0/aclk]]]
set clk_dac1 [get_clocks -of_objects [get_nets -of_objects [get_pins d_1_i/axis_signal_gen_v4_4/aclk]]]
set clk_adc2_x2  [get_clocks -of_objects [get_nets -of_objects [get_pins d_1_i/axis_readout_v2_0/aclk]]]
    
set_clock_group -name clk_axi_to_dac0 -asynchronous \
    -group [get_clocks $clk_axi] \
    -group [get_clocks $clk_dac0]
    
set_clock_group -name clk_axi_to_dac1 -asynchronous \
    -group [get_clocks $clk_axi] \
    -group [get_clocks $clk_dac1]    

set_clock_group -name clk_axi_to_adc -asynchronous \
    -group [get_clocks $clk_axi] \
    -group [get_clocks $clk_adc2_x2]  
    
set_clock_group -name clk_dac0_to_adc -asynchronous \
    -group [get_clocks $clk_dac0] \
    -group [get_clocks $clk_adc2_x2]       
    
set_clock_group -name clk_dac1_to_adc -asynchronous \
    -group [get_clocks $clk_dac1] \
    -group [get_clocks $clk_adc2_x2]    
    
set_clock_group -name clk_dac0_to_dac1 -asynchronous \
    -group [get_clocks $clk_dac0] \
    -group [get_clocks $clk_dac1]

[jramsey123]

@leoeltipo and @meeg

I now have the single pulse demo working on the ZCU208. I'll move on to the demo that will transmit a Matlab generated signal file, and chunk it up depending on its size (this was my goal at the start). Thank you both very much for your help.