SAWG not generating sinusoids when amplitude >=0.9

[jbqubit]

Running ARTIQ built from 8fd57e6. Board boots, Ethernet ping works (0% packet loss) and HMC830 lock is fine. Running the following ARTIQ python program. Looking at SMP outputs on scope for four of eight channels. Expectation is sinusoids at 49 MHz. Observe digital-looking output on DACs.

from artiq.experiment import *


class SAWGTestSines(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl_sma_out")
        self.sawgs = [self.get_device("sawg"+str(i)) for i in range(8)]

    @kernel
    def run(self):
        self.core.reset()

        for sawg in self.sawgs:
            delay(1*ms)
            sawg.reset()
            delay(100*ns)
            sawg.amplitude1.set(.9)
            sawg.frequency0.set(49*MHz)

        while True:
            delay(0.5*ms)
            self.ttl_sma_out.pulse(0.5*ms)

[jbqubit]

Will next build a version of gateware with --without-sawg option. Will look for sawtooth pattern.

[jbqubit]

We built using --without-sawg option from master 38b5128. We see sawtooth pattern.

[sbourdeauducq]

Is that the original sines example or did you modify it?
What is in the rtio analyzer?

[jbqubit]

OK. Here's a clue. If sawg.amplitude1.set(.8) there is sinusoidal output. If sawg.amplitude1.set(.9) there is garbage like in screenshot above.

[hartytp]

OK. Here's a clue. If sawg.amplitude1.set(.8) there is sinusoidal output

ooh, can I see a scope/SA shot?

[jbqubit]

from artiq.experiment import *


class SAWGTestSines(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl_sma_out")
        self.sawgs = [self.get_device("sawg"+str(i)) for i in range(8)]

    @kernel
    def run(self):
        self.core.reset()

        for sawg in self.sawgs:
            delay(1*ms)
            sawg.reset()
            delay(100*ns)
            sawg.amplitude1.set(.5)
            sawg.frequency0.set(49*MHz)

        while True:
            delay(0.5*ms)
            self.ttl_sma_out.pulse(0.6*ms)

---

DAC outputs

• yellow: J36
• blue: J29
• red: J26 ---- why is this funky?
• green: J39

---

DAC outputs

• yellow: J16
• blue: J49 ---- why is this funky?
• red: J46
• green: J19

[hartytp]

What are the wiggles there? Reflections or some kind of quantization noise?

[jbqubit]

They're not reflections. They only appear on some channels. For some reason my scope doesn't do AC coupling AND 50-Ohm internal termination at the same time. I'm getting proper external terminators post haste. Prior to posting the present screen shots I confirmed that the wiggles were present even with 50-Ohm internal termination and DC coupling. But its easier to see with AC coupling .

[hartytp]

hmmm...I definitely saw something qualitatively like that even without the SAWG (looking at the saw tooth). Didn't track down it's origin.

[jbqubit]

Oddly, after rebooting and programming sines.py again the "funky/wiggly" behavior went away on the two channels where it was formerly present. Here's SA for "good looking" channel.

[jbqubit]

Now it reappears after a reboot of Sayma. Running sines.py. This time SA is of misbehaving channel.

[jbqubit]

@sbourdeauducq The amplitude dependence looks reproducible. Have you seen this too?

[hartytp]

@sbourdeauducq The amplitude dependence looks reproducible. Have you seen this too?

I agree, it would be helpful to know if @sbourdeauducq can reproduce these results (unless there is a reason that it's hard for him to do that test).

But, if we want to track this down, let's see what the minimally complex setup that reproduces the issue is. Those "wiggles" you've nicely captured on sinusoids look a lot like the ones I saw on the sawtooth generated without the SAWG enabled. They also look like the ones you posted here. @jbqubit are you able to reproduce that when building the latest ARTIQ master when compiling without SAWG and not using Allaki? If so, that suggests that we can begin debugging this without the SAWG.

[jordens]

The steps are expected. This is a sampled system. The sawtooth generators run at lower sample rate and have correspondingly larger steps. The distortion on the sawtooths after the jump is unexpected.

I can't reproduce any of this (#1022, #1039, #1040) with the same SAWG gateware in simulation. Let's first eliminate JESD. What do the sawtooths look like currently? Please measure differentially with proper termination.

[dhslichter]

Those wiggles look like they could be coming from some incorrect interleaving from the CORDICs -- if there was an incorrect phase offset between the start of each interleaved CORDIC, you could get something that looked like this instead of a clean sine wave.

[hartytp]

I suspect it's a JESD issue, since I see glitches on the sawtooth generator (https://irclog.whitequark.org/m-labs/2018-06-01) https://pasteboard.co/HnT4Gf6.jpg

[dhslichter]

If the JESD is incorrectly serializing samples (out of order) this would also produce the effect I was describing, and looking at @hartytp's scope trace it does seem like this might be the case. The sawtooth is a lot more jagged-looking than one would expect on the smooth part (the jaggedness is substantially larger than the spec'ed DNL, to my eye).

[jbqubit]

Also sweeping amplitude over a narrow range results in glitches in output.

from artiq.experiment import *


class SAWGTestSines(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl_sma_out")
        self.sawgs = [self.get_device("sawg"+str(i)) for i in range(8)]
        self.amp_list = self.linspace(.4, .6, 100)

    def linspace(self, start, stop, num):
        dx = (stop-start)/(num-1) 
        return [start+n*dx for n in range(num)]

    @kernel
    def run(self):
        self.core.reset()
        delay(1*ms)
        freq0 = 10*MHz
        for sawg in self.sawgs:
            sawg.reset()
            delay(500*us)
            sawg.frequency0.set(freq0)

        while True:
            for amp in self.amp_list:
                self.ttl_sma_out.pulse(1*us)
                for sawg in self.sawgs:
                    sawg.amplitude1.set(amp)
                    delay(1*ms)
            

[jbqubit]

What I assume:

• JESD is agnostic about its data payload
• it's always running at full data rate (600 MHz right now)
  What I see:
• JESD can successfully transmit data for a variety of waveforms without error
• by changing a SAWG-argument I get glitches
  Conclusion:
• this isn't a JESD issue.

Detail. I can produce complex waveforms indefinitely (>10 minutes) if i avoid certain SAWG parameters. See for example the following where frequency sweeping works (over narrow range) provided I don't sweep amplitude. If I add amplitude sweep there are glitches in sinusoid.

from artiq.experiment import *

class SaymaManyTests(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl_sma_out")
        self.sawgs = [self.get_device("sawg"+str(i)) for i in range(8)]

        nstep = 20
        amp0 = 0.5
        amp1 = 0.5
        #amp1 = 0.6  # this causes glitches
        self.amps = self.linspace(amp0, amp1, nstep)
        self.amps.extend(self.linspace(amp1, amp0, nstep))
        f0 = 2*MHz
        f1 = 4*MHz
        self.fs = self.linspace(f0, f1, nstep)
        self.fs.extend(self.linspace(f1, f0, nstep))


    def linspace(self, start, stop, num):
        dx = (stop-start)/(num-1) 
        return [start+n*dx for n in range(num)]

    @kernel
    def run(self):
        self.core.reset()
        delay(1*ms)
        for sawg in self.sawgs:
            sawg.reset()
            delay(1*us)

        delay(1*ms)
        while True:
            for amp in self.amps:
                for f in self.fs:
                    for sawg in self.sawgs:
                        sawg.amplitude1.set(amp)
                        sawg.frequency0.set(f)
                        self.ttl_sma_out.pulse(50*us)
                        delay(50*us)

[jbqubit]

@sbourdeauducq @jordens What's the next step here? It looks to me like there may be both JESD and SAWG bugs here.

[hartytp]

Probably just a small JESD issue. Fix on the way. See IRC logs...

[enjoy-digital]

@jbqubit: as discussed on IRC, i did some changes on the reset of the elastic buffers + implemented the stpl test. Can you test again? (i'm not able to visualize outputs).

[sbourdeauducq]

i'm not able to visualize outputs

On what board? At M-Labs there is the Red Pitaya on 192.168.1.200.

[sbourdeauducq]

m-labs/migen@33bb06a#commitcomment-29242188

[sbourdeauducq]

@enjoy-digital when I look at the bigger picture of the JESD system, I don't see how your fix can help. You have the GTH input clock which gets multiplied to many-GHz by the GTH PLL(s), and then a number of unsynchronized dividers generate the clocks for each lane. So, lane clocks, which are also the read clocks for the elastic buffers, can have any phase with respect to each other, with random skew in integer multiples of the GTH PLL VCO period.

The elastic buffers can only have random latency from lane to lane, due to the read clock phase randomization that covers one full period - sharing the write reset is not enough.

So you must either:

• rearchitect the JESD core so that the lane-to-lane latency variation is dealt with by the JESD204 protocol.
• align the transceiver clocks like it's done in DRTIO, and have only one elastic buffer for the whole core (or no elastic buffer at all, i.e. clock the whole JESD/SAWG/RTIO design from TXOUTCLK of the master lane).

I suggest removing m-labs/migen@33bb06a from Migen, since I cannot see any problem that can be adequately solved by messing around with the write resets, and having this option encourages mistakes such as the one you made. In general you use an elastic buffer when you do not know the phases of the read and write clocks, and you assume completely random latency jitter between initializations of the elastic buffer.

[sbourdeauducq]

I just tested the latest code on hardware, and expectedly (from my understanding, the JESD behavior has not changed at all): this and the #1040 bug are still there.

[sbourdeauducq]

align the transceiver clocks like it's done in DRTIO,

That may get tricky when there are 16 lanes scattered on two JESD cores, and IIRC Xilinx also has some annoying clocking/transceiver location restrictions for multilane alignment, so the first option sounds better IMHO.

[sbourdeauducq]

BTW, for the first option:

the built-in transceiver elastic buffer is one of the rare features that IME works correctly and with an acceptable design (just set TXBUF_EN and then drive TXUSRCLK/TXUSRCLK2 with a clock that can have any phase but is derived from the same oscillator as TXOUTCLK). So you may want to just use that.

[jbqubit]

@enjoy-digital I still see messed up output at 0.9 amplitude. 4.0.dev+1117.g38dac160

from artiq.experiment import *


class SAWGTestSines(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl_sma_out")
        self.sawgs = [self.get_device("sawg"+str(i)) for i in range(8)]

    @kernel
    def run(self):
        self.core.reset()
        delay(1*ms)

        for sawg in self.sawgs:
            sawg.reset()
            delay(1*us)
            sawg.amplitude1.set(.9)

            sawg.frequency0.set(10*MHz)
            
            delay(100*ns)

        while True:
            delay(0.5*ms)
            self.ttl_sma_out.pulse(0.6*ms)

@jordens @enjoy-digital Please comment on my inferences: #1022 (comment)

[jbqubit]

Running 4.0.dev+1132.g5b73dd86. Glitches still present. Not out of the woods yet.

https://www.youtube.com/watch?v=ynmZMRwUIqI

from artiq.experiment import *


class SAWGTestSines(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl_sma_out")
        self.sawgs = [self.get_device("sawg"+str(i)) for i in range(8)]
        self.amp_list = self.linspace(0.0, 1.0, 100)

    def linspace(self, start, stop, num):
        dx = (stop-start)/(num-1) 
        return [start+n*dx for n in range(num)]

    @kernel
    def run(self):
        self.core.reset()
        delay(1*ms)
        freq0 = 10*MHz
        for sawg in self.sawgs:
            sawg.reset()
            delay(500*us)
            sawg.frequency0.set(freq0)

        while True:
            for amp in self.amp_list:
                self.ttl_sma_out.pulse(1*us)
                for sawg in self.sawgs:
                    sawg.amplitude1.set(amp)
                    delay(1*ms)