DAC synchronization across Sayma cards #795
Comments
jordens
added
area:gateware
area:sawg
|
depends on m-labs/jesd204b#5 |
|
@sbourdeauducq Has this been tested on the M-Labs setup? |
|
The code is there and somewhat works intermittently, but I cannot do anything until sinara-hw/sinara#567 is resolved. |
|
@gkasprow Can you expedite the rework, testing, and shipment of the replacement Sayma? |
|
I'm solving #475 . It was caused by at least 3 independent factors. 2 were found and fixed, the last one I'm trying to identify but I think I'm very close. |
|
The microtca mess is annoying but it's not blocking other people's developments and experiments, unlike this. |
|
I solved the issue. I will ship one Sayma AMC ASAP. |
|
I cannot use it without another rtm, can you ship that as well? |
|
sure. |
|
In email today with @marmeladapk and @hartytp, @sbourdeauducq said
By this do you mean synchronization between DAC chip on a single Sayma AMC v2? What debugging steps did you do? |
|
DAC to FPGA. I cannot test between DACs since only one DAC is working on the board I have. |
|
Thanks for the summary. Is there a write up of any of the symptoms you see? Anyway, it sounds like synchronisation between a single DAC and TTL is not reliable yet. |
If you're not getting what you need out of Creotech or me or Tom or Xilinx to progress on your work make more noise. Let's get the hardware sorted so the synchronization testing can progress.
Do please get on with trying to reproduce these bugs and create Issues. Understood that you may not have a timeline for fixing bugs which you've not seen reproduce. What's your timeline for running tests on the hardware to try to reproduce the bugs? |
Do you mean me or @sbourdeauducq? Right now I don't have a good description of the bugs from @sbourdeauducq. Also, AFIACT @sbourdeauducq isn't having any trouble reproducing these issues, so I don't see that me reproducing them as well would help. I can do it but it will take a non-negligible amount of my time without contributing much clear value to the project. AFAICT no one has done any real work/testing on Sayma for months now. It would be useful to understand what the issues are in more detail, who is going to work on them and when. |
Install the beta firmware (with synchronization) and reboot the board a few times while looking at the log. You'll see the sync errors, unless this is a problem with my board in particular. |
|
Can you post a log? Anyway, currently I do not know how the synchronisation process works in any details. It's been heavily rewritten/modified since I last used Sayma. As I don't think there are any docs, I don't expect that I would understand the log messages without investing a significant amount of time reverse engineering the process from the source code. I can do that, but it would be time consuming and I'm not clear that it would add any value to the project. I'm not clear on the responsibilities or expected time commitments for Sayma. I naively assume that the plan here is for @sbourdeauducq to investigate the synchronisation issues on hw he already has. But please correct me if that's not the plan/not possible. |
This was in reply to @sbourdeauducq. I'd like to know more about the "unexplained/unpredictable/obscure bugs." |
We know that it works well:
@hartytp If you want to help, could you validate, with the latest code, that you get reproducible RTIO clock phases (and to what tolerance?) between AMC and Kasli, and between AMC and RTM? You can check the outputs of the three Si5324s after the message "INFO(board_artiq::si5324::siphaser): calibration successful" on the satellites. If that doesn't work, this should be fixed before further synchronization attempts. |
|
OK. Will give it a go next time I have access to my lab. |
|
Okay so it seems that the synchronization problems are three-fold:
|
Where is the DRTIO corruption described? |
|
Thanks! Uuurgh, nasty! Any plan for investigating? |
FWIW, we very rarely (once every few weeks between 5 DRTIO links) see bursts of invalid aux packet errors on Kasli <-> Kasli links as well. Might not be related. |
Contracts spell out who is doing what.
General DRTIO corruption and "Marginal AMC DDMTD jitter on SAWG builds." If this never happens on non-SAWG builds it suggests the hardware is working correctly. Given how subtle these configuration-dependent "features" are in Vivado/Ultrascale/GTH I'm not sure that @hartytp can substantially contribute. It makes most sense for @hartytp to work on testing aspects that are part of his contractual focus and/or tap his unique skills. |
|
There is still this bug where outputs are randomly 180 degrees out of phase (aka "55ns bug" since I've been using a 9MHz waveform to test, but it really is an inversion - which becomes obvious when changing the frequency). This can't be caused by JESD204 sync problems. @jordens Any idea? Can that be inside SAWG e.g. something that isn't reset? |
|
What are you doing exactly, what do you expect and what are you observing? |
|
Start this and keep it running: Reboot Sayma and measure phase difference between Urukul and Sayma after each Sayma reboot. It randomly jumps by 180 degrees instead of remaining constant.
Can you try doing that? Or try other experiments to find out where the problem is? |
Can that be reduced to an MRE?
How would I do that?
Not in the next days. Probably much more efficient if you continue debugging it on-site. |
I am open to suggestions on how to reduce that code. I don't see any obvious ones.
The second Red Pitaya that I connected has one channel on Sayma and the other on Urukul. |
|
After replacing SAWG with a simple DDS core there are no more inversions and synchronization at 2.4GHz often works. So the issue is either in SAWG or the
I'm just also using the Red Pitaya and the remote development infra, and using a physical oscilloscope etc. would not make it more efficient. |
|
I posted this before but here is the script I am using to measure the Urukul/Sayma phase difference. import socket
import numpy as np
import matplotlib.pyplot as plot
from scipy import signal, constants
class RPSCPI:
def connect(self, host):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect((host, 5000))
self.sock_f = self.sock.makefile()
def close(self):
self.sock_f.close()
self.sock.close()
def sendmsg(self, msg):
self.sock.send(msg.encode() + b"\r\n")
def recvmsg(self):
return self.sock_f.readline().strip()
def trigger(self):
self.sendmsg("ACQ:START")
self.sendmsg("ACQ:TRIG NOW")
while True:
self.sendmsg("ACQ:TRIG:STAT?")
if self.recvmsg() == "TD":
break
def get_data(self, channel):
self.sendmsg("ACQ:SOUR{}:DATA?".format(channel))
buff_string = self.recvmsg()[1:-1].split(',')
return np.array(list(map(float, buff_string)))
rp = RPSCPI()
rp.connect("127.0.0.1")
try:
rp.trigger()
y1 = rp.get_data(1)
y2 = rp.get_data(2)
t = np.arange(y1.shape[0])/125e6
y = np.c_[y1, y2].T
z = signal.decimate(y*np.exp(1j*2*np.pi*9e6*t), q=10, ftype="fir", zero_phase=True)[:, 10:]
z = signal.decimate(z, q=10, ftype="fir", zero_phase=True)[:, 10:]
angle = np.angle(np.mean(z[0]*z[1].conj()))
print(angle)
plot.plot(y1)
plot.plot(y2)
plot.show()
finally:
rp.close() |
We've had plenty of examples in the past where this simple logic was plain wrong.
That's intentionally ignoring the most important differences.
|
What is your better idea then? |
The network delays are small compared to the compilation/flashing/boot time of Sayma, and are not really important for grabbing waveforms with the script above. What is the interactive work you want to do? Most of the information about the setup is in your inbox, if you have questions please ask. |
|
@jordens Have you looked into this? |
Codefrom artiq.experiment import *
class SinesUrukulSayma(EnvExperiment):
def build(self):
self.setattr_device("core")
self.sawgs = [self.get_device("sawg"+str(i)) for i in range(4)]
self.basemod = self.get_device("basemod_att0")
self.rfsws = [self.get_device("sawg_sw"+str(i)) for i in range(4)]
self.ttl = [self.get_device("ttl_mcx"+str(i)) for i in range(4)]
# DRTIO destinations:
# 0: local
# 1: Sayma AMC
# 2: Sayma RTM
@kernel
def drtio_is_up(self):
for i in range(3):
if not self.core.get_rtio_destination_status(i):
return False
return True
@kernel
def run(self):
f = 50*MHz
sawg_ftw = self.sawgs[0].frequency0.to_mu(f)
self.core.reset()
print("waiting for DRTIO ready...")
while not self.drtio_is_up():
pass
print("OK")
delay(10*ms)
self.core.reset()
delay(10*ms)
rfsw = self.rfsws[3]
sawg = self.sawgs[0]
ttl = self.ttl[0]
ttl.output()
delay(10*ms)
self.basemod.reset()
delay(10*ms)
self.basemod.set(3., 3., 3., 3.)
delay(10*ms)
rfsw.on()
delay(1*ms)
sawg.reset()
delay(1*us)
sawg.config.set_clr(1, 1, 1)
delay(100*us)
# align to coarse RTIO period
at_mu(now_mu() & ~0x7)
sawg.frequency0.set_mu(sawg_ftw)
delay_mu(1<<8)
sawg.phase0.set_mu(0) # sawg_ftw*now_mu() >> 17)
delay_mu(1<<8)
sawg.amplitude1.set(.4)
delay_mu(1<<9)
sawg.amplitude1.set(.0)
delay_mu(1<<8)
ttl.on()
delay_mu(1<<9)
ttl.off()
Kasli logSayma logThere are two non-determinisms and one problem.
Note that:
I'd propose:
|
No description provided.
The text was updated successfully, but these errors were encountered: