-
Notifications
You must be signed in to change notification settings - Fork 8
/
feed.xml
8026 lines (6759 loc) · 524 KB
/
feed.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>The ZipCPU by Gisselquist Technology</title>
<description>The ZipCPU blog, featuring how to discussions of FPGA and soft-core CPU design. This site will be focused on Verilog solutions, using exclusively OpenSource IP products for FPGA design. Particular focus areas include topics often left out of more mainstream FPGA design courses such as how to debug an FPGA design.
</description>
<link>https://zipcpu.com/</link>
<atom:link href="https://zipcpu.com/feed.xml" rel="self" type="application/rss+xml"/>
<pubDate>Sat, 06 Jul 2024 17:20:39 -0400</pubDate>
<lastBuildDate>Sat, 06 Jul 2024 17:20:39 -0400</lastBuildDate>
<generator>Jekyll v4.2.0</generator>
<image>
<url>https://zipcpu.com/img/gt-rss.png</url>
<title></title>
<link></link>
</image>
<item>
<title>My Personal Journey in Verification</title>
<description><p>This week, I’ve been testing a CI/CD pipeline. This has been my opportunity
to shake the screws and kick the tires on what should become a new verification
product shortly.</p>
<p>I thought that a good design to check might be my
<a href="https://github.com/ZipCPU/sdsdpi">SDIO project</a>. It has roughly all the
pieces in place, and so makes sense for an automated testing pipeline.</p>
<p>This weekend, the CI project engineer shared with me:</p>
<blockquote>
<p>It’s literally the first time I get to know a good hardware project needs
such many verifications and testings! There’s even a real SD card
simulation model and RW test…</p>
</blockquote>
<p>After reminiscing about this for a bit, I thought it might be worth taking a
moment to tell how I got here.</p>
<h2 id="verification-the-goal">Verification: The Goal</h2>
<p>Perhaps the best way to explain the “goal” of verification is by way of an
old “war story”–as we used to call them.</p>
<p>At one time, I was involved with a DOD unit whose whole goal and purpose was
to build quick reaction hardware capabilities for the warfighter. We bragged
about our ability to respond to a call on a Friday night with a new product
shipped out on a C-130 before the weekend was over.</p>
<p>Anyone who has done engineering for a while will easily recognize that this
sort of concept violates all the good principles of engineering. There’s no
time for a requirements review. There’s no time for prototyping–or perhaps
there is, to the extent that it’s always the <em>prototype</em> that heads out the
door to the warfighter as if it were a <em>product</em>. There’s no time to build a
complete test suite, to verify the new capability against all things that could
go wrong. However, we’d often get only one chance to do this right.</p>
<p>Now, how do you accomplish quality engineering in that kind of environment?</p>
<p>The key to making this sort of shop work lay in the “warehouse”, and what
sort of capabilities we might have “lying on the shelf” as we called it.
Hence, we’d spend our time polishing prior capabilities, as well as
anticipating new requirements. We’d then spend our time building, verifying,
and testing these capabilities against phantom requirements, in the hopes that
they’d be close to what we’d need to build should a real requirement arise.
We’d then place these concept designs in the “warehouse”, and show them off
to anyone who came to visit wondering what it was that our team was able to
accomplish. Then, when a new requirement arose, we’d go into this “warehouse”
and find whatever capability was closest to what the customer required and
modify it to fit the mission requirement.</p>
<p>That was how we achieved success.</p>
<table align="center" style="float: right"><tr><td><img src="/img/vlog-wait/rule-of-gold.svg" width="320" /></td></tr></table>
<p>The same applies in digital logic design. You want to have a good set of
tried, trusted, and true components in your “library” so that whenever a new
customer comes along, you can leverage these components quickly to meet his
needs. This is why I’ve often said that well written, well tested, well
verified design components are gold in this business. Such components allow
you to go from zero to product in short order. Indeed, the more well-tested
components you have that you can
<a href="/blog/2020/01/13/reuse.html">reuse</a>, the faster you’ll be
to market with any new need, and the cheaper it will cost you to get there.</p>
<p>That’s therefore the ultimate goal: a library of
<a href="/blog/2020/01/13/reuse.html">reusable</a>
components that can be quickly composed into new products for customers.</p>
<p>As I’ve tried to achieve this objective over the years, my approach to
component verification has changed, or rather grown, many times over.</p>
<h2 id="hardware-verification">Hardware Verification</h2>
<p>When I first started learning FPGA design, I understood nothing about
simulation. Rather than learning how to do simulation properly, I instead
learned quickly how to test my designs in hardware. Most of these designs
were DSP based. (My background was DSP, so this made sense …) Hence,
the following approach tended to work for me:</p>
<ul>
<li>
<p>I created access points in the hardware that allowed me to read and write
registers at key locations within the design.</p>
</li>
<li>
<p>One of these “registers” I could write to controlled the inputs to my DSP
pipeline.</p>
</li>
<li>
<p>Another register, when written to, would cause the design to “step” the
entire DSP pipeline as if a new sample had just arrived from the A/D.</p>
</li>
<li>
<p>A set of registers within the design then allowed me to read the state of
the entire pipeline, so I could do debugging.</p>
</li>
</ul>
<p>This worked great for “stepping” through designs. When I moved to processing
real-time information, such as the A/D results from the antenna connected to
the design, I build an internal logic analyzer to catch and capture key
signals along the way.</p>
<p>I called this “Hardware in the loop testing”.</p>
<p>Management thought I was a genius.</p>
<p>This approach worked … for a while. Then I started realizing how painful it
was. I think the transition came when I was trying to debug
<a href="/2018/10/02/fft.html">my FFT</a> by writing test vectors to
an Arty A7 circuit board via UART, and reading the results back to display
them on my screen. Even with the hardware in the loop, hitting all the test
vectors was painfully slow.</p>
<p>Eventually, I had to search for a new and better solution. This was just too
slow. Later on, I would start to realize that this solution didn’t catch
enough bugs–but I’ll get to that in a bit.</p>
<h2 id="happy-path-simulation-testing">Happy Path Simulation Testing</h2>
<p><a href="https://en.wikipedia.org/wiki/Happy_path">“Happy path” testing</a>
is a reference to simply testing working paths
through a project’s environment. To use an aviation analogy, a <a href="https://en.wikipedia.org/wiki/Happy_path">“happy path”
test</a>
might make sure the ground avoidance radar never alerted when you
weren’t close to the ground. It doesn’t make certain that the radar
necessarily does the right thing when you are close to the ground.</p>
<p>So, let’s talk about my next project: the
<a href="/about/zipcpu.html">ZipCPU</a>.</p>
<p>Verification of the <a href="/about/zipcpu.html">CPU</a>
began with an <a href="https://github.com/ZipCPU/zipcpu/blob/a20b6064ea794d66fdeb2e00929287d7f2dc9ac6/bench/asm/simtest.s">assembly
program</a>
the <a href="/about/zipcpu.html">ZipCPU</a> would run. The
<a href="https://github.com/ZipCPU/zipcpu/blob/a20b6064ea794d66fdeb2e00929287d7f2dc9ac6/bench/asm/simtest.s">program</a>
was designed to test all the instructions of the
<a href="/about/zipcpu.html">CPU</a>
with sufficient fidelity to know when/if the
<a href="/about/zipcpu.html">CPU</a> worked.</p>
<p>The test had one of two outcomes. If the program halted, then the test was
considered a success. If it detected an error, the
<a href="/about/zipcpu.html">CPU</a> would execute a
<code class="language-plaintext highlighter-rouge">BUSY</code> instruction (i.e. jump to current address) and then perpetually loop.
My test harness could then detect this condition and end with a failing exit
code.</p>
<p>When the <a href="/about/zipcpu.html">ZipCPU</a> acquired a software
tool chain (GCC+Binutils) and C-library support, this <a href="https://github.com/ZipCPU/zipcpu/blob/a20b6064ea794d66fdeb2e00929287d7f2dc9ac6/bench/asm/simtest.s">assembly
program</a>
was abandoned and replaced with a <a href="https://github.com/ZipCPU/zipcpu/blob/a20b6064ea794d66fdeb2e00929287d7f2dc9ac6/sim/zipsw/cputest.c">similar program in
C</a>.
While I still use <a href="https://github.com/ZipCPU/zipcpu/blob/a20b6064ea794d66fdeb2e00929287d7f2dc9ac6/sim/zipsw/cputest.c">this
program</a>,
it’s no longer the core of the <a href="/about/zipcpu.html">ZipCPU</a>’s
verification suite. Instead, I tend to use it to shake out any bugs in any
new environment the <a href="/about/zipcpu.html">ZipCPU</a> might be
placed into.</p>
<p>This approach failed horribly, however, when I tried integrating an <a href="https://github.com/ZipCPU/zipcpu/blob/a20b6064ea794d66fdeb2e00929287d7f2dc9ac6/rtl/core/pfcache.v">instruction
cache</a>
into the <a href="/about/zipcpu.html">ZipCPU</a>. I built the
<a href="https://github.com/ZipCPU/zipcpu/blob/a20b6064ea794d66fdeb2e00929287d7f2dc9ac6/rtl/core/pfcache.v">instruction
cache</a>.
I tested the <a href="https://github.com/ZipCPU/zipcpu/blob/a20b6064ea794d66fdeb2e00929287d7f2dc9ac6/rtl/core/pfcache.v">instruction
cache</a>
in isolation. I tested the
<a href="https://github.com/ZipCPU/zipcpu/blob/a20b6064ea794d66fdeb2e00929287d7f2dc9ac6/rtl/core/pfcache.v">cache</a>
as part of the
<a href="/about/zipcpu.html">CPU</a>. I convinced myself that it worked.
Then I placed my “working” design onto hardware and <a href="/zipcpu/2017/12/28/ugliest-bug.html">all
hell broke lose</a>.</p>
<p>This was certainly not “the way.”</p>
<h2 id="formal-verification">Formal Verification</h2>
<p>I was then asked to <a href="/blog/2017/10/19/formal-intro.html">review a new, open source, formal verification tool called
SymbiYosys</a>. The tool
handed my cocky attitude back to me, and took my pride down a couple steps. In
particular, I found a bunch of bugs in a FIFO I had used for years. The bugs
had never shown up in hardware testing (that I had noticed at least), and
certainly hadn’t shown up in any of my <a href="https://en.wikipedia.org/wiki/Happy_path">“Happy path”
testing</a>. This left me wondering,
how many other bugs did I have in my designs that I didn’t know about?</p>
<p>I then started <a href="/blog/2018/01/22/formal-progress.html">working through my previous projects, formally verifying all my
prior work</a>. In every
case, I found more bugs. By the time I got to the
<a href="/about/zipcpu.html">ZipCPU</a>–<a href="/blog/2018/04/02/formal-cpu-bugs.html">I found a myriad of bugs
in what I thought was a “working”</a>
<a href="/about/zipcpu.html">CPU</a>.</p>
<p>I’d like to say that the quality of my IP went up at this point. I was
certainly finding a lot of bugs I’d never found before by using formal methods.
I now knew, for example, how to guarantee I’d never have any more of those
cache bugs I’d had before.</p>
<p>So, while it is likely that my IP quality was going up, the unfortunate
reality was that I was still finding bugs in my “formally verified”
IP–although not nearly as many.</p>
<p>A <a href="/formal/2020/06/12/four-keys.html">couple of improvements</a>
helped me move forward here.</p>
<ol>
<li>
<p>Bidirectional formal property sets</p>
<p>The biggest danger in formal verification is that you might <code class="language-plaintext highlighter-rouge">assume()</code>
something that isn’t true. The first way to limit this is to make
sure you never <code class="language-plaintext highlighter-rouge">assume()</code> a property within the design, but rather you
only <code class="language-plaintext highlighter-rouge">assume()</code> properties of inputs–never outputs, and never local
registers.</p>
<p>But how do you know when you’ve assumed too much? This can be a challenge.</p>
<p>One of the best ways I’ve found to do this is to create a bidirectional
property set. A bus master, for example, would make assumptions about
how the slave would respond. A similar property set for the bus slave
would make assumptions about what the master would do. Further, the slave
would turn the master’s assumptions into verifiable assertions–guaranteeing
that the master’s assumptions were valid. If you can use the same property
set in this manner for both master and slave, save that you swap
assumptions and assertions, then you can verify both in isolation to
include only assuming those things that can be verified elsewhere.</p>
<p>Creating such property sets for both AXI-Lite and AXI led me to find
many bugs in Xilinx IP. This alone suggested that I was on the “right path”.</p>
</li>
<li>
<p>Cover checking</p>
<p>I also learned to use <a href="/formal/2018/07/14/dev-cycle.html">formal coverage
checking</a>, in
addition to straight assertion
based verification. Cover checks weren’t the end all, but they could
be useful in some key situations. For example, a quick cover check might
help you discover that you had gotten the reset polarity wrong, and so
all of your formal assertions were passing because your design was assumed
to be held in reset. (This has happened to me more than once. Most
recently, the <a href="/blog/2024/06/13/kimos.html">cost was a couple of months
delay</a> on what should’ve
otherwise been a straight forward hardware bringup–but that wasn’t really
a <em>formal</em> verification issue.)</p>
<p>For a while, I also <a href="/formal/2018/07/14/dev-cycle.html">used cover checking to quickly discover (with minimal
work) how a design component might work within a larger
environment</a>. I’ve
since switched to simulation checking (with assertions enabled) for my
most recent examples of this type of work, but I do still find it valuable.</p>
</li>
<li>
<p><a href="/blog/2018/03/10/induction-exercise.html">Induction</a></p>
<p><a href="/blog/2018/03/10/induction-exercise.html">Induction</a> isn’t
really a “new” thing I learned along the way, but it is worth mentioning
specially. As I learned formal verification, I learned to use
<a href="/blog/2018/03/10/induction-exercise.html">induction</a>
right from the start and so I’ve tended to use
<a href="/blog/2018/03/10/induction-exercise.html">induction</a>
in every proof I’ve ever done. It’s just become my normal practice from day
one.</p>
<p><a href="/blog/2018/03/10/induction-exercise.html">Induction</a>,
however, takes a lot of work. Sometimes it takes so much work I wonder
if there’s really any value in it. Then I tend to find some key bug or
other–perhaps a buffer overflow or something–some bug I’d have never found
without
<a href="/blog/2018/03/10/induction-exercise.html">induction</a>.
That alone keeps me running
<a href="/blog/2018/03/10/induction-exercise.html">induction</a>.
every time I can. Even better, once the
<a href="/blog/2018/03/10/induction-exercise.html">induction</a>.
proof is complete, you can often <a href="/formal/2019/08/03/proof-duration.html">trim the entire formal proof down from
15-20 minutes down to less than a single
minute</a>.</p>
</li>
<li>
<p>Contract checking</p>
<p>My initial formal proofs were haphazard. I’d throw assertions at the wall
and see what I could find. Yes, I found bugs. However, I never really had
the confidence that I was “proving” a design worked. That is, not until I
learned of the idea of a “formal contract”. The “formal contract” simply
describes the essence of how a component worked.</p>
<p>For example, in a memory system, the formal contract might have the solver
track a single value of memory. When written to, the value should change.
When read, the value should be returned. If this contract holds for all such
memory addresses, then the memory acts (as you would expect) … like a
<em>memory</em>.</p>
</li>
<li>
<p>Parameter checks</p>
<p>For a while, I was maintaining <a href="https://github.com/ZipCPU/zbasic">“ZBasic”–a basic ZipCPU
distribution</a>. This was where I did all
my simulation based testing of the
<a href="/about/zipcpu.html">ZipCPU</a>. The problem was, this
approach didn’t work. Sure, I’d test the
<a href="/about/zipcpu.html">CPU</a> in one configuration, get it
to work, and then put it down believing the
“<a href="/about/zipcpu.html">CPU</a>” worked. Some time later,
I’d try the <a href="/about/zipcpu.html">CPU</a> in a different
configuration–such as pipelined vs non-pipelined, and … it
would fail in whatever mode it had not been tested in. The problem with the
<a href="https://github.com/ZipCPU/zbasic">ZBasic approach</a> is that it tended to only
check one mode–leaving all of the others unchecked.</p>
<p>This lead my to adjust the proofs of the
<a href="/about/zipcpu.html">ZipCPU</a> so that the
<a href="/about/zipcpu.html">CPU</a> would at least be formally
verified with as many parameter configurations as I could to make sure it
would work in all environments.</p>
</li>
</ol>
<p>I’ve written more about <a href="/formal/2020/06/12/four-keys.html">these parts of a proof some time
ago</a>, and I still stand
by them today.</p>
<p>Yes, formal verification is hard work. However, a well verified design is
highly valuable–on the shelf, waiting for that new customer requirement to
come in.</p>
<p>The problem with all this formal verification work lies in its (well known)
Achilles heel. Because formal verification includes an exhaustive
combinatorial search for bugs across all potential design inputs and states,
it can be computationally expensive. Yeah, it can take a while. To reduce
this expense, it’s important to limit the scope of what is verified. As a
result, I tend to verify design <em>components</em> rather than entire designs. This
leaves open the possibility of a failure in the logic used to connect all
these smaller, verified components together.</p>
<h2 id="autofpga-and-better-crossbars">AutoFPGA and Better Crossbars</h2>
<p>Sure enough, the next class of bugs I had to deal with were integration bugs.</p>
<p>I had to deal with several. Common bugs included:</p>
<ol>
<li>
<p>Using unnamed ports, and connecting module ports to the wrong signals.</p>
<p>At one point, I decided the
<a href="/zipcpu/2017/11/07/wb-formal.html">Wishbone</a>
“stall” port should come before the
<a href="/zipcpu/2017/11/07/wb-formal.html">Wishbone</a>
acknowledgment port. Now, how many designs had to change to accommodate
that?</p>
</li>
<li>
<p>I had a bunch of problems with my <a href="/blog/2017/06/22/simple-wb-interconnect.html">initial interconnect
design</a>
methodology. Initially, I used the slave’s
<a href="/zipcpu/2017/11/07/wb-formal.html">Wishbone</a>
strobe signal as an address decoding signal. I then had a bug where the
address would move off of the slave of interest, and the acknowledgment
was never returned. The result of that bug was that the design hung any
time I tried to read the entirety of <a href="/blog/2019/03/27/qflexpress.html">flash
memory</a>.</p>
<p>Think about how much simulation time and effort I had to go through to
simulate reading an <em>entire</em> <a href="/blog/2019/03/27/qflexpress.html">flash
memory</a>–just to find
this bug at the end of it. Yes, it was painful.</p>
</li>
</ol>
<p>Basically, when connecting otherwise “verified” modules together by hand,
I had problems where the result wasn’t reliably working.</p>
<p>The first and most obvious solution to something like this is to use a linting
tool, such as <code class="language-plaintext highlighter-rouge">verilator -Wall</code>.
<a href="https://www.veripool.org/verilator/">Verilator</a> can find things like
unconnected pins and such. That’s a help, but I had been doing that from
early on.</p>
<p>My eventual solution was twofold. First, I redesigned my <a href="/blog/2019/07/17/crossbar.html">bus
interconnect</a> from the
top to the bottom. You can find the new and redesigned
<a href="/blog/2019/07/17/crossbar.html">interconnect</a> components
in my <a href="https://github.com/ZipCPU/wb2axip">wb2axip repository</a>. Once these
components were verified, I then had a proper guarantee: all masters would get
acknowledgments (or errors) from all slave requests they made. Errors would
no longer be lost. Attempts to interact with a non-existent slave would
(properly) return bus errors.</p>
<p>To deal with problems where signals were connected incorrectly, I built a tool
I call <a href="/zipcpu/2017/10/05/autofpga-intro.html">AutoFPGA</a> to
connect components into designs. A special tag given to the tool would
immediately connect all bus signals to a bus component–whether it be a slave
or master, whether it be connected to a
<a href="/zipcpu/2017/11/07/wb-formal.html">Wishbone</a>,
<a href="/formal/2018/12/28/axilite.html">AXI-Lite</a>, or
<a href="/formal/2019/05/13/axifull">AXI</a> bus. This required that my
slaves followed one of two conventions. Either all the bus ports had to
follow a basic port ordering convention, or they needed to follow a bus
naming convention. Ideally, a slave should follow both. Further, after
finding even more port connection bugs, I’m slowly moving towards the practice
of naming all of my port connections.</p>
<p>This works great for composing designs of bus components. Almost all of my
designs now use this approach, and only a few (mostly test bench) designs
remain where I connect bus components by hand manually.</p>
<h2 id="mcy">MCY</h2>
<p>At one time along the way, I was asked to review <a href="https://github.com/YosysHQ/mcy">MCY: Mutation Coverage with
Yosys</a>. My review back to the team was …
mixed.</p>
<p><a href="https://github.com/YosysHQ/mcy">MCY</a>
works by intentionally breaking your design. Such changes to the design are
called “mutations”. The goal is to determine whether or not the mutated
(broken) design will trigger a test failure. In this fashion, the test suite
can be evaluated. A “good” test suite will be able to find any mutation.
Hence, <a href="https://github.com/YosysHQ/mcy">MCY</a>
allows you to measure how good your test suite is in the first place.</p>
<p>Upon request, I tried <a href="https://github.com/YosysHQ/mcy">MCY</a> with the
<a href="/about/zipcpu.html">ZipCPU</a>. This turned into a bigger
challenge than I had expected. Sure, <a href="https://github.com/YosysHQ/mcy">MCY</a>
works with <a href="https://github.com/steveicarus/iverilog">Icarus Verilog</a>,
<a href="https://www.veripool.org/verilator/">Verilator</a>, and even (perhaps) some other
(not so open) simulators as well. However, when I ran a design under
<a href="https://github.com/YosysHQ/mcy">MCY</a>, my simulations tended to find only a
(rough) 70% of any mutations. The formal proofs, however, could find 95-98% of
any mutations.</p>
<p>That’s good, right?</p>
<p>Well, not quite. The problem is that I tend to place all of my formal
logic in the same file as the component that would be mutated. In order to
keep the mutation engine from mutating the formal properties, I had to remove
the formal properties from the file to be mutated into a separate file.
Further, I then had to access the values that were to be assumed or asserted
external from the file under test using something often known as “dot notation”.
While (System)Verilog allows such descriptions natively, there weren’t any open
source tools that allowed such external formal property descriptions.
(Commercial tools allowed this, just not the open source
<a href="https://github.com/YosysHQ/sby">SymbiYosys</a>.) This left me stuck with a couple
of unpleasant choices:</p>
<ol>
<li>I could remove the ability of the
<a href="/about/zipcpu.html">ZipCPU</a>
(or whatever design) to be formally verified with Open Source tools,</li>
<li>I could give up on using
<a href="/blog/2018/03/10/induction-exercise.html">induction</a>,</li>
<li>I could use <a href="https://github.com/YosysHQ/mcy">MCY</a> with simulation only, or</li>
<li>I could choose to not use <a href="https://github.com/YosysHQ/mcy">MCY</a> at all.</li>
</ol>
<p>This is why I don’t use <a href="https://github.com/YosysHQ/mcy">MCY</a>. It may be a
“good” tool, but it’s just not for me.</p>
<p>What I did learn, however, was that my
<a href="/about/zipcpu.html">ZipCPU</a> test suite was checking the
<a href="/about/zipcpu.html">CPU</a>’s functionality nicely–just not
the debugging port. Indeed, none of my tests checked the debugging port to the
<a href="/about/zipcpu.html">CPU</a>
at all. As a result, none of the (simulation-based) mutations of the
debugging port were ever caught.</p>
<p>Lesson learned? My test suite still wasn’t good enough. Sure, the
<a href="/about/zipcpu.html">CPU</a> might
“work” today, but how would I know some change in the future wouldn’t break it?</p>
<p>I needed a better way of knowing whether or not my test suite was good enough.</p>
<h2 id="coverage-checking">Coverage Checking</h2>
<p>Sometime during this process I discovered
<a href="https://en.wikipedia.org/wiki/Code_coverage">coverage checking</a>.
<a href="https://en.wikipedia.org/wiki/Code_coverage">Coverage checking</a>.
is a process of automatically watching over all of your simulation based tests
to see which lines get executed and which do not. Depending on the tool,
coverage checks can also tell whether particular signals are ever flipped or
adjusted during simulation. A good coverage check, therefore, can provide
some level of indication of whether or not all control paths within a design
have been exercised, and whether or not all signals have been toggled.</p>
<p>Coverage metrics are actually kind of nice in this regard.</p>
<p>Sadly, coverage checking isn’t as good as mutation coverage, but … it’s
better than nothing.</p>
<p>Consider a classic coverage failure: many of my simulations check for
AXI <a href="https://en.wikipedia.org/wiki/Back_pressure">backpressure</a>. Such
<a href="https://en.wikipedia.org/wiki/Back_pressure">backpressure</a> is generated when
either <code class="language-plaintext highlighter-rouge">BVALID &amp;&amp; !BREADY</code>, or <code class="language-plaintext highlighter-rouge">RVALID &amp;&amp; !RREADY</code>. If your design is to
follow the AXI specification, it should be able to handle
<a href="https://en.wikipedia.org/wiki/Back_pressure">backpressure</a>
properly. That is, if you hold <code class="language-plaintext highlighter-rouge">!BREADY</code> long enough, it should be possible
to force <code class="language-plaintext highlighter-rouge">!AWREADY</code> and <code class="language-plaintext highlighter-rouge">!WREADY</code>. Likewise, it should be possible to hold
<code class="language-plaintext highlighter-rouge">RREADY</code> low long enough that <code class="language-plaintext highlighter-rouge">ARREADY</code> gets held low. A well verified,
bug-free design should be able to deal with these conditions.</p>
<p>However, a “good” design should never create any significant
<a href="https://en.wikipedia.org/wiki/Back_pressure">backpressure</a>.
Hence, if you build a simulation environment from “good” working components,
you aren’t likely to see much
<a href="https://en.wikipedia.org/wiki/Back_pressure">backpressure</a>. How then should a
component’s <a href="https://en.wikipedia.org/wiki/Back_pressure">backpressure</a>
capability be tested?</p>
<p>My current solution here is to test
<a href="https://en.wikipedia.org/wiki/Back_pressure">backpressure</a>
via formal methods, with the unfortunate consequence that some conditions
will never get tested under simulation. The result is that I’ll never get
to 100% coverage with this approach.</p>
<p>A second problem with coverage regards the unused signals. For example,
AXI-Lite has two signals, <code class="language-plaintext highlighter-rouge">AWPROT</code> and <code class="language-plaintext highlighter-rouge">ARPROT</code>, that are rarely used by
any of my designs. However, they are official AXI-Lite (and AXI) signals.
As a result,
<a href="/zipcpu/2017/10/05/autofpga-intro.html">AutoFPGA</a>
will always try to connect them to an AXI-Lite (or AXI) port, yet none of my
designs use these. This leads to another set of exceptions that needs to be
made when measuring coverage.</p>
<p>So, coverage metrics aren’t perfect. Still, they can help me find
what parts of the design are (and are not) being tested well. This can then
help feed into better (and more complete) test design.</p>
<p>That’s the good news. Now let’s talk about some of the not so good parts.</p>
<p>When learning formal verification, I spent some time formally verifying
Xilinx IP. After finding several bugs, I spoke to a Xilinx executive
regarding how they verified their IP. Did they use formal methods? No.
Did they use their own AXI Verification IP? No. Yet, they were very proud of
how well they had verified their IP. Specifically, their executive bragged
about how good their coverage metrics were, and the number of test points
checked for each IP.</p>
<p>Hmm.</p>
<p>So, let me get this straight: Xilinx IP gets good coverage metrics, and hits
a large number of test points, yet still has bugs within it that I can find
via formal methods?</p>
<p>Okay, so … how severe are these bugs? In one case, the bugs would totally
break the AXI bus and bring the system containing the IP down to a screeching
halt–if the bug were ever tripped. For example, if the system requested both
a read burst and a write burst at the same time, one particular slave might
accomplish the read burst with the length of the write burst–or vice versa.
(It’s been a while, so I’d have to look up the details to be exact regarding
them.) In another case dealing with a network controller, it was possible
to receive a network packet, capture that packet correctly, and then return
a corrupted packet simply because the <a href="/blog/2021/08/28/axi-rules.html">AXI bus
handshakes</a> weren’t properly
implemented. To this day this bugs have not been fixed, and it’s nearly five
years later.</p>
<p>Put simply, if it is possible for an IP to lock up your system completely,
then that IP shouldn’t be trusted until the bug is fixed.</p>
<p>How then did Xilinx manage to convince themselves that their IP was high
quality? By “good” coverage metrics.</p>
<p>Lesson learned? <a href="https://en.wikipedia.org/wiki/Code_coverage">Coverage
checking</a> is a good thing, and it
can reveal holes in any simulation-based verification suite. It’s just not
good enough on its own to find all of what you are missing.</p>
<p>My conclusion? Formal verification, followed by a simulation test suite that
evaluates coverage statistics is something to pay attention to, but not the
end all be-all. One tool isn’t enough. Many tools are required.</p>
<h2 id="self-checking-testbenches">Self-Checking Testbenches</h2>
<p>I then got involved with ASIC design.</p>
<p><a href="/blog/2017/10/13/fpga-v-asic.html">ASIC design differs from FPGA design in a couple of
ways</a>. Chief among them
is the fact that the ASIC design must work the first time. There’s little to
no room for error.</p>
<table align="center" style="float: right"><caption>Fig 1. A typical verification environment</caption><tr><td><img src="/img/vjourney/verilogtb.svg" width="320" /></td></tr></table>
<p>When working with my first ASIC design, I was introduced to a more formalized
simulation flow. Let me explain it this way, looking at Fig. 1. Designs
tend to have two interfaces: a bus interface, together with a device I/O
interface. A test script can then be used to drive some form of bus functional
model, which will then control the design under test via its bus interface. A
device model would then mimic the device the design was intended to talk to.
When done well, the test script would evaluate the values returned by the
design–after interacting with the device, and declare “success” or “failure”.</p>
<p>Here’s the key to this setup: I can run many different tests from this starting
point by simply changing the test script and nothing else.</p>
<p>For example, let’s imagine an external memory controller. A “good” memory
controller should be able to accept any bus request, convert it into
I/O wires to interact with the external memory, and then return a response from
the memory. Hence, it should be possible to first write to the external memory
and then (later) read from the same external memory. Whatever is then read
should match what was written previously. This is the minimum test
case–measuring the “contract” with the memory.</p>
<p>Other test cases might evaluate this contract across all of the modes the
memory supports. Still other cases might attempt to trigger all of the faults
the design is supposed to be able to handle. The only difference between these
many test cases would then be their test scripts. Again, you can measure
whether or not the test cases are sufficient using coverage measures.</p>
<p>The key here is that all of the test cases must produce either a “pass” or
“fail” result. That is, they must be self-checking. Now, using self checking
test cases, I can verify (via simulation) something like the
<a href="/about/zipcpu.html">ZipCPU</a> across all of its instructions,
in SMP and single CPU environments, using the DMA (or not), and so forth.
Indeed, the <a href="/about/zipcpu.html">ZipCPU</a>’s test environment
takes this approach one step farther, by not just changing the test script
(in this case a <a href="/about/zipcpu.html">ZipCPU</a> software program)
but also the configuration of the test environment as well. This allows me
to make sure the <a href="/about/zipcpu.html">ZipCPU</a> will continue
to work in 32b, 64b, or even wider bus environments in a single test suite.</p>
<p>Yes, this was a problem I was having before I adopted this methodology: I’d
test the <a href="/about/zipcpu.html">ZipCPU</a> with a 32b bus, and then
deploy the <a href="/about/zipcpu.html">ZipCPU</a> to a board whose
memory was 64b wide or wider. The <a href="https://github.com/ZipCPU/kimos">Kimos
project</a>, for example, has a 512b bus. Now
that I run test cases on multiple bus widths, I have the confidence that I
can easily adjust the <a href="/about/zipcpu.html">ZipCPU</a> from one
bus width to another.</p>
<p>This is now as far as I’ve now come in my verification journey. I now use
formal tests, simulation tests, coverage checking, and a self-checking test
suite on new design components. Is this perfect? No, but at least its more
rigorous and repeatable than where I started from.</p>
<h2 id="next-steps-softwarehardware-interaction">Next Steps: Software/Hardware interaction</h2>
<p>The testing regiment discussed above continues to have a very large and
significant hole: I can’t test software drivers very well.</p>
<p>Consider as an example my <a href="https://github.com/ZipCPU/sdsdpi">SD card
controller</a>. The
<a href="https://github.com/ZipCPU/sdsdpi">repository</a> actually contains three
controllers: <a href="https://github.com/ZipCPU/sdspi/blob/master/rtl/sdspi.v">one for interacting with SD cards via their SPI
interface</a>, <a href="https://github.com/ZipCPU/sdspi/blob/master/rtl/sdio_top.v">one via
the SDIO interface</a>,
and a third for use with eMMC cards (<a href="https://github.com/ZipCPU/sdspi/blob/master/rtl/sdio_top.v">using the SDIO
interface</a>).
The <a href="https://github.com/ZipCPU/sdsdpi">repository</a> contains formal proofs
for all leaf modules, and two types of SD card models–a <a href="https://github.com/ZipCPU/blob/master/bench/cpp/sdspi.cpp">C++ model for
SPI</a> and all Verilog
models for
<a href="https://github.com/ZipCPU/blob/master/sdspi/bench/verilog/mdl_sdio.v">SDIO</a> and
<a href="https://github.com/ZipCPU/sdspi/blob/master/bench/verilog/mdl_emmc.v">eMMC</a>.</p>
<p>This controller IP also contains a set of <a href="https://github.com/ZipCPU/sdspi/tree/master/sw">software
drivers</a> for use when working
with SD cards. Ideally, these drivers should be tested together with the
<a href="https://github.com/ZipCPU/sdsdpi">SD card controller(s)</a>, so they could be
verified together.</p>
<p>Recently, for example, I added a <a href="https://github.com/ZipCPU/sdspi/blob/master/rtl/sddma.v">DMA
capability</a> to the
<a href="/zipcpu/2017/11/07/wb-formal.html">Wishbone</a>
version of <a href="https://github.com/ZipCPU/sdspi/blob/master/rtl/sdio.v">the SDIO (and eMMC)
controller(s)</a>. This
(new) <a href="https://github.com/ZipCPU/sdspi/blob/master/rtl/sddma.v">DMA
capability</a>
then necessitated quite a few changes to the
<a href="https://github.com/ZipCPU/sdspi/tree/master/sw">control software</a>, so that it
could take advantage of it. With no tests, how well do you think
<a href="https://github.com/ZipCPU/sdspi/blob/master/sw/sdiodrv.c">this software</a>
worked when I first tested it in hardware?</p>
<p>It didn’t.</p>
<p>So, for now, the <a href="https://github.com/ZipCPU/sdspi/tree/master/sw">software
directory</a> simply holds the
software I will copy to other designs and test in actual hardware.</p>
<table align="center" style="float: right"><caption>Fig 2. Software driven test bench</caption><tr><td><img src="/img/cpusim/softwaretb.svg" width="320" /></td></tr></table>
<p>The problem is, testing the <a href="https://github.com/ZipCPU/sdspi/tree/master/sw">software
directory</a> requires many
design components beyond just the
<a href="https://github.com/ZipCPU/sdspi">SD card controllers</a> that would be under test.
It requires memory, a console port, a CPU, and the CPU’s tool chain–all in
addition to the <a href="https://github.com/ZipCPU/sdspi">design</a> under test.
These extra components aren’t a part of the <a href="https://github.com/ZipCPU/sdspi">SD controller
repository</a>, nor perhaps should they be. How
then should these <a href="https://github.com/ZipCPU/sdspi/tree/master/sw">software
drivers</a> be tested?</p>
<p>Necessity breeds invention, so I’m sure I’ll eventually solve this problem.
This is just as far as I’ve gotten so far.</p>
<h2 id="automated-testing">Automated testing</h2>
<p>At any rate, I submitted this
<a href="https://github.com/ZipCPU/sdspi">repository</a> to an automated continuous
integration facility the team I was working with was testing. The utility
leans heavily on the existence of a variety of <code class="language-plaintext highlighter-rouge">make test</code> capabilities within
the <a href="https://github.com/ZipCPU/sdspi">repository</a>, and so the
<a href="https://github.com/ZipCPU/sdspi">SD Card repository</a> was a good fit for
testing. Along the way, I needed some help from the test facility engineer to
get <a href="https://github.com/YosysHQ/sby">SymbiYosys</a>,
<a href="https://github.com/steveicarus/iverilog">IVerilog</a> and
<a href="https://www.veripool.org/verilator/">Verilator</a> capabilities installed. His
response?</p>
<blockquote>
<p>It’s literally the first time I get to know a good hardware project needs
such many verifications and testings! There’s even a real SD card
simulation model and RW test…</p>
</blockquote>
<p>Yeah. Actually, there’s three SD card models–as discussed above. It’s been
a long road to get to this point, and I’ve certainly learned a lot along the
way.</p>
<hr /><p><em>Watch therefore: for ye know not what hour your Lord doth come. (Matt 24:42)</em></description>
<pubDate>Sat, 06 Jul 2024 00:00:00 -0400</pubDate>
<link>https://zipcpu.com/formal/2024/07/06/verifjourney.html</link>
<guid isPermaLink="true">https://zipcpu.com/formal/2024/07/06/verifjourney.html</guid>
<category>formal</category>
</item>
<item>
<title>Debugging video from across the ocean</title>
<description><p>I’ve come across two approaches to video synchronization. The first, used by
a lot of the Xilinx IP I’ve come across, is to hold the video pipeline in
reset until everything is ready and then release the resets (in the right and
proper order) to get the design started. If something goes wrong, however,
there’s no room for recovery. The second approach is the approach I like to
use, which is to <a href="/video/2022/03/14/axis-video.html">build video components that are inherently
“stable”</a>: 1) if they
ever lose synchronization, they will naturally work their way back into
synchronization, and 2) once synchronized they will not get out of sync.</p>
<p>At least that’s the goal. It’s a great goal, too–when it works.</p>
<p>Today’s story is about what happens when a “robust” video display isn’t.</p>
<h2 id="system-overview">System Overview</h2>
<p>Let’s start at the top level: I’m working on building a SONAR device.</p>
<p>This device will be placed in the water, and it will sample acoustic data.
All of the electronics will be contained within a pressure chamber, with
the only interface to the outside world being a single cable providing both
Ethernet and power.</p>
<p>Here’s the picture I used to capture this idea when <a href="/blog/2022/08/24/protocol-design.html">we discussed the network
protocols that would be required to debug this
device</a>.</p>
<table align="center" style="float: none"><caption>Fig 1. Controlling an Underwater FPGA</caption><tr><td><img src="/img/netbus/sysdesign.svg" alt="" width="780" /></td></tr></table>
<p>This “wet” device will then connect to a “dry” device (kept on land, via
Ethernet) where the sampled data can then be read, stored and processed.</p>
<p>Now into today’s detail: while my customer has provided no requirement for
real-time processing, there’s arguably a need for it during development testing.
Even if there’s no need for real-time processing in the final delivery, there’s
arguably a need for it in the lab leading up to that final delivery. That is,
I’d like to be able to just glance at my lab setup and know (at a glance or
two) that things are working. For this reason, I’d like some real time
displays that I can read, at a glance, and know that things are working.</p>
<p>So, what do we have available to us to get us closer?</p>
<h2 id="display-architecture">Display Architecture</h2>
<p>Some time ago, I built several RTL “display” modules to use for this
lab-testing purpose. In general, these modules take an <a href="/blog/2022/02/23/axis-abort.html">AXI stream of incoming
data</a>,
and they produce an <a href="/video/2022/03/14/axis-video.html">AXI video stream for
display</a>. At present,
there are only five of these graphics display modules:</p>
<ul>
<li>
<p><a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_histogram.v">A histogram display</a></p>
<p><a href="/dsp/2019/12/21/histogram.html">Histograms are exceptionally useful for diagnosing any A/D collection
issues</a>, so having a live
<a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_histogram.v">histogram display</a>
to provide insight into the sampled data distribution just makes sense.</p>
<p>However, <a href="/dsp/2019/12/21/histogram.html">histogram</a>
<a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_histogram.v">displays</a>
need a tremendous dynamic range. How do you handle that in hardware? Yeah,
that was part of the challenge when building this display. It involved
figuring out how to build multiplies and divides without doing either
multiplication or division. A fun project, though.</p>
</li>
<li>
<p><a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_trace.v">A trace module</a></p>
<p>By “trace”, I mean something to show the time series, such as a plot of
voltage against time. My big challenge with this display so far has been
the reality that the SONAR A/D chips can produce more data than they eye can
quickly process.</p>
<p>Now that we’ve been through a test or two with the hardware, I have a better
idea of what would be valuable here. As a result, I’m likely going to take
the absolute value of voltages across a significant fraction of a second,
and then use that approach to display a couple of seconds worth of data on
the screen. Thankfully, my <a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_trace.v">trace display
module</a> is
quite flexible, and should be able to display anything you give to it by way
of an AXI Stream input.</p>
</li>
<li>
<p><a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_waterfall.v">A falling raster</a></p>
<p>The very first time my wife came to a family day at the office, way back
in the 1995-96 time frame or so, the office had a display set up with a
microphone and a sliding spectral raster. I was in awe! You could speak,
and see what your voice “looked” like spectrally over time. You could hit
the table, whistle, bark, whatever, and every sound you made would look
different.</p>
<p>I’ve since <a href="https://github.com/ZipCPU/fftdemo">built this kind of capability</a>
many times over, and even <a href="/dsp/2020/11/21/spectrogram.html">studied the best ways to do it from a
mathematical standpoint</a>.</p>
<p>In the SONAR world, you’ll find this sort of thing really helps you visualize
what’s going on in your data streams–what sounds are your sensors picking
up, what frequencies are they at, etc. A good raster will let you “see”
motors in the water–all very valuable.</p>
</li>
<li>
<p>A spectrogram, via the same <a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_trace.v">trace
module</a></p>
<p>This primarily involves plotting the absolute values of the data coming out
of an <a href="/dsp/2018/10/02/fft.html">FFT</a>,
applied to the incoming data. Thankfully, the <a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_trace.v">trace
module</a>
is robust enough to handle this kind of input as well.</p>
</li>
<li>
<p><a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_split.v">A split screen display</a>,
that can place both an <a href="/dsp/2018/10/02/fft.html">FFT</a>
<a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_trace.v">trace</a>
and a falling raster on the same screen.</p>
</li>
</ul>
<p>We’ll come back to the split screen display in a bit. In general, however,
the processing components used within it look (roughly) like Fig. 2 below.</p>
<table align="center" style="float: none"><caption>Fig 2. Split display video processing pipeline</caption><tr><td><img src="/img/qoi-debug/split-pipeline.svg" alt="" width="780" /></td></tr></table>
<p>Making this happen required some other behind the scenes components as well,
to include:</p>
<ul>
<li>
<p><a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_empty.v">An empty video generator</a>–to
generate an <a href="/video/2022/03/14/axis-video.html">AXI video
stream</a> from scratch.
The video out of this device is a constant color (typically black). This
then forms a “canvas” (via the <a href="/video/2022/03/14/axis-video.html">AXI video
stream protocol</a>)
that other things can be overlaid on top of.</p>
<p>This generator leaves <code class="language-plaintext highlighter-rouge">TVALID</code> high, for reasons we’ve
<a href="/video/2022/03/14/axis-video.html">discussed before</a>,
and that we’ll get to again in a moment.</p>
</li>
<li>
<p><a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_mux.v">A video multiplexer</a>–to
select between one of the various “displays”, and send only one to the
outgoing video display.</p>
<p>One of the things newcomers to the hardware world often don’t realize is that
the hardware used for a display can often not be reused when you switch
display types. This is sort of like an ALU–the CPU will include support
for ADD, OR, XOR, and AND instructions, even if only one of the results is
selected on each clock cycle. The same is true here. Each of the various
displays listed
above is built in hardware, occupies a separate area of the FPGA (whether used
or not), and so something is needed to select between the various outputs to
choose which we’d like.</p>
<p>It did take some thought to figure out how to maintaining video
synchronization while multiplexing multiple video streams together.</p>
</li>
<li>
<p><a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/axisvoverlay.v">A video overlay module</a>–to merge two displays together, creating a result that
looks like it has multiple independent “windows” all displaying real time
data.</p>
</li>
</ul>
<p>I wrote these modules years ago. They’ve all worked beautifully–in simulation.
So far, these have only been designed to be engineering displays, and not
necessarily great finished products. Their biggest design problem? None of
them display any units. Still, they promise a valuable debugging
capability–provided they work.</p>
<p>Herein lies the rub. Although these display modules have worked nicely in
simulation, and although many have been formally verified, for some reason
I’ve had troubles with these modules when placed into actual hardware.</p>
<p>Debugging this video chain is the topic of today’s discussion.</p>
<h2 id="axi-video-rules">AXI Video Rules</h2>
<p>For some more background, each of these modules produces an AXI video stream.
In general, these components would take data input, and produce a video
stream as output–much like Fig. 3 below.</p>
<table align="center" style="float: right"><caption>Fig 3. General AXI Stream Video component</caption><tr><td><img src="/img/qoi-debug/gendisplay.svg" alt="" width="420" /></td></tr></table>
<p>In this figure, acoustic data arrives on the left, and video data comes out on
the right. Both use AXI streams.</p>
<p>The <a href="/video/2022/03/14/axis-video.html">AXI stream protocol, however, isn’t necessarily a good fit for video
proccessing</a>.
You really have to be aware of who drives the pixel clock,
and where the blanking intervals in your design are handled.</p>
<ul>
<li>
<p>Sink</p>
<p>If video comes into your device, the pixel clock is driven by that video
source. The source will also determine when blanking intervals need to
take place and how long they should be. This will be controlled via the
video’s <code class="language-plaintext highlighter-rouge">VALID</code> signal.</p>
</li>
<li>
<p>Source</p>
<p>Otherwise, if you are not consuming incoming video but producing video out,
then the pixel clock and blanking intervals will be driven by the video
controller. This will be controlled by the display controllers <code class="language-plaintext highlighter-rouge">READY</code>
signal.</p>
</li>
</ul>
<p>In our case, these intermediate display modules also need to be aware that
there’s often <em>no</em> buffering for the input. If you drop the <code class="language-plaintext highlighter-rouge">SRC_READY</code> line,
data will be lost. Acoustic sensor data is coming at the design whether you
are ready for it or not. Likewise, the <a href="/blog/2022/02/23/axis-abort.html">video output data needs to get to the
display module, and there’s no room in the HDMI standard for <code class="language-plaintext highlighter-rouge">VALID</code> dropping
when a pixel needs to be
produced</a>.</p>
<p>Put simply, there are two constraints to these controllers: 1) the source can’t
handle <code class="language-plaintext highlighter-rouge">VALID &amp;&amp; !READY</code>, and 2) the display controller at the end of the video
processing chain can’t handle <code class="language-plaintext highlighter-rouge">READY &amp;&amp; !VALID</code>. Any IP in the middle needs
to do what it can to avoid these conditions.</p>
<p>This leads to some self-imposed criteria, that I’ve “added” to the AXI stream
protocol. Here are my extra rules for processing AXI video stream data:</p>
<ol>
<li>
<p>All video processing components should keep READY high.</p>
<p>Specifically, nothing <em>within</em>
the module should ever drop the ready signal. Only the downstream display
driver should ever drop READY by more than a cycle or two between lines.
This drop in READY then needs to propagate through all the way through any
video processing chain.</p>
<p>My <a href="https://github.com/ZipCPU/vgasim/blob/master/rtl/gfx/vid_mux.v">video multiplexer</a>
module is an example of an exception to this rule: It drops READY on all
of the video streams that aren’t currently active. By waiting until the
end of a frame before adjusting/swapping which source is active, it can keep
all sources synchronized with the output. This component will fail,
however, if one of those incoming streams is a true video source.</p>
</li>
<li>