Merge pull request YosysHQ#6 from daveshah1/libtrellis-db

Code to manage and manipulate database

.github/autolabeler.yml

@@ -6,5 +6,6 @@ experiments: ["experiments"]
 docs: ["docs", "*.md", "COPYING"]
 infra: [".travis", ".travis.yml", ".github", "Docker*", ".gcloud*", ".style.yapf", "vagrant", "cloudbuild.*", "download-latest-db.sh"]
 tools: ["tools", "utils", "htmlgen"]
+libtrellis: ["libtrellis"]
 
 third-party: ["third_party", ".gitmodules"]

.update-contributing.py

@@ -2,7 +2,7 @@
 
 # Header
 contrib = ["""\
-# Contributing to Project X-Ray
+# Contributing to Project Trellis
 """]
 
 # Extract the "Contributing" section from README.md

CONTRIBUTING.md

@@ -1,4 +1,4 @@
-# Contributing to Project X-Ray
+# Contributing to Project Trellis
 
 There are a couple of guidelines when contributing to Project Trellis which are
 listed here.
@@ -10,7 +10,7 @@ All contributions should be sent as
 
 ### License
 
-All code in the Project X-Ray repository is licensed under the very permissive
+All code in the Project Trellis repository is licensed under the very permissive
 [ISC Licence](COPYING). A copy can be found in the [`COPYING`](COPYING) file.
 
 All new contributions must also be released under this license.

README.md

@@ -8,19 +8,22 @@ bit-stream format of Lattice ECP5 series FPGAs.
 More documentation can be found published on
 [prjtrellis ReadTheDocs site](http://prjtrellis.readthedocs.io/en/latest/) -
 this includes;
- * FIXME(@tinyfpga): Make these documents.
  * [Highlevel Bitstream Architecture](http://prjtrellis.readthedocs.io/en/latest/architecture/overview.html)
  * [Overview of DB Development Process](http://prjrellis.readthedocs.io/en/latest/db_dev_process/overview.html)
+ * [libtrellis Documentation](http://prjrellis.readthedocs.io/en/latest/libtrellis/overview.html)
 
 This follows follows the lead of
 [Project X-Ray](https://github.com/SymbiFlow/prjxray) - which is documenting
 the bitstream format for the Xilinx Series 7 devices.
 
 # Quickstart Guide
 
-Install XXXXX
-    FIXME(@tinyfpga): Put instructions here.
-
+Install the dependencies:
+ - Lattice Diamond 3.10
+ - Python 3.5 or later
+ - A modern C++14 compiler (g++ or Clang recommended)
+ - Boost
+
 Pull submodules:
 
     git submodule update --init --recursive
@@ -33,15 +36,19 @@ Get a head start by downloading current database:
     # - git if you want to use the git protocol
     ./download-latest-db.sh
 
-Always make sure to set the environment for the device you are working on before
-running any other commands:
+For a generic environment:
+
+    source environment.sh
+
+TODO: device specific environments, like X-ray
 
-    source database/XXXXXX/settings.sh
+Build libtrellis:
 
-Creating HTML documentation:
+    cd libtrellis
+    cmake .
+    make
 
-    cd htmlgen
-    python3 htmlgen.py
+Once fuzzers are written, the below will also work:
 
 (Re-)creating the database:
 
@@ -55,7 +62,7 @@ Creating HTML documentation:
 
 # Process
 
-The documentation is done through a "black box" process were XXXXX is asked to
+The documentation is done through a "black box" process were Diamond is asked to
 generate a large number of designs which then used to create bitstreams. The
 resulting bit streams are then cross correlated to discover what different bits
 do.
@@ -103,8 +110,13 @@ only be used inside this repository.
 
 ### [Third Party](third_party)
 
-Third party contains code not developed as part of Project X-Ray.
+Third party contains code not developed as part of Project Trellis.
+
+### [libtrellis](libtrellis)
 
+libtrellis is a library for manipulating ECP5 bitstreams, tiles and the Project
+Trellis databases. It is written with C++, with Python bindings exposed using
+Boost::Python so that fuzzers and utilities can be written in Python.
 
 # Database
 
@@ -121,7 +133,13 @@ but we can not do this alone, **we need your help**!
 
 ## TODO List
 
- - [ ] Write a TODO list
+ - [ ] Write fuzzing framework for configuration bit and routing fuzzing
+ - [ ] Fuzz logic tile init and config bits
+ - [ ] Fuzz logic tile routing
+ - [ ] Fuzz other routing tiles (CIBs)
+ - [ ] Fuzz IO tiles
+ - [ ] Fuzz global clock tiles
+ - [ ] Fuzz other function tiles (EBR, DSP, SERDES, etc)
 
 
 # Contributing
@@ -136,7 +154,7 @@ All contributions should be sent as
 
 ### License
 
-All code in the Project X-Ray repository is licensed under the very permissive
+All code in the Project Trellis repository is licensed under the very permissive
 [ISC Licence](COPYING). A copy can be found in the [`COPYING`](COPYING) file.
 
 All new contributions must also be released under this license.

docs/architecture/glossary.rst

@@ -90,6 +90,10 @@ Glossary
     hardware logic elements on the :term:`FPGA`, and then routing the signals
     between the placed elements. 
 
+  Quadrant
+    The ECP5 FPGAs are arranged into four quadrants for the purpose of global signal
+    distribution, upper left (UL), upper right (UR), lower left (LL) and lower right (LR).
+
   Routing fabric
     The :term:`wires <wire>` and programmable interconnects (:term:`arcs <arc>`)
     connecting the logic blocks in an :term:`FPGA`.

docs/architecture/overview.rst

@@ -3,3 +3,9 @@
 Overview
 ========
 
+The ECP5 FPGA is arranged internally as a grid of :doc:`Tiles <tiles>`. Each tile contains bits that configure routing
+and/or the tile's functionality.
+
+Inside the ECP5 there is both :doc:`general routing <general_routing>`, which connects nearby tiles together
+(spanning up to 12 tiles) and :doc:`global routing <global_routing>`, which allows high fanout signals to connect to all
+tiles within a :term:`quadrant` (such as clocks).

docs/db_dev_process/overview.rst

@@ -0,0 +1,59 @@
+Database Development Overview
+=============================
+
+A targeted approach is used to construct bitstream databases as quickly as possible. The planned flow is for every
+routing mux or configuration setting we want to determine, to create post-place-and-route designs for all possibilities
+then run them through bitstream generation and compare the outputs.
+
+NCL Files
+----------
+NCL files are file format used by Lattice (originating from NeoCAD), which are a textual representation of the internal
+design database. Although there is little documentation on them, the format is relatively straightforward. There are
+tools included with Diamond to convert the design database to and from a NCL file (``ncd2ncl`` and ``ncl2ncd``). These
+are wrapped by the script ``diamond.sh`` included in Project Trellis, that allows two possibilities:
+
+ - If given a Verilog file, it will use Diamond for synthesis and PAR, and as well as a bitstream also dump the
+   post-place-and-route design as a NCL file. This way you can inspect how the design maps to an NCL file, and the
+   routing and configuration inside the tile.
+ - If given a NCL file, it will skip synthesis and PAR. It will convert the NCL file to a design database, then
+   generate a bitstream from that.
+
+In the planned fuzzing flow, we will first create a Verilog design for what we want to fuzz by hand, and convert it to
+an NCL file. Then we will manually create a template NCL file containing only the mux/config to be fuzzed. The Python
+fuzzer script will then subsitute this template file for each fuzz possibility.
+
+A template file for LUT initialisation is shown as an example:
+
+.. code-block:: none
+
+    ::FROM-WRITER;
+    design top
+    {
+       device
+       {
+          architecture sa5p00;
+          device LFE5U-25F;
+          package CABGA381;
+          performance "8";
+       }
+
+       comp SLICE_0
+          [,,,,A0,B0,D0,C0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,]
+       {
+          logical
+          {
+             cellmodel-name SLICE;
+             program "MODE:LOGIC "
+                     "K0::H0=${lut_func} "
+                     "F0:F ";
+             primitive K0 i3_4_lut;
+          }
+          site R2C2A;
+       }
+
+    }
+
+The NCL file contains information about the device, components and routing (routing is not included in this example). As
+this was from a LUT initialisation fuzzer, ${lut_func} will be replaced by a function corresponding to the LUT init bit
+to be fuzzed (NCL files require an expression for LUT initialisation, rather than a series of bits).
+

docs/index.rst

@@ -20,3 +20,14 @@ to develop a free and open Verilog to bitstream toolchain for these devices.
    architecture/bitstream_format
    architecture/glossary
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Database Development Process
+
+   db_dev_process/overview
+
+.. toctree::
+   :maxdepth: 1
+   :caption: libtrellis Documenation
+
+   libtrellis/overview

docs/libtrellis/overview.rst

@@ -0,0 +1,71 @@
+libtrellis Overview
+====================
+
+libtrellis is a C++ library containing utilities to manipulate ECP5 bitstreams, and the databases that correspond tile
+bits to functionality (routing and configuration). Although libtrellis can be used as a standard shared library, its
+primary use in Project Trellis is as a Python module (called pytrellis), imported by the fuzzing and utility scripts.
+The C++ classes are bound to Python using Boost::Python.
+
+Bitstream
+---------
+This class provides functionality to read and write Lattice bitstream files, parse their commands, and convert them
+into a chip's configuration memory (in terms of frames and bits).
+
+To read a bitstream, use ``read_bit`` to create a ``Bitstream`` object, then call ``deserialise_chip`` on that to
+create a ``Chip``.
+
+Chip
+-----
+This represents a configured FPGA, in terms of its configuration memory (``CRAM``), tiles and metadata. You can either
+use ``deserialise_chip`` on a bitstream to construct a Chip from an existing bitstream, or construct a Chip by device
+name or IDCODE.
+
+The ``ChipInfo`` structure contains information for a particular FPGA device.
+
+CRAM
+-----
+This class stores the entire configuration data of the FPGA, as a 2D array (frames and bits). Although the array can be
+accessed directly, many cases will use ``CRAMView`` instead. ``CRAMView`` provides a read/write view of a window of the
+CRAM. This is usually used to represent the configuration memory of a single tile, and takes frame and bit offsets
+and lengths.
+
+Subtracting two ``CRAMView`` s, if they are the same size, will produce a ``CRAMDelta``, a list of the changes between
+the two memories. This is useful for fuzzing or comparing bitstreams.
+
+Tile
+-----
+This represents a tile of the FPGA. It includes a ``CRAMView`` to represent the configuration memory of the tile.
+
+TileConfig
+-----------
+This represents the actual configuration of a tile, in terms of arcs (programmable connections), config words (such as
+LUT initialisation) and config enums (such as IO type). It is the result of decoding the tile CRAM content using the bit
+database, and can be converted to a FASM-like format.
+
+The contents of ``TileConfig`` are ``ConfigArc`` for connections, ``ConfigWord`` for non-routing configuration words
+(which also includes single config bits), ``ConfigEnum`` for enum configurations with multiple textual values, and
+``ConfigUnknown`` for unknown bits not found in the database, which are simply stored as a frame, bit reference.
+
+The contents of a tile's configuration RAM can be converted to and from a ``TileConfig`` by using the ``tile_cram_to_config``
+and ``config_to_tile_cram`` methods on the ``TileBitDatabase`` instance for the tile.
+
+TileBitDatabase
+----------------
+There will always be only one ``TileBitDatabase`` for each tile type, which is enforced by requiring calling the
+function ``get_tile_bitdata`` (in ``Database.cpp``) to obtain a ``shared_ptr`` to the ``TileBitDatabase``.
+
+The ``TileBitDatabase`` stored the function of all bits in the tile, in terms of the following constructs:
+
+ - Muxes (``MuxBits``) specify a list of arcs that can drive a given node. Each arc (``ArcData``) contains
+   specifies source, sink and the list of bits that enable it as a ``BitGroup``.
+ - Config words (``WordSettingBits``) specify non-routing configuration settings that are arranged as one or more bits.
+   Each config bit has an associated list of bits that enable it. This would be used both for single-bit settings
+   and configuration such as LUT initialisation and PLL dividers.
+ - Config enums (``EnumSettingBits``) specify non-routing configuration settings that have a set of possible textual
+   values, used for either modes/types (i.e. IO type) or occasionally "special" muxes not part of general routing. These
+   are specified as a map between possible values and the bits that enable those values.
+
+``TileBitDatabase`` instances can be modified during runtime, in a thread-safe way, to enable parallel fuzzing. They can
+be saved back to disk using the ``save`` method.
+
+They can also be used to convert between tile CRAM data and higher level tile config, as described above.

experiments/interconnect_poc/fuzz_single_mux.py

@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+
+import os
+from os import path
+import shutil
+
+import diamond
+from string import Template
+import pytrellis
+
+device = "LFE5U-25F"
+sink = "R2C2_A0"
+# Drivers found using ispTcl
+drivers = [
+    "R1C2_V02S0501",
+    "R2C2_H02W0501",
+    "R2C2_H01E0001",
+    "R2C3_H02W0701",
+    "R2C3_H02W0501",
+    "R2C2_H02W0701",
+    "R2C2_H02E0501",
+    "R3C2_V02N0501",
+    "R1C2_V02S0701",
+    "R2C2_F5",
+    "R2C2_H00L0000",
+    "R2C2_F7",
+    "R2C2_H02E0701",
+    "R2C2_V02N0701",
+    "R2C1_H02E0701",
+    "R2C3_H01E0001",
+    "R2C2_V02S0501",
+    "R3C2_V02N0701",
+    "R2C2_V02S0701",
+    "R2C2_V01N0101",
+    "R2C2_V02N0501",
+    "R2C1_H02E0501",
+    "R2C2_H00L0100",
+    "R2C2_H00R0000"
+]
+
+
+def run_get_bits(mux_driver):
+    route = ""
+    if mux_driver != "":
+        route = "route\n\t\t\t" + mux_driver + "." + sink + ";"
+    with open("mux_template.ncl", "r") as inf:
+        with open("work/mux.ncl", "w") as ouf:
+            ouf.write(Template(inf.read()).substitute(route=route))
+    diamond.run(device, "work/mux.ncl")
+    bs = pytrellis.Bitstream.read_bit("work/mux.bit")
+    chip = bs.deserialise_chip()
+    tile = chip.tiles["R2C2:PLC2"]
+    return tile.cram
+
+
+def main():
+    pytrellis.load_database("../../../prjtrellis-db")
+    shutil.rmtree("work", ignore_errors=True)
+    os.mkdir("work")
+    baseline = run_get_bits("")
+    with open("a0_mux_out.txt", "w") as f:
+        for d in drivers:
+            bits = run_get_bits(d)
+            diff = bits - baseline
+            diff_str = ["{}F{}B{}".format("!" if b.delta < 0 else "", b.frame, b.bit) for b in diff]
+            print("{0: <18}{1}".format(d, " ".join(diff_str)), file=f)
+            f.flush()
+
+
+if __name__ == "__main__":
+    main()