misoc (re)organization

[jordens]

There have been a few discussions of organizing the misoc tree. There seem to be two groups of challenges:

misoc python package namespace

Misoc could probably become a real python package. Then using misoc components from a project does not require reverse calling make.py -X ... from the misoc directory but can be handled at the project's discretion and from within the project. miscolib could also be just misoc for that purpose.

The top executable misoc (or misoc_make, f.k.a make.py) could live inside misoc. and an executable script with the correct python interpreter can be generated by the packaging infrastructure.

__init__.py should either be empty everywhere or consist purely of code that initializes/manages the namespace. The file name should do a reasonable job of describing the file's contents. Directories containing only __init__.py can always be replaced by a single .py file with the name of the directory a level up.

We could get rid of the topic groupings. Reliably grouping into com, mem, video, others, tools, and soc is difficult or impossible. Does PCI do just communication or can it also do memory-like things? Can SATA be used to do communication? Or are both tools because they sound like more or less generic busses? What is in others as compared to tools? Why is timer in cpu? Do all components that do not fit any other category but use CSRs go into cpu? Why is the memtest tool deep in misoclib.mem.sdram.frontend but LiteScopeLA in misoclib.tools.litescope.frontend.la?

Also, within a component, if there are frontends, why are there never any backends but many phys? Do the components really need directories within them? Would litescope not be very readable with just port.py, storage.py, trigger.py, io.py (or gpio.py), logic_analyzer.py, and an __init__.py that pulls them together?

Instead of

from misoclib.tools.litescope.common import *
from misoclib.tools.litescope.core.port import LiteScopeTerm
from misoclib.tools.litescope.frontend.io import LiteScopeIO
from misoclib.tools.litescope.frontend.la import LiteScopeLA

we could have from misoc import litescope and then self.submodules += litescope.LogicAnalyzer(...).

Top-level examples and targets that use several misoc components to build actual bitstreams should probably not be living deep in the misoc namespace (misoclib.tools.litescope.example_designs.targets.simple) but either at misoc.targets.litescope (maybe add _simple, no deeper hierarchy) if they can be reused or in /examples outside the misoc package namespace. And they could do something useful if executed (i.e. run/build themselves).

Having non-gateware python code like host tools, data format codecs etc in the various components does not seem to be too confusing and rhymes well with the metaprogramming style of migen. Could the contents of misoclib.tools.litescope.software.driver.la not be in the same file (or at least in the same directory) as misoclib.tools.litescope.frontend.la?

It seems to be a standard pattern to locate unittests (probably not testbenches where you have to look at a vcd afterwards) within test/ next to the module they test. nose and others can find them.

Making it a real python package suggests moving non-python contents out of misoclib/ somewhere else.

Non-python components

Documentation that currently lives deep inside misoclib could be moved to /doc.

/tools/* could be moved into misoc if they are python scripts and executables with the correct interpreter can be generated automatically using the packaging infrastructure.

/extcores/* items could be located and handled using the packaging infrastructure (https://pythonhosted.org/setuptools/setuptools.html#including-data-files).

/software/* items could support building out-of-tree. That solves the pitfalls of having to rebuild bios, library, and headers and potentially flashing the wrong one when switching between projects using misoc. E.g. https://github.com/libopencm3/libopencm3 seems to do that nicely for very variable configurations.

[enjoy-digital]

I agree that MiSoC should become a real python package. I also agree it's sometimes difficult to group things in topics. When it's ambiguous I don't mind trying to flatten things. But for example:

• LitePCIe is clearly a communication core. You will maybe access a memory through it but then it's a use case for this core and it does not preclude that LitePCIe is still a communication core. Same for LiteUSB, LiteEth.
• LiteSATA is also clearly a memory core. If lower layers were not specific, I totally agree they should live in communication. But here everything is specific to this application:
  • You are not going to reuse the transceiver code since it has more than 300 parameters that have to be configured specifically for SATA application.
  • Control of the transceiver is also very specific to SATA applications (OOB, speed negociation). Upper layers are also specific the SATA specification (Link, Transport, Command layers)

So there is no interest in reusing anything from LiteSATA for high speed serial communications. It can be interesting in the future to develop a generic low level core to create a high speed serial communication link between 2 FPGA, but even in that case you are not going to reuse that for LiteSATA.

So I'd like to keep "big" cores as self contained and organized in topics (maybe outside of MiSoC it it prevents packaging MiSoC as you want to do) For small cores that do not belong to existing topics, we can try to flatten things (soc, cpu, tools).

About the cores organization:

• phys are the hardware equivalent to software backends.
• core implements the main fonctionnality or the core.
• frontend provides modules on top of the core that are used by the user (DMAs, bridges, crossbar, test modules such memtest or a bist, ...).

About moving software, examples_designs, doc outside of cores, from this:

liteeth

• core
• doc
• examples_designs
• frontend
• phy
• software
• test

litepcie

• core
• doc
• examples_designs
• frontend
• phy
• software
• test

liteusb

• core
• doc
• examples_designs
• frontend
• phy
• software
• test

I have the impression we will endup with something like this:

doc

• liteeth
• litepcie
• liteusb

software

• liteeth
• litepcie
• liteusb

example_designs

• liteeth
• litepcie
• liteusb

tests

• liteeth
• litepcie
• liteusb

Except maybe for packaging MiSoC, I don't see real advantages for that compared to the actual organization and I'm not sure I want to split the big cores like that.

So we have to find the best compromise for Migen and MiSoC in the globality.
As we already discussed we should move all bus, csr generation, complex modules from Migen to MiSoC. And maybe move "big" cores outside of MiSoC (as I initially started to do) if having them self containted is not compatible with what you want to do.

We can then have:
In Migen: All that is needed to generate HDL and build a design.
In MiSoC: All that is needed to generate a SoC and the basic cores/tools for that.
And the "big" cores outside of that.

Or finally maybe something that would be in fact easier for everyone:
Only one global repository (and give it a name) with subdirectories:

• migen
• mibuild
• miplatforms (For me mibuild shoud not provides platforms directly).
• misoc
• micores

And package each of this indidually to allow that:

from misoc import soc
from miplaforms import kc705
from micores imports litepcie

What do you think of that?

(My answer is in fact more a reflexion that has evolved since I was writing...)

Florent

[jordens]

Fine with me. If it would be in one big repo I fail to see how exporting five top-level packages is helpful (nobody else in the python ecosystem does it that way AFAICT) but that is secondary. IMHO splitting into repositories only becomes useful if there is a release of one, if the dependencies become inconvenient, or if the repositories become huge. So far we tend to demand that all of mi* is in sync anyway.

I grant you the labeling of pcie as communication and sata as memory. But is that the dominant and obvious category? What about serial busses versus parallel? Where does an 8b10b codec go? Do I understand you correctly that you are OK with dropping the topics? Otherwise I would have asked you to weigh the advantages of having allegedly related components side-by-side against the confusion of misplacing cores and having related cores far apart. I would claim that somebody looking for a sata core will look -- well -- for a sata core and not be inclined start browsing memory cores (spiflash, norflash those are not orthogonaly as well). There would still be a grouping of the cores by tags (or topics) in the documentation/README. Then you can relate cores side by side by different metrics much better.

The same reasoning for/against works in the case of frontend/phy/core/... directories. It is really hard to slot stuff in there and much harder to predict where you will find what you are looking for and what you will find if you peek into a directory. The need for a directory should be completely obvious. Splitting something into frontend/core/phy is usually not reliable and future proof as in the case of the UART where we now have a frontend/core that tries to adapt itself to idiosyncrasies originating in the phys (synchronous versus asynchronous, non-full versus almost-empty). It is probably only reliable if the interface between the two is defined/standardized.

If we get the documentation (there is very little, and stub documentation files that bounce a search for information with a request to contribute the very same are not helpful) into one common tree, generating common output from it becomes much cleaner. If the examples are not inside the misoc tree, it becomes much harder to litter the tree with build/ directories.

[enjoy-digital]

I'm OK to flatten things if we can keep the "big" cores in a separate directory (cores directory in misoc, or micores out of misoc).

We would have something like that without topics:
(mi)cores:

• litescope
• liteeth
• litepcie
• litesata.
• ...

And if each "big" core keep the organization it already has with its doc, example designs and so on.
But each core can have its own organization if it's more relevent. Things that are not specific to the core and that can be reused by others cores are placed into misoc. (8b10b, generic packetizers, etc...) New cores can be designed in misoc and moved in cores when they are big enough.

Also what about moving targets from misoc and then have something like that? :

• (mi)gen: Elements to generate HDL.
• (mi)build: Elements to build a design and handle vendor's tools.
• (mi)soc: Element to build a SoC. (flattened actual misoc - "big" cores + csr, bus,... from actual migen)
• (mi)cores: "big" cores.
• (mi)platforms: Supported platforms
• (mi)targets: Targets for the supported platforms.

Or eventually group platforms and targets in a same directory.

[jordens]

Sure. Once a component needs multiple files, it gets a directory. But it is an obligation to be concise and export its stuff in an organized fashion using e.g. __init__.py.

Allowing "the big cores" to cram everything into their own directory is probably not a good idea. If these cores refuse to integrate with the rest of the misoc package (documentation, executables, examples, tests, packaging) they can just as well be maintained outside misoc. Wasn't the idea to try to keep misoc from becoming a dump where everybody is king of his own directory-empire like opencores?

Moving cores around as a standard operating procedure is promising to break everybodies code.

Why is there a need to draw a line between soc and cores?

There is a good reason to keep the platforms with migen: Otherwise you don't get a useful package out of migen and you will always have to look around for platform definitions. With the platforms and mibuild in migen, you can start right away and don't have to juggle three things.

[enjoy-digital]

So it seems we both came to the conclusion that the "big" cores must live outside misoc.

Merging them in misoc was maybe not a good idea since they can in fact be seen as others project using misoc as a tool to build them or to integrate them in others design.

I'm going to create a repository with all the litexxx cores (and probably call it litecores) and then remove them from misoc. I will maybe just keep the ethernet mac if you want to avoid external dependency.

Then we'll be able to reorganize the code. Is it fine for you?

[jordens]

I would still love to see the big cores in misoc, just integrated nicely in terms of documentation, examples, tests, executables, and packaging.

But for the reorg, sure. This is a pretty big thing. How do we schedule/track it?

[enjoy-digital]

I'm sorry but no: I don't want to split these cores. If that's not compatible with the new organization, lets move them outside of misoc.

For the reorganization, we should maybe create a fork for that and then merge it in master once stabilized.

[jordens]

Hmm. That is unfortunate. What could be done is -- as you suggested -- have an area for cores that are not integrated, maybe as git submodules, and then merge their code with misoc using the namsepace-package mechanism. Then you can still from misoc import litescope even if litescope is not contained/shipped in misoc.

[enjoy-digital]

Yes that would be fine.

[enjoy-digital]

For me, the documentation, example designs, tests and hdl all together constitute the core. That's why I want to keep them together in a single directory. I know this is not what you have in mind, but let's try to find a compromise since this is the only thing I'd like to keep untouched, all your others ideas are fine for me.

I've been thinking about a simpler reorganization:

Keep migen and misoc separated but move bus and bank from migen to misoc.
Have a global organization like this:

• build
• cores
• structure
• doc
• test
• targets

Actual misoclib would be splitted in structure and cores. Things related to the misoc structure: bus, bank, soc generation, tools, make.py, cpuif.py... would go to structure. Cores would have a flattened organization without topics, something like this:

• cpu
• dvisampler
• flash
• framebuffer
• gpio.py
• spi.py
• liteeth
• liteusb
• litescope
• litepcie
• litesata
• sdram

Since actual software is in fact closely related to the cpu cores, this would be moved to cores/cpu/. We can also remove ext_cores directory and move submodules directly to the correct place: lm32/mor1kx submodules would be moved to cores/cpu/. Software would also be packaged with misoc to allow building it out of misoc directory.

The doc directory would contain the documentation for the structure and the basic cores with the tools to build it. The big cores would use the same tools for the documentation and building big cores documentation would be done from this directory. Documentation for the big cores would be generated as separated documents or integrated as chapters in a bigger documentation that will cover misoc in its globality (structure + basic cores + big cores).

The test directory would contain the tests for the structure, basic cores and we would also contain a single script that would be used to discover and run all the tests contained in cores.

The build directory would probably not be necessary if make.py is packaged with misoc.

Does it seem better for you? I'm really trying to find the best compromise for both of us :)

[jordens]

Sounds good. Ack to everything, just a few things:

• Tests can remain close to the code they test (e.g. misoc.litescope.test). There is no need for a global /test/ directory. Tools like nose can find them. You will probably like that.
• I would still merge your structure and cores (misoc.wishbone and misoc.litesata). And I would also be ok with having misoc.lm32, misoc.mor1kx, misoc.spiflash, and misoc.norflash at the top level, unless they start sharing code. The sdram stuff seems so intermingled that it probably benefits from a directory.
• /build should probably become ./build-$target (xilinx does not like concurrentl builds in the same directory) and within misoc only be used for the targets from targets, out-of-tree projects will end up containing their own build-$target
• Having the verilog code scattered seems ok to me.
• Having software scattered around is problematic and error-prone because it makes search and include paths and makefile combinations really complicated.
• If you scatter the documentation, you can not build it with the usual tools. Remember, this should be able to become a real python package.

[sbourdeauducq]

• If tests remain close to the code they test what do we do for the simple cores that reside in a single Python file? Integrate the test with that file?
• Where do we put all the other cores that we move from Migen, e.g. migen.genlib.sort, migen.genlib.mhamgen, migen.flow.*?
• I guess what Florent meant is simply that software should be in /cores/cpu. But it should be fine to put it all in a /misoc/cores/cpu/software similar to the current /software.

[enjoy-digital]

The idea with the structure and cores directories is to split things between modules/tools that are used to build a core or a SoC and the effectives cores. Structure would contain most of the generic modules we would move from Migen. ie something like this:
misoc.structure.bus.wishbone, misoc.structure.bank.description, misoc.structure.soc
and for the cores:
misoc.cores.litescope, misoc.cores.spliflash, misoc.cores.norflash16, misoc.cores.cpu.lm32

I'm not considering the modules used for plumbing, arbitrating, adapting,... as cores, so having the wishbone arbiter or the dataflow converters for example in structure seems fine to me.
I'm OK to flatten things in structure and cores but I have the feeling that whithout at least these 2 main topics, this will be a mess.

For the software, I don't want it to be scattered. As Sebastien said I would just move it to /misoc/cores/cpu/software.
Others cores can also have software (ex the linux software of litepcie, or a possible future fx2 or fx3 firmware for LiteUSB) A SoC does not necessary have an embedded cpu, so its software should be treated as it's done for the others cores.

For the test of simple cores, yes we could integrate the test in that file. (I would also like to move to misoc.structure the tools used in the testbenchs of the big cores like: packetstreamer, packetlogger, data checkers...). That would reduce the size of the integrated tests of simple cores.

I would also like to have higher level tests using verilator (or future Python simulation). For example be able to run the simple target with stubs on UART, Ethernet or others interfaces for checking. This would probably fit in misoc.test

If that's really not possible to keep the doc scattered, then I would be OK to do compromises on that.

[jordens]

• I would not hesitate to put tests for that in misoc.test.spiflash
• misoc.sort, misoc.mhamgen etc. I suspect problems and little benefit from delineating structure, cores, genlib, flow from each other reliably and in an obvious future proof way.
• Where does target support code for sata, pcie, flash etc go? Also misoc/cpu/software and not misoc/litesata/software? If litesata code gets separated from misoc/litesata, the entire thing could just as well go to /software.

[sbourdeauducq]

Robert, are you ok if we keep the tests for small cores into the same file (which would be misoc.spiflash in that case)? This way there will be no global test directory.

[sbourdeauducq]

And I guess what Florent meant is that software for the embedded CPU (BIOS, baselib, etc.) goes to misoc/cpu/software, whereas software associated with the other cores (e.g. PCIe driver for Linux) goes with each core. Is that right, Florent?

[jordens]

Re tests with cores in the same file: sure. that's nice.

About scattering the code: Sure. I would also put the software (python host-side code) in the core's directory. And it should be standard practice to hook up with the packaging infrastructure and generate executable scripts with the correct interpreter for CLI tools.

But about the c-code (bios, runtime, libbase): imagine you then have a project that uses a cpu, litesata, spiflash and litescope. It builds its software (the runtime that runs on the soc) out of tree (outside the misoc tree in its own working directory) as it should. Would'nt it be a bit of a mess to glob up all the makefile snippets for the different parts if the c-code is scattered?

[sbourdeauducq]

Would there be a significant amount of code that can be shared between a LiteSATA driver for PC and one for the embedded CPU? And same with litescope...

[jordens]

All the build infrastructure (Makefile stuff) would be shared. And litescope will probably want to use/include/link code from misoc/cpu/software.

[sbourdeauducq]

The out-of-tree software build system? Yes, that one could go in its own directory. Do you have examples of Makefiles that do fully-out-of-tree builds to get a precise idea of how this would work?
Does litescope need anything else than generated/csr.h?

[jordens]

I had thought libopencm3 (as mentioned above) does it. They support a slew of different processors and combinations of flags. buildroot certainly has it. And it does not look that hard. .o and generated headers would just land where the project is.

A few cores would want to use crc*, hook up their own commands into the bios shell, use console.h.

[enjoy-digital]

Yes Sebastien I meant that software for the embedded CPU should go in misoc/cpu/software misoc/cpu/software. That's also why I would create a cpu directory in misoc with lm32 and mor1kx in it: If we add another cpu, we would be able to share the code in c

Not sure we understood correctly each others, but I don't want for example to move the spi_flash C code that is in software in misoc/spiflash, it would stay in misoc/cpu/software, same for uart and others drivers.

Software of the others cores does not share anything with what would be in misoc/cpu/software.
It's mostly host software scripts or drivers that only require the csr.h or csr.csv that will be generated during the build.

Now I agree that in misoc/cpu/software, you will have code that is related to cores that are outside of that. (spi_flash, uart, dram...) but having them in the cores would probably be a mess. Potentially, you will also want have code to initialize the dram in LitePCIe or the access the flash, and it probably won't share anything with the embedded software.

So this not perfect, but seems acceptable, no?

[enjoy-digital]

Hi,

I've moved the cores that would have been difficult to refactor in a flattened organization outside of misoc. So I no longer have restrictions on what Robert was suggesting. I'm willing to help for this reorganization, but I'll wait until we will all be ready to start. (I don't want to break ARTIQ or others designs and we can probably work in a branch first as Robert suggested).

Florent

[sbourdeauducq]

Any comments on the organization currently implemented in the new branch?

[mithro]

Lots of good things about the new structure!

Some things I thought after having a quick glance;

• It was nice that previously all the "external cores" where collected together into one directory. The new structure makes it hard to tell when it is external.
• The misoc/software directory feels a little bit "out of place" because it is really "data" rather than Python code.
• With there being one misoc/cores directory, the naming of cores becomes a lot more important. It feels a bit more confusing then it was previously. For example, why is sdram split into sdram_phy, sdram_{phy,mode,settings,tester}?
• Is it expected that cores in external repos would also end up placing things under the "misoc/cores" structure?
• It would be nice if we could improve the documentation on the cores :)

[sbourdeauducq]

• What do you mean by "external" cores and why does it matter? lm32, or1k, etc. are git submodules and installed/distributed with misoc.
• this is a requirement for making it part of an installable python package.
• the sdram structure is: sdram_settings (like dfi) is shared by controllers and PHYs, and then the other cores are largely independent. How would you organize this?
• external repos never modify misoc, which becomes a regular python package. If you need a core which is not in misoc, you should import it from its own repository (the "targets" are now executable files, which makes things more flexible here).

[jordens]

The few bits in migen that I have played bit so far look good. What would help is a porting.rst or NEWS.rst that describes what has changes and where the new things are. Also (and that would be a point for NEWS.rst) I have not jet figured out where Source, Sink and all the actor stuff have gone.

[sbourdeauducq]

It's in misoc.interconnect.stream (and simplified).