Directory structure #23
Description
Some questions were asked on PRs about the expected directory structure. Such a new project can start a bit messy, but it's good if we can agree on the longer term structure and work towards it.
Some comments I've seen question the repetition, while others the need for the python directory. Below is an initial proposal. Let's iterate until we're happy with it.
- lighthouse (core python modules)
- ingress (modules that create payload MLIR files/modules)
- pipeline (modules that create and run schedules and pass pipelines)
- runtime (modules that take payload, drivers and libraries and run them on hardware)
- ingress (reusable scripts and utilities that handle different payload ingress)
- pipeline (reusable scripts and utilities that run schedules and pipelines)
- runtime (reusable scripts and utilities that run and benchmark payloads and schedules)
- examples (a set of end-to-end scripts that demonstrate usage of the modules and utilities)
- linalg-to-cpu (a particular example with a full end-to-end pipeline)
- kernel-bench (an example to take python models and convert them all to MLIR)
- ...
- cache (a repository of cached files generated by the scripts above in special conditions)
Note the repetition, this is intentional. Alternatively, we can just move the scripts and utilities into the lighthouse sub-directory.
We should also create a build directory for virtual-env, MLIR caches, temporary downloads and future building of sub projects.
One key aspect discussed is also the idea of keeping checked-in caches for MLIR inputs. This can help testing and reproduction of results. Any upstream cached file should come with a script to re-create it. We could also have in there IR that was created downstream, for cross testing purposes.