| MLPX Docs | Wavegen Docs |
|---|---|
 |
|
This repository contains specifications and implementations for bespoke file formats created for the HeRC lab, or for file formats we use in our work that don't have good quality existing public implementations.
Because of the nature of this repository, all tools and libraries are usually installed as a single bundle. However, check the documentation for each sub-project, many can be installed independently of each other if you so desire
Usually, downloading the latest release as pre-compiled binaries from the GitHub repository is preferred.
Otherwise, you can run make build && sudo make install.
The ./build_release.sh script can be used to generate a .deb file.
Check each sub-project for dependencies.
Each directory is a separate file format. Directories should be named after
their file format, and should contain a doc/ directory documenting the
specification for that format, as well as any other relevant documentation.
Implementations should be in a sub-directory named after the implementation
language. So for example, a C implementation of TNX might reside in ./tnx/c/.
Tools implemented in this repository should be placed in a sub-directory named
cmd. For example, a tool called tnxutil that operates on tnx files would
be long in ./tnx/cmd/tnxutil/. To be clear, this applies to tools intended
for users to use and install on their system, not tools internal to this
repository used for building or CI.
- Specifications of file formats used in the HeRC lab, either bespoke ones we have defined for ourselves, or third-party formats that we have had to work with.
- Libraries for inter-operating with such file formats.
- Tools whose express purpose is to interact with one of these file formats,
or which are used to convert to or from such a file format.
- Tools which have a broader purpose beyond say, validating or converting a format belong in their own repos.
- For example, a tool for converting MLPX files to ONNX could be placed
in
./mlpx/cmd/mlpx2onnx, but a tool for executing MLPs which uses MLPX as an input format should be placed in it's own repo.
| Format | Current Version | Status | Provenance | Purpose |
|---|---|---|---|---|
| TNX (Trivial Network eXchange) | unreleased | on-hold | Bespoke | Format for representing compute graphs, with a focus on neurual networks. |
| MLPX (MLP eXchange) | unreleased | in-progress | Bespoke | Portable format for checkpointing MLP networks. |
| Wavegen | 0.0.3 | in-progress | Bespoke | Wave generation tool & portable format for exchanging such waves. |
Current Version should be either "unreleased", for formats which have no stable release yet, or the version number of the library contained in the folder. In general, all implementations of a format in different should have the same version number.
Status should be one of "unreleased", "in-progress", "maintenance", or "complete".
- Unreleased formats have no stable public API, and may change or be removed at any time without notice.
Provenance should be either "Bespoke" for in-house formats, or the origin of the format otherwise.
Each individual format or tool may have it's own internal version. When a new version needs to be released, a monotonically increasing version number will be selected as the herc-file-formats release, and used for any generated packages. This version number will attempt to follow semantic versioning to the extend possible.
Released formats:
- mlpx 0.0.1
- wavegen 0.0.4
- mlpx 0.0.2
- wavegen 0.0.4
There are several existing bespoke formats, which should be "brought into the fold" of this repo. These formats were created before this repo.
- Several bespoke image formats in
herc-imgtoolin the herc-tools-public repo.de2rawhAKAhif24de2rawhif1hif8
herc-imgtoolitself