Skip to content
master
Switch branches/tags
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 

README.md

k5-spec

Idea here is pretty simple. Encapsulate the details of a trained neural network into a on-disk format. The network should be human readable, as in https://github.com/jsongraph/json-graph-specification:

{
    "graph": {
        "nodes": [
            {
                "id": "A",
            },
            {
                "id": "B",
            }
        ],
        "edges": [
            {
                "source": "A",
                "target": "B"
            }
        ]
    }
}

Coupled with this format are a TENSOR (think matrix or vector but N-dimentional) of weights that correspond to each layer or "node" in the network above. These weights could be fp16 fp32 or fp64. These need to be stored with the network above but weights are not human readable. Which is why we can just cram all these data structures into the widely supported HDF5 format (https://www.hdfgroup.org/HDF5/)

So if we were to look at a K5 file, it would look something like:

$ h5ls working/mnist.k5

/                        Group
/author                  Dataset {25}
/description             Dataset {82}

/model                   Group
/model/network           Dataset {1484}
/model/weights           Group
/model/weights/A         Dataset {10000, 1}
/model/weights/B         Dataset {10000, 1}

/validate                Group
/validate/input          Dataset {32,32}
/validate/output         Dataset {10,1}

/training                Group
/training/data           Dataset {60000/Inf, 32, 32}
/training/labels         Dataset {60000, 1}

/evaluate                Group
/evaluate/data           Dataset {10000/Inf, 32, 32}
/evaluate/labels         Dataset {10000, 1}

In the above example, we have 4 distinct parts:

  • Metadata
  • The model (network as JSON and weights separated by layers)
  • Just one input and output to validate that the network was correctly loaded
  • Training data
  • Testing data

About

Encapsulate the details of a trained neural network into a on-disk format

Resources

License

Releases

No releases published

Packages

No packages published