In order to be able to intelligently match containers to systems, whether that means pulling and then allocating to an appropriate node, or making a decision to retrieve a container based on available architectures, the core of this need is being able to understand what is inside the container, and how that is (or is not) compatible with the host. Our base library of information needs to capture enough metadata do address several common use cases:
- Can I run X on my host? Can I optimally run it?
- Are the libraries in my container compatible with my host?
- Can I bind mount library X and it will still work?
If you look around, many container orchestration and similar tools are implemented in Go. It's the language of this ecosystem, even if that goes against what are considered traditional academic languages (Python, C++, etc.) I also really love Go and I don't get enough opportunity to use it. Another reason is strong typing and having data structures conform to a know standard without needing libraries like jsonschema.
To build the library:
$ make
This will compile an executable, containerspec
that you can interact with.
The first thing you might want to do is extract metadata for a host. Technically, the only difference between a host and container is the layer of abstraction. We'd want to run the same thing on our host as in the container.
$ $ ./containerspec host
{x86 [] [generic] 0 [] map[]}
Under development I'd like to have commands that can read a Dockerfile, or
a directory / repository of Dockerfile
s, and be able to tell us:
- Is the digest up to date?
- Are there new tags we might want to build?
And then this can be integrated into a GitHub action. The reason this is packaged
here is because if this tool can parse and reason about Dockerfile
, we want to
make sure that we add the right metadata (for now, LABEL
s) that can then inform
about later compatibility.
I'm not decided yet about how to implement this, but I want the following:
For base images and updating them:
- User can target a Dockerfile directly for one off update, or a folder with tags for scaled
- Read in Dockerfile, keep track of labels and FROMS (add dockerfile parser)
- Look at labels to see if a tag is there for the hash
- For each FROM, look up list of tags, update hash
- For each FROM, if a label exists after it for opencontainers, delete it
- Update label to use new tag
Try to use the labels here with opencontainers labels So far we have the following labels demonstrated in this paper:
Label | Values | Comment |
---|---|---|
org.supercontainers.mpi | {mpich,openmpi} | Required MPI support, ABI compatibility |
org.supercontainers.gpu | {cuda,opencl,rocm, etc} | Required GPU library support |
org.supercontainers.glibc | Semantic version: XX.YY.Z | Specific version of GLIBC |
- archspec/archspec-go doesn't have parsing / matching functions?
- start with a basic container with some mpi and operating system
- create a grid that can generate the "same" container with different operating systems
- create a library and extract from the container:
- analyze with container diff
- get information about host
- Need actual use cases of what people want to do.
- archspec: by the Spack team, which provides the database of architectures to start with. We want to use archspec-go which I don't think is done yet.