registry.md

Template Registries

DM lets configurations instantiate templates using both imports and references.

Because template references can use any public HTTP endpoint, they provide a way to share templates. While you can store templates anywhere you want and organize them any way you want, you may not be able to share them effectively without some organizing principles. This document defines conventions for template registries that store templates in Github and organize them by name and by version to make sharing easier.

For a working example of a template registry, please see the Kubernetes Template Registry.

Template Versions

Since templates referenced by configurations and by other templates may change over time, we need a versioning scheme, so that template references can be reliably resolved to specific template versions.

Every template must therefore carry a version based on the Semantic Versioning specification. A template version consists of a MAJOR version, a MINOR version and a PATCH version, and can be represented as a three part string starting with the letter v and using dot delimiters between the parts. For example v1.1.0.

Parts may be omitted from right to left, up to but not include the MAJOR version. All omitted parts default to zero. So, for example:

• v1.1 is equivalent to v1.1.0, and
• v2 is equivalent to v2.0.0

As required by Semantic Versioning:

• The MAJOR version must be incremented for incompatible changes
• The MINOR version must be incremented functionality is added in a backwards-compatible manner, and
• The PATCH version must be incremented for backwards-compatible bug fixes.

When resolving a template reference, DM will attempt to fetch the template with the highest available PATCH version that has the same MAJOR and MINOR versions as the referenced version. However, it will not automatically substitute a higher MINOR version for a requested MINOR version with the same MAJOR version, since although it would be backward compatible, it would not have the same feature set. You must therefore explicitly request the higher MINOR version in this situation to obtain the additional features.

Template Validation

Every template version should include a configuration named example.yaml that can be used to deploy an instance of the template. This file, along with any supporting files it requires, may be used automatically in the future by a template testing framework to validate the template, and should therefore be well formed.

Template Organization

Technically, all you need to reference a template is a directory at a public HTTP endpoint that contains a template file named either <template-name>.py or <template-name>.jinja, depending on the implementation language, along with any supporting files it might require, such as an optional schema file named <template-name>.py.schema or <template-name>.jinja.schema, respectively, helper files used by the implementation, files imported by the schema, and so on.

Basic structure

These constraints impose a basic level of organization on the template definition by ensuring that the template and all of its supporting files at least live in the same directory, and that the template and schema files follow well-defined naming conventions.

They do not, however, provide any encapsulation. Without additional constraints, there is nothing to prevent template publishers from putting multiple templates, or multiple versions of the same template, in the same directory. While there might be some benefits in allowing templates to share a directory, such as avoiding the duplication of helper files, the cost of discovering and maintaining templates would quickly outweigh them as the number of templates in the directory increased.

Also, since it may reduce management overhead to store many different templates, and/or many versions of the same template, in a single repository, we need a way to organize templates within a repository.

Therefore:

• Every template version must live in its own directory named for the version.
• The version directory must contain exactly one top-level template file and supporting files for exactly one template version.
• All of the versions of a given template must live under a directory named for the template without extensions.

For example:

templateA/
  v1/
    example.yaml
    templateA.py
    templateA.py.schema
  v1.0.1/
    example.yaml
    templateA.py
    templateA.py.schema
  v1.1/
    example.yaml
    templateA.py
    templateA.py.schema
    helper.py

In this example, templateA is a template directory, and v1, v1.01, and v1.1 are template version directories that hold the versions of templateA.

Registry based template references

In general, templates references are just URLs to HTTP endpoints. However, because a template registry follows the conventions outlined above, references to templates in a template registry can be shorter and simpler than generalized template references.

In a registry based template reference, the scheme part of the URL and the name of the top level template file are omitted, and the version number is delimited by a colon. So for example, instead of

https://raw.githubusercontent.com/ownerA/repository2/master/templateA/v1/templateA.py

you can simply write

github.com/ownerA/repository2/templateA:v1

The general pattern for a registry based template reference is as follows:

github.com/<owner>/<repository>/<collection>/<template>:<version>

The collection segment, described below, is optional, and may be omitted.

Grouping templates

Of course, a flat list of templates won't scale, and it's unlikely that any fixed taxonomy would work for all registries. Template directories may therefore be grouped in any way that makes sense to the repository maintainers.

For example, this flat list of template directories is valid:

templates/
  templateA/
    v1/
    ...
  templateB/
    v2/
    ...

This example, where template directories are organized by category, is also valid:

templates/
  big-data/  
    templateA/
      v1/
      ...
    templateB/
      v2/
      ...
  signals
    templateC/
      v1/
      ...
    templateD/
      v1.1/
      ...

Template collections

A side effect of allowing arbitrary grouping is that we don't know how to find templates when searching or listing the contents of a registry without walking the directory tree down to the leaves and then backtracking to identify template directories.

Since walking the repository is not very efficient, we introduce the concept of collections.

Definition

A collection is a directory that contains a flat list of templates. Deployment manager will only discover templates at the root of a collection.

So for example, templateA and templateB live in the templates collection in the first example above, and in the big-data collection in the second example.

A registry may contain any number of collections. A single, unnamed collection is implied at the root of every registry, but additional named collections may be created at other points in the directory structure.

Usage

Of course, collections are useless if we can't reference them efficiently. A registry based template reference may therefore include a collection name. A collection name is the only path segment allowed between the repository name and the template name. So, for example, this is a valid template reference:

github.com/ownerA/repository2/collectionM/templateA:v1

but this is not:

github.com/ownerA/repository2/multiple/path/segments/are/not/allowed/templateA:v1

Because it may appear in a template reference, a collection name must not contain URL path separators (i.e., slashes). However, it may contain other delimiters (e.g., dots). So, for example, this is a valid template reference:

github.com/ownerA/repository2/dot.delimited.strings.are.allowed/templateA:v1

Mapping

Currently, deployment manager maps collection names to top level directory names. This mapping implies that registries can be at most one level deep. Soon, however, we plan to introduce a metadata file at the top level that maps collection names to paths. This feature will allow registries to have arbitrary organizations, by making it possible to place collections anywhere in the directory tree.

When the metadata file is introduced, the current behavior will be the default. So, if the metadata file is not found in a given registry, or if a given collection name is not found in the metadata file, then deployment manager will simply map it to a top level directory name by default. This approach allows us to define collections at the top level now, and then move them to new locations later without breaking existing template references.

Using Template Registries

Accessing a template registry

The Deployment Manager client, dm, can deploy templates directly from a registry using the following command:

$ dm deploy <template-name>:<version>

To resolve the template reference, dm looks for a template version directory with the given version in the template directory with the given template name in the default template registry.

The default is the Kubernetes Template Registry, but you can set a different default using the --registry flag:

$ dm --registry my-org/my-repo/my-collection deploy <template-name>:<version>

Alternatively, you can specify a complete template reference using the pattern described above, like this:

$ dm deploy github.com/my-org/my-repo/my-collection/<template-name>:<version>

If a template requires properties, you can provide them on the command line:

$ dm --properties prop1=value1,prop2=value2 deploy <template-name>:<version>

Changing a template registry

DM relies on Github to provide the tools and processes needed to add, modify or delete the contents of a template registry. Conventions for changing a template registry are defined by the registry maintainers, and should be published in the top level README.md or a file it references, following standard Github practices.

The Kubernetes Template Registry follows the workflow used by Kubernetes.