Jsonnet Custom-Importers

• Extend the jsonnet import and importstr functions with the help of custom importers.
• Choose either a specific importer or use the NewMultiImporter() function to enable all together. The prefix of an import path will route to the right importer. As a fallback the default go-jsonnet FileImporter will be used.
• A custom importer can be set for a jsonnet VM like:

import (
    ...
	importer "github.com/peterbueschel/jsonnet-custom-importers"
)
...
    
    m := importer.NewMultiImporter()
    vm := jsonnet.MakeVM()
    vm.Importer(m)

---

Custom Importers

The main idea is to add a kind of intrinsic functionality into the import path string by adding extra content. The custom importers can parse these extra content and act on them.

The pattern for the import path is based on URLs

• url:

[scheme:][//[userinfo@]host][/]path[?query][#fragment]

• import pattern:

import '[<importer-prefix>://][<glob-pattern>|<filepath>][?<query-parameters>]'

Example patterns

Original go-jsonnet FileImporter

import 'example.jsonnet'

where the [<importer-prefix>://] is empty and the <filepath> is example.jsonnet

Custom GlobImporter

import 'glob.stem+://**/*.jsonnet'

where glob.stem+ is the <importer-prefix> and the <glob-pattern> is **/*.jsonnet.

Custom GlobImporter With Query-Parameter

import 'glob.stem+://**/*.jsonnet?exclude=**/*ignore.*'

same as before but in addition with the <query-parameter> exclude=**/*ignore.*.

List of available custom importers and the supported prefixa

Name	<importer-prefix> in import path	<importer-prefix> in importstr path	<query-parameters>
MultiImporter	any - will address the right importer	any	logLevel=<info|debug>, importGraph=<filepath>
GlobImporter	glob.<?>, glob.<?>+, glob+	glob-str.<?>, glob-str.<?>+, glob-str+	logLevel=<info|debug>, exclude=<glob-pattern>

---

MultiImporter

• This importer includes all custom importers and as fallback the default go-jsonnet FileImporter. The MultiImporter tries to find the right custom importer with the help of the <importer prefix>. If it found one, the import string will be forwarded to this custom importer, which in turn takes care of the string.
• Optionally, custom importers can be chosen via:

  m := NewMultiImporter(NewGlobImporter(), NewFallbackFileImporter())

GlobImporter

• Is a custom importer, which:
  • Imports multiple files at once via glob patterns handled by the doublestar library.
  • Supports Continuous imports: If inside the resolved files other glob-patterns will be found, the GlobImporter will also take these glob-imports and resolves the underlying files.
  • Can Exclude imports: use exclude=<glob pattern> as query parameter to exclude files from further handlings.
  • Supports extra JPaths: extra search paths for additional libraries. (⚠️ matches of library paths have a lower priority then matches in the current work dir)
  • Sorts the resolved files: in lexicographical and hierarchical order. Example: [a0 b02 a0/b/c a1 d/x c b2 a1/b b10] becomes [a0 a0/b/c a1 a1/b b02 b10 b2 c d/x]
• Activate the glob-import via the prefix glob.<?> or glob.<?>+ to get the content of the resolved files as object. The content of each file will be available under its resolved path, filename, stem (filename with file extension) or dirname. (see also table in section "Prefix glob.<?> And glob.<?>+")
• Use the prefix glob+ to merge the returned imports. (similar to the jsonnet +: functionality)

Examples

(More examples can be found in the testings file.)

Folder structure:

models
├── blackbox_exporter.json
├── blackbox_exporter.libsonnet
├── node_exporter.json
├── node_exporter.libsonnet
├── development
│   └── grafana.libsonnet
├── production
│   ├── victor_ops.libsonnet
│   └── grafana.libsonnet
└── wavefront.libsonnet

Prefix `glob.<?>` And `glob.<?>+`

• Each resolved file, which matched the glob pattern, will be handled individually and will be available in the code under a specific variable name. The variable name can be specified in the <?> part.

• <?> can be one of the following options:

  option	example result
  path	/foo/bar/baa.jsonnet
  file	baa.jsonnet
  stem	baa
  dir	/foo/bar/

• ⚠️ On colliding file|stem|dir -names, only the last resolved result in the hierarchy will be used. Use the glob.<?>+ (extra +) prefix to merge colliding names instead. The imports will be merged in hierarchical and lexicographical order similar to glob+. (also note: glob.path and glob.path+ are the same)

Example Input glob.path

import 'glob://models/**/*.libsonnet';

Example Result glob.path

Code which will be evaluated in jsonnet:

 {
   'models/blackbox_exporter.libsonnet': import 'models/blackbox_exporter.libsonnet',
   'models/node_exporter.libsonnet': import 'models/node_exporter.libsonnet',
   'models/wavefront.libsonnet': import 'models/wavefront.libsonnet',
   'models/development/grafana.libsonnet': import 'models/development/grafana.libsonnet',
   'models/production/grafana.libsonnet': import 'models/production/grafana.libsonnet',
   'models/production/victor_ops.libsonnet': import 'models/production/victor_ops.libsonnet',
 }

Example Input glob.stem

import 'glob.stem://models/**/*.libsonnet'

Example Result glob.stem

Code which will be evaluated in jsonnet:

  {
    blackbox_exporter: import 'models/blackbox_exporter.libsonnet',
    node_exporter: import 'models/node_exporter.libsonnet',
    wavefront: import 'models/wavefront.libsonnet',
    grafana: import 'models/production/grafana.libsonnet',
    victor_ops: import 'models/production/victor_ops.libsonnet',
  }

Example Input glob.stem+

import 'glob.stem+://models/**/*.libsonnet'

Example Result glob.stem+

Code which will be evaluated in jsonnet:

 {
   blackbox_exporter: import 'models/blackbox_exporter.libsonnet',
   node_exporter: import 'models/node_exporter.libsonnet',
   wavefront: import 'models/wavefront.libsonnet',
   grafana: (import 'models/development/grafana.libsonnet') + (import 'models/production/grafana.libsonnet'),
   victor_ops: import 'models/production/victor_ops.libsonnet',
 }

Prefix `glob+`

These files will be merged in the hierarchical and lexicographical order.

Example Input

import 'glob+://models/**/*.libsonnet'

Example Result

Code which will be evaluated in jsonnet:

(import 'models/blackbox_exporter.libsonnet') +
(import 'models/node_exporter.libsonnet') +
(import 'models/wavefront.libsonnet') +
(import 'models/sub-folder-2/grafana.libsonnet')

Options

Logging

Enable/add a zap.Logger via <Importer>.Logger() per importer or just use the this method on the MultiImporter instance to enable this of all underlying custom importers:

import (
  ...
  "go.uber.org/zap"
)
...
l := zap.Must(zap.NewDevelopment()) // or use zap.NewProduction() to avoid debug messages
m := NewMultiImporter()
m.Logger(l)
...

(update since v0.0.3-alpha) Another option is to use the special config import inside an jsonnet file:

// set the logLevel
local importers = import 'config://set?logLevel=debug';

// enable it for example via:
local myother_imports = importers + (import 'somethingElse.jsonnet');
...

Aliases

Add an alias for an importer prefix, like:

 g := NewGlobImporter()
 if err := g.SetAliasPrefix("glob", "glob.stem+"); err != nil {
   return err
 }
 m := NewMultiImporter(g)

The SetAliasPrefix() can be used multiple times, whereby only the last setting for an alias-prefix pair will be used.

Import Graph

The MultiImporter can detect import cycles and creates an import graph in dot format once it found a cycle.

In addition such import graphs can also be enable independently via the special config import inside an jsonnet file:

// set the import graph file name
local importers = import 'config://set?importGraph=import_graph.gv';

// enable the file creation:
local myother_imports = importers + (import 'somethingElse.jsonnet');
...

Example image from testdata/inFileConfigs/importGraph.jsonnet:

the image was created via dot -Tsvg -O graph.gv command (ref. graphviz cli tool)

Ignore Import Cycles

To disable the tests and therefore any error handling for import cycles, you can use the following config in your jsonnet code:

// set the import graph file name
local importers = import 'config://set?ignoreImportCycles';

// enable the file creation:
local myother_imports = importers + (import 'somethingElse.jsonnet');
...

Or directly in your go code via:

 m := NewMultiImporter(g)
 m.IgnoreImportCycles()

Dependencies

• https://github.com/google/go-jsonnet the reason for everything :-)
• https://github.com/bmatcuk/doublestar support for double star (**) glob patterns
• https://github.com/uber-go/zap for structured logging
• https://github.com/dominikbraun/graph to check for import cycles and generate import graphs

Other Projects

• another glob–importer https://qbec.io/reference/jsonnet-glob-importer/
  • example usage https://github.com/splunk/qbec/blob/main/vm/internal/importers/glob_test.go#L32

Follow-Up Tasks

• importstr: add support for importstr
• Ignore paths: add support in the GlobImporter for ignore paths
• Alias: add a prefix to a custom importer via <Importer>.AddPrefix(string)
• HTTP support: loads single files per url
• Git support: loads files in branches from repositories