Including external YAML files

Philippe Proulx edited this page Mar 19, 2016 · 5 revisions

It is possible to include other YAML files in the following objects:

When including an external YAML file into a given object, the object properties contained in the included file become the base properties of the given object. Then, the properties of the given object (in the file including the external file) "patch", or extend the base properties in the following way:

  • Booleans, numbers, and strings:
    • Property exists in base object: base property is replaced by included property
    • Property does not exist in base object: included property is appended to base object
  • Arrays:
    • Property exists in base object:
      • Base property is an array: new elements are appended to base array
      • Base property is not an array: base property is replaced by included property
    • Property does not exist in base object: included property is appended to base object
  • Associative arrays:
    • Property exists in base object:
      • Base property is an associative array: apply those three rules to each property, in order, where the base object is this associative array
      • Base property is not an associative array: base property is replaced by included property
    • Property does not exist in base object: included property is appended to base object

You can consider an external YAML file inclusion as a smart inheritance from the object in that file. Note that the rules are slightly different from type inheritance here, where associative arrays are not processed recursively (first-level associative array properties are replaced rather than extended).

Including one or more external YAML files is done with the special $include property of one of the objects above. The value of this property can be:

  • A string: name of the YAML file to include
  • An array of strings: names of the YAML files to include, processed in order

When multiple files are given, the second object inherits from the first one, the third object inherits from the second one, and so on, until finally the object with this $include property inherits from the last one. This allows you to use inclusions as mixins.

Inclusion order

The deeper $include properties are processed first. For example, consider the following metadata object:

$include: metadata.yaml
type-aliases:
  uint16:
    class: int
    size: 16
clocks:
  my_clock:
    $include: clock.yaml
trace:
  $include: trace.yaml
  byte-order: le
streams:
  my_stream:
    $include: stream.yaml
    packet-context-type:
      class: struct
      fields:
        packet_size: uint16
        content_size: uint16
    events:
      my_event:
        $include: event.yaml
        payload-type:
          class: struct
          fields:
            my_field:
              class: int
              size: 8

In this example, the inclusions are processed in this order:

  1. event.yaml
  2. clock.yaml, stream.yaml, and trace.yaml (order is not important here, because none of the objects including those is the parent of another)
  3. metadata.yaml

Include paths

Included YAML files can include other YAML files themselves, as long as the include graph stays acyclic.

An included YAML file name is absolute if it starts with /.

Relative included YAML files are searched for relative to the current working directory by default, and relative to the provided include directory if not found. Other directories can be searched into before searching the default search directories by using the -I option of the barectf tool one or more times.

By default, the barectf tool stops and prints an error when an included file is not found. You can instruct the tool to ignore such errors with the --ignore-include-not-found option.

Provided includes

The barectf package ships with a few provided includable configuration files. This directory is part of the default search directories, so the file names can be included without specifying an absolute path. As of this version, the provided files are:

Example

base.yaml:

$include: stdint.yaml
$log-levels:
  critical: 50
  error: 40
  warning: 30
  info: 20
  debug: 10
type-aliases:
  clock-int:
    $inherit: uint64
    property-mappings:
      - type: clock
        name: sys_clock
        property: value
clocks:
  sys_clock:
    description: System clock
    offset:
      seconds: 1458353794
trace:
  $include: trace-basic.yaml
  byte-order: be
streams:
  default:
    packet-context-type:
      class: struct
      fields:
        timestamp_begin: clock-int
        timestamp_end: clock-int
        packet_size: uint32
        content_size: uint32
        events_discared: uint16
    event-header-type:
      class: struct
      fields:
        timestamp: clock-int
        id: uint8

event-net-base.yaml:

payload-type:
  class: struct
  fields:
    ctx_id: uint16
    app_proto:
      class: enum
      value-type: uint8
      members:
        - HTTP
        - HTTPS
        - FTP
        - TFTP
        - SMTP
        - DNS
        - BOOTP

event-net-recv-send.yaml:

log-level: debug
payload-type:
  class: struct
  fields:
    src: addr_type
    dst: addr_type
    msg_size: uint32

config.yaml:

version: '2.1'
metadata:
  $include:
    - base.yaml
    - stdint.yaml
    - stdmisc.yaml
  type-aliases:
    addr_type: string
  clocks:
    sys_clock:
      freq: 2500000000
      offset:
        seconds: null
        cycles: 10028
  streams:
    default:
      events:
        net_send:
          log-level: info
          $include:
            - event-net-base.yaml
            - event-net-recv-send.yaml
        net_recv:
          $include:
            - event-net-base.yaml
            - event-net-recv-send.yaml
          payload-type:
            fields:
              app_proto:
                members:
                  - label: IMAP
                    value: 100
                  - NTP
                  - POP
                  - LDAP

After running barectf config.yaml, the effective configuration file is:

version: '2.1'
metadata:
  type-aliases:
    uint8:
      class: int
      size: 8
      align: 8
    byte: uint8
    int8:
      $inherit: uint8
      signed: true
    uint16:
      class: int
      size: 16
      align: 16
    word: uint16
    int16:
      $inherit: uint8
      signed: true
    uint32:
      class: int
      size: 32
      align: 32
    dword: uint32
    int32:
      $inherit: uint8
      signed: true
    uint64:
      class: int
      size: 64
      align: 64
    int64:
      $inherit: uint8
      signed: true
    byte-packed-uint8: uint8
    byte-packed-int8: int8
    byte-packed-uint16:
      $inherit: uint16
      align: 8
    byte-packed-int16:
      $inherit: int16
      align: 8
    byte-packed-uint32:
      $inherit: uint32
      align: 8
    byte-packed-int32:
      $inherit: int32
      align: 8
    byte-packed-uint64:
      $inherit: uint64
      align: 8
    byte-packed-int64:
      $inherit: int64
      align: 8
    bit-packed-uint8:
      $inherit: uint8
      align: 1
    bit-packed-int8:
      $inherit: int8
      align: 1
    bit-packed-uint16:
      $inherit: uint16
      align: 1
    bit-packed-int16:
      $inherit: int16
      align: 1
    bit-packed-uint32:
      $inherit: uint32
      align: 1
    bit-packed-int32:
      $inherit: int32
      align: 1
    bit-packed-uint64:
      $inherit: uint64
      align: 1
    bit-packed-int64:
      $inherit: int64
      align: 1
    clock-int:
      $inherit: uint64
      property-mappings:
        - type: clock
          name: sys_clock
          property: value
    uuid:
      class: array
      length: 16
      element-type:
        class: int
        size: 8
    ctf-magic:
      class: int
      size: 32
      align: 32
    string:
      class: string
    addr_type: string
  $log-levels:
    critical: 50
    error: 40
    warning: 30
    info: 20
    debug: 10
  clocks:
    sys_clock:
      description: System clock
      offset:
        seconds: null
        cycles: 10028
      freq: 2500000000
  trace:
    byte-order: be
    uuid: auto
    packet-header-type:
      class: struct
      fields:
        magic:
          class: int
          size: 32
          align: 32
        uuid:
          class: array
          length: 16
          element-type:
            class: int
            size: 8
        stream_id:
          class: int
          size: 8
  streams:
    default:
      packet-context-type:
        class: struct
        fields:
          timestamp_begin: clock-int
          timestamp_end: clock-int
          packet_size: uint32
          content_size: uint32
          events_discared: uint16
      event-header-type:
        class: struct
        fields:
          timestamp: clock-int
          id: uint8
      events:
        net_send:
          payload-type:
            class: struct
            fields:
              ctx_id: uint16
              app_proto:
                class: enum
                value-type: uint8
                members:
                  - HTTP
                  - HTTPS
                  - FTP
                  - TFTP
                  - SMTP
                  - DNS
                  - BOOTP
              src: addr_type
              dst: addr_type
              msg_size: uint32
          log-level: info
        net_recv:
          payload-type:
            class: struct
            fields:
              ctx_id: uint16
              app_proto:
                class: enum
                value-type: uint8
                members:
                  - HTTP
                  - HTTPS
                  - FTP
                  - TFTP
                  - SMTP
                  - DNS
                  - BOOTP
                  - label: IMAP
                    value: 100
                  - NTP
                  - POP
                  - LDAP
              src: addr_type
              dst: addr_type
              msg_size: uint32
          log-level: debug