puppet-agent expects a value for struct key

[XMol]

Hello,

our multipath module defines the custom data type Multipath::Config as shown below. When calling puppet agent -t for an example host, an error is thrown and catalog compilation is aborted:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Class[Multipath]:
parameter 'config' entry 'devices' index 0 expects a value for key 'vendor'
parameter 'config' entry 'devices' index 0 expects a value for key 'product'
parameter 'config' entry 'multipaths' index 0 expects a value for key 'wwid'
parameter 'config' entry 'multipaths' index 1 expects a value for key 'wwid'
(...)
parameter 'config' entry 'multipaths' index 47 expects a value for key 'wwid' on node f01-119-121.gridka.de

As far as I could verify what happens, everything should work out just fine. Most surprising is, that the same client can request a catalog from the master, which does contain the desired parameters for the multipath class:

$ curl -s --cert node.cert.pem --key node.key.pem https://p6master:8140/puppet/v3/catalog/node?environment=multipath_transition_to_Puppet_data_types >catalog.json

$ jq '.resources[] | select( .title == "Multipath" ) | .parameters.config | {devices, multipaths}' catalog.json
{
  "devices": [
    {
      "vendor": "(LSI|NETAPP)",
      "product": "INF-01-00",
      "product_blacklist": "Universal Xport",
      "path_grouping_policy": "group_by_prio",
      "path_selector": "service-time 0",
      "path_checker": "rdac",
      "features": "2 pg_init_retries 50",
      "hardware_handler": "1 rdac",
      "prio": "rdac",
      "failback": "manual",
      "no_path_retry": "30",
      "rr_weight": "uniform",
      "rr_min_io": "100",
      "rr_min_io_rq": "1"
    }
  ],
  "multipaths": [
    {
      "alias": "119_912_DDP05_03",
      "wwid": "36d039ea00009466f0000010f5e30445e"
    },
    (...)
    {
      "alias": "119_912_DDP04_04",
      "wwid": "36d039ea00009466f0000010e5e3043fd"
    }
  ]
}

Applying this catalog with puppet apply --catalog catalog.json fails. Though not because of the multipath class, but rather several file resources that fail to source the data from the Puppet master (none the source URLs have the Puppet master in them).

I've run successful tests where the parameters are compared against the Multipath::Config::Device and Multipath::Config::Multipath data types (basically notify { 'validate multipath config': message => $devices =~ Array[Multipath::Config::Device]]). Hence, I'm quite confident that the actual data struct is correct and should be accepted and applied by puppet agent.

puppet agent --version ⇒ 6.26.0
puppetserver --version ⇒ 6.18.0

In case that matters, the code for the multipath class is found in a feature environment - "multipath_transition_to_Puppet_data_types" - while all else is merged in from the production environment.

Does anyone have an idea what is going wrong in this situation?

Thank you for your time,
Xavier.

---

Multipath::Config data type definition, shortened for clarity:

type Multipath::Config = Struct[{
  Optional['defaults']             => Multipath::Config::Defaults,
  Optional['blacklist']            => Multipath::Config::Blacklist,
  Optional['blacklist_exceptions'] => Multipath::Config::Blacklist_exceptions,
  Optional['devices']              => Array[Multipath::Config::Device],
  'multipaths'                     => Array[Multipath::Config::Multipath],
}]

type Multipath::Config::Device = Struct[{
  'vendor' => String,
  'product' => String,
  Optional['alias_prefix'] => String,
  (...)
}]

type Multipath::Config::Multipath = Struct[{
  'wwid' => String,
  Optional['alias'] => String,
  (...)
}]

---

EDIT: As requested on the Puppet Slack, here is the full source code of the multipath class:

class multipath (
  String $package = 'device-mapper-multipath',
  Stdlib::Unixpath $multipath_conf = '/etc/multipath.conf',
  String $service = 'multipathd',
  Boolean $locked = false,
  Multipath::Config $config = {},
) {

  package { $package: }
  -> file { $multipath_conf:
    owner   => 'root',
    group   => 'root',
    replace => ! $locked,
    content => template("${module_name}/multipath.conf.erb"),
  }
  ~> service { $service:
    ensure     => 'running',
    enable     => true,
    hasrestart => true,
    hasstatus  => true,
  }
}

---

EDIT2: Also as requested on Slack, this is how the multipath class is declared in Puppet code:

class { 'multipath':
        locked => $multipath_locked,
        config => {
            'defaults'   => $multipath_defaults,
            'devices'    => $multipath_devices,
            'blacklist'  => {
                'devnode' => [
                    '^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*',
                    '^hd[a-z]'
                ],
            },
            'multipaths' => $gridka_gxfs_multipath_aliases_struct,
        },
    }

$multipath_defaults and $multipath_devices are hard-coded:

            $multipath_defaults = {
                'hw_str_match'         => 'yes',
                'user_friendly_names'  => 'yes',
                (...)
                'log_checker_err'      => 'always',
            }

            $multipath_devices = [
                {
                    'vendor'               => '(LSI|NETAPP)',
                    'product'              => 'INF-01-00',
                    'product_blacklist'    => 'Universal Xport',
                    (...)
                    'rr_min_io_rq'         => 1,
                },
            ]

$gridka_gxfs_multipath_aliases_struct is a product of stdlib/parsejson() of a (custom) fact.

[XMol]

Thanks to all the support found on Puppet Slack! We determined that the issue is not with the input data, nor with the custom Puppet data type or the multipath class, but in fact the ERB template. It processes the hash of multipaths like this:

multipaths {
<% @config['multipaths'].sort { |x, y| x['wwid'] <=> y['wwid'] }.each do |multipath| -%>
  multipath {
    <%-# Start each multipath subsection with the wwid. -%>
    wwid <%= multipath.delete('wwid') %>
    <%- multipath.keys.sort.each do |k| -%>
    <%= k %> <%= multipath[k] %>
    <%- end -%>
  }

<% end -%>
}

The issue is triggered by multipath.delete('wwid'). It obviously removes the wwid key and value from the hash. And when the template is fully rendered, Puppet realizes the mandatory key is missing, which violates the data type definition, and errors out.

The solution ultimately is to avoid using delete(). E.g. by transitioning the template to EPP, work with a copy of the hash or simply iterate non-destructively over the hash.