v0.4.0

Added

• pixie render <target> NAME... [--out DIR] renders a target's artifacts to
  standard output or a directory, arming nothing.

Changed

• Configuration is loaded by anycfg, the new name of yaconfiglib:
  netboot depends on anycfg>=0.14.0,<0.15 and tplx>=0.2.0,<0.3 instead of
  yaconfiglib and jinja2, and netboot[config] installs anycfg[yaml].
  Includes are confined by ANYCFG_CONFINE_TO; YACONFIGLIB_CONFINE_TO is
  not read, so rename it where you set it. A malformed file is still a
  one-line error, now naming file:line:col. netboot.utils.config.Namespace
  and the config classes build on anycfg.TypedNamespace / OpaqueMerge.

• Every public module declares __all__; netboot.utils exports Namespace
  too. PixieTarget, PixieImage and PixieContext are defined in private
  modules and imported, as before, from netboot or netboot.engine.

• netboot.dhcp.DhcpZone is DHCPZone (also importable from netboot), as
  every other public name spells DHCP; rename the import.

• Pixie.lookup_image returns None when nothing matches, as
  lookup_target and lookup_dhcpzone do, rather than {}; a
  FoundTargetImage hook sees None too.

• The command line is netboot.cli, one duho class per command
  (Initiate, Complete, Render, DhcpConfig); the console script is
  netboot.cli:main, and netboot.main re-exports main, PixieArgs,
  Pixie_ and parse_path. main(argv=None, *, name=None): argv is the
  first parameter, so main(["--version"]) works; pass name= by keyword.
  netboot.cmds is gone: netboot.cmds.dhcp_config.recipe is
  netboot.dhcp.condition_recipe, conditions_for is build_conditions, and
  a --cmdspath plugin with run(netboot, args, conf) works as before.

• pixie no longer imports a pixie_env settings module from sys.path,
  which under python -m netboot meant the working directory: running it in
  a directory runs no code from there. Export PIXIE_* variables instead, or
  name a module with --load-module.

• Building the engine does no DNS. A hostname-only target's ip is
  looked up by PixieTarget.resolve_ip(), which the engine calls for the
  target a command acts on (and for every hostname-only target when an IP
  query matches no configured address), so an unresolvable target no longer
  slows every command. A PixieTarget built directly has ip == "" until
  resolve_ip() is called.

• initialize(target, **options) and complete(target, **options) hand
  options to their Start/End hooks as kwargs: initiate --iscsi now reaches
  them as kwargs["iscsi"].

• A DHCP server's credentials are its own config entry's. A dhcpservers
  entry is a URI or a mapping of uri: and that backend's secrets (user,
  password; dhcpd keyname, omapi_key), written in the config or taken
  from the environment there with {{ env.NAME }}, which config files can now
  use. PIXIE_WINDHCP_PASSWORD, PIXIE_WINDHCP_USER and
  PIXIE_DHCPD_OMAPI_KEY are no longer read, and
  netboot.dhcp.WINDHCP_PASSWORD_ENV_VAR, WINDHCP_USER_ENV_VAR and
  DHCPD_OMAPI_KEY_ENV_VAR are gone: move each value into the entries of the
  servers that use it (password: "{{ env.PIXIE_WINDHCP_PASSWORD }}" keeps a
  variable you already set). An entry's secrets override a user or password
  in the URI. pixie dhcp-config --as passes the prompted user and password
  to each server directly, overriding the URI's.

• The errors are in netboot.exceptions (PixieError, PixieConfigError,
  PixieLookupError), still re-exported from netboot.

• dhcp_options, dhcp_when and dhcp_complete are read as what a target is
  served, a pydhcp DHCPLease: next-server and boot-file-name are its
  BOOTP header fields siaddr and file, as they always reached the server,
  and file, sname and siaddr may be written as such. bootfile-name is
  option 67 itself and is now sent as an option. build_next_server is
  replaced by build_lease(ctx, server). A Kea client class from dhcp_when
  carries the boot file as its boot-file-name field, not as option 67.

• An option with no type of its own (option-224, unless dhcp.options
  declares one) takes its octets as hex text or {hex: ...} in
  dhcp_options, as pydhcp's DHCPOptions.from_mapping reads them.

• The DHCP backends are dhcpctl's. netboot depends on
  pydhcpctl>=0.1.0,<0.2 (imported as dhcpctl), and netboot.dhcp is a module that builds each
  target's reservation for it: DhcpZone, connect(uri), build_options,
  build_conditions, build_completion, reservation_for(ctx, server),
  arm and disarm. What a server is sent is unchanged, byte for byte,
  including the names netboot writes on it (# >>> netboot regions,
  netboot-<mac> Windows policies). Removed, with what to do:

  • DhcpServer and its scheme-by-class-name dispatch: subclass
    dhcpctl.DHCPServer and call dhcpctl.register_backend(scheme, cls) from
    the --load-module plugin, or declare it in the dhcpctl.backends entry
    point. A backend implements add(reservation) / remove(reservation)
    (were add_target(ctx) / remove_target(ctx)), keep(reservation) (was
    keep_target(ctx, completion)), and extras(reservation, phase) (was
    extras(ctx, phase)); the netboot context is reservation.context.
  • netboot.dhcp.dnsmasq/.kea/.dhcpd/.windhcp: the backends are
    dhcpctl.backends.DnsmasqBackend, KeaBackend, DHCPDBackend and
    WinDHCPBackend; to override one, register the subclass with
    replace=True.
  • netboot.dhcp.options, .conditions and .completion: GENERIC_OPTIONS,
    APPLY_TIME, DhcpOptions, split_query, DhcpCondition, MATCH_KEYS,
    DhcpCompletion and BOOT_OPTIONS are dhcpctl's (DHCPOptions,
    Condition, Completion, ...); build_options, build_conditions,
    build_completion and zone_defaults are in netboot.dhcp.
    ConditionMissing / ConditionUnsupported are
    dhcpctl.ConditionMissingError / ConditionUnsupportedError.
  • windhcp.PASSWORD_ENV_VAR / USER_ENV_VAR are
    netboot.dhcp.WINDHCP_PASSWORD_ENV_VAR / WINDHCP_USER_ENV_VAR, and
    dhcpd.KEY_ENV_VAR is DHCPD_OMAPI_KEY_ENV_VAR; the variables are the
    same PIXIE_* ones, now read only when the URI has no user or password
    of its own: windhcp://admin:pw@host and dhcpd://keyname:key@host work.
    Logs, errors and pixie dhcp-config show a server's URI with the password
    masked.
  • netboot.DhcpServer is no longer exported.
    The kea, dhcpd, winrm and ssh extras install dhcpctl's of the same
    name. An unknown scheme, or a URI a backend cannot use, is a
    PixieConfigError; a server refusing is a dhcpctl.ServerError, reported by
    the CLI as one line with status 2. PixieEvent.BuildDhcpOptions now fires
    once per server for every backend, including one that never asks for its
    options.

• DHCP options are pydhcp's, through dhcpctl. Every IANA option is named by
  its pydhcp name (dns, bootfile-name, interface-mtu, ...) or
  option-<n>, and the names netboot already took still work
  (LEGACY_OPTION_NAMES). A config declares more in a new top-level
  dhcp.options section: an option of its own with a code and a pydhcp type,
  or an alias of a name or a code; dhcp_options, dhcp_when and
  dhcp_complete all read them. Each value is read by the
  option's type when the config is applied, so router: gateway is a
  PixieConfigError naming the image or target; a list option takes one
  address per item, so write several as a YAML list or a repeated URI key,
  not one comma-joined string. Kea is sent options by code, so a zone with a
  gateway or a lease time no longer sends Kea option names it does not define
  (router, lease-time). What to do in a BuildDhcpOptions hook: it
  receives pydhcp's DHCPOptions, keyed by code,
  so write options[DHCPOptionCode.ROUTER] = ... or options[3] where it
  wrote options["router"], or return a mapping of option names.
  next-server is no longer an option a hook sees: it is the reservation's
  BOOTP field, still set from dhcp_options: {next-server: ...}.

• A server URI says only how to reach the server. Its query takes the
  backend's connection settings and nothing else: DHCP options
  (?router=...), options_builder= and raw.<backend>= text are
  unknown-key errors. What to do: move options to the zone's settings or an
  image's or target's dhcp_options; replace an options_builder with a hook
  on PixieEvent.BuildDhcpOptions, which receives the same options plus the
  target and the server; for the iPXE chainload use dhcp_when:, and for other
  backend-native text register a backend subclass whose
  extras(reservation, phase) returns it.

• Template roots resolve against the config file's directory. Through the
  CLI, a relative templates: root is joined to the directory of the config
  file, as !include is, and with no templates: the root is templates
  beside the config file (./config/templates by default). The working
  directory's ./templates is no longer prepended. What to do: a config that
  relied on ./templates names it, relative to the config file (the example
  config now says templates: ../templates).

• The path layer is pathapi and uripath, not pathlib-next. netboot
  depends on pathapi>=0.1.0,<0.2 and pathapi-uri>=0.1.0,<0.2 (import
  package uripath) and no longer installs pathlib-next or uritools. The
  paths netboot returns are their classes: Repository.service(), .get()
  and .local are uripath.URIPath (were pathlib_next.uri.UriPath),
  parse_path() returns pathapi.LocalPath or URIPath, and a template's
  Path and Uri globals are pathapi.Path and uripath.URIPath. What to
  do: import them from there in a command module, hook or template that used
  pathlib_next; a pathlib_next path handed to netboot is not a Path.
  The URLs netboot builds are the same, but uripath differs in these ways a
  hook or template can see:

  • A join keeps the query and fragment of the service URI it starts from
    (http://h/boot?sig=1 joined with a is http://h/boot/a?sig=1); it
    dropped both.
  • Query is no longer a str, and Source.from_str() / as_str() are
    Source.parse() / str().
  • file:///x and file:/x are different URIs. netboot still writes the
    service of .local as file:/srv/tftp.

• yaconfiglib floor moves to >=0.13.1,<0.14, the first release built on
  pathapi; 0.13.0 needs pathlib-next. Configuration loading, !include
  globs and merging behave as before.

• The http extra is removed. http/https repository services are sent
  on the standard library by uripath, so they need no extra and
  Repository.service() no longer raises an ImportError naming
  netboot[http]. What to do: drop [http] from the install line (pip warns
  about the unknown extra and installs the rest). netboot[kea] still installs
  requests; install pathapi-requests yourself only to send http/https
  services through a requests.Session.

• The ssh extra is sftppath[uri,paramiko]>=0.1.0,<0.2 (was
  pathlib_next[sftp]): remote dnsmasq:// paths are sftp:// paths of
  sftppath, over paramiko. SFTPPATH_BACKEND replaces
  PATHLIB_NEXT_SFTP_BACKEND to choose paramiko or asyncssh; the old name is
  not read.

• duho floor moves to >=0.7.5,<0.8 (pre-1.0 minor, documented API break;
  0.7.4 adds the unsorted multi-source command discovery that --cmdspath
  now uses, 0.7.5 the exit="parse" mode of duho.run).
  The pixie command line is unchanged: same flags, exit statuses and output,
  with one exception in --help: the commands of one --cmdspath directory are
  listed by file name, then in the order a file defines them, where they were
  sorted by command name. The two differ only for a file that defines several
  commands or is named differently from its command.

• pixie runs through duho.run(..., exit="parse") and reports operator errors
  through duho's _errors_ / _report_error_: the same one-line log record and
  the same status 2 as before. New with it: Ctrl-C ends with status 130 and a
  broken pipe with status 1, both without a traceback; a plain run(args)
  command's returned value (not None or an int) is printed to standard
  output, where it was handed to sys.exit; with DUHO_TRACEBACK set an
  operator error also prints its traceback to standard error, and the status
  stays 2. --help, --version and a usage error still raise SystemExit
  from main, as before.

• netimps moves to >=0.4.0,<0.5 (pre-1.0 minor, documented API break).
  The network values netboot hands out are netimps', so its renames reach
  templates: target.mac.as_str("-") is now target.mac.format("-").

• Templates are rendered by tplx
  (tplx[jinja]>=0.1.0,<0.2), with the same engines, suffixes, search order and
  templates_undefined modes; the mako, liquid, handlebars and
  mustache extras install tplx's. The same templates render the same, with
  these differences:

  • An include goes through the per-target names and the image's
    template_path, and a file of another engine is rendered by that engine
    and inserted: a Jinja {% include "net" %} of net.shtpl works, where it
    failed with AttributeError. mako, Liquid, Handlebars and Mustache
    includes and partials search the template roots too.
  • Every engine sees each context field at the top level (target,
    image, ...) beside ctx; Jinja and mako saw only ctx.
  • A MAC address in a Liquid, Handlebars or Mustache template renders as
    aa:bb:cc:dd:ee:ff, where it rendered as DataView(MACAddress(...)).
  • An .epp template's output no longer gains a trailing newline the
    template did not write.
  • The shell engine leaves out _-prefixed context names other than _id.
  • Errors are tplx's: tplx.TemplateNotFound (was jinja2.TemplateNotFound),
    tplx.UndefinedError (was jinja2.UndefinedError, or KeyError from the
    shell engine), tplx.NoEngineError, tplx.EngineUnavailableError and
    tplx.RenderError (was netboot.templates.TemplateEngineError, or an
    ImportError naming netboot[mako]). pixie reports every
    tplx.TplxError as one line and exits 2.
  • PIXIE_RUBY and PIXIE_PUPPET are now TPLX_RUBY and TPLX_PUPPET.

• shell_quote is shlex.quote: a value with nothing special to the shell
  is no longer quoted ({{ shell_quote("web01") }} renders web01, was
  'web01'), and an embedded ' is written '"'"' (was '\''). Both read
  back as the same word in a POSIX shell. The quote= argument and its
  double-quote form are gone.

Removed

• netboot.utils.flatten, netboot.utils.arr_get and the
  netboot.utils.dicts module; shell_quote and T are in
  netboot.utils.misc and still importable from netboot.utils.
• netboot.templates: use tplx (Loader, the engine classes,
  register_engine for register_template_type, unregister_engine,
  registered_engines for TEMPLATE_TYPES, DataEncoder for
  template_data/jsonable). A plugin engine subclasses tplx.Template and
  implements render_values(values), where values holds ctx and the context
  fields. PixieContext._renderer is gone; the context's tplx.Loader is
  _loader.
• netboot.utils.net, netboot.netutils and the network names
  netboot.utils re-exported (Host, IPAddress, IPInterface,
  IPNetwork, MACAddress, parse, try_parse, is_valid, is_wildcard,
  resolve, ping, get_interfaces, iter_addresses, normalize_host):
  import them from netimps. netboot's Host subclass goes with them, and
  with it .try_ip() (use host.ip() or str(host)), .address (use
  .value, and build a new Host rather than assigning) and Host() with no
  argument (Host(None)). Repository.address is a netimps.Host.
  A test that faked DNS through netboot.netutils.resolve patches
  netimps.resolve.

Fixed

• A configuration mistake is one line and exit 2, not a traceback: a MAC,
  address or network that does not parse (named targets.<id>,
  dhcpzones.<id>), a collection that is not a mapping, a key that is not
  text, a --load-module that does not import, and a zone whose only DHCP
  servers could not be reached (no DHCP server armed ...). A configured ip
  that is not an address is now that error instead of being dropped.
• A numeric MAC is refused. YAML reads an unquoted all-digit MAC
  (mac: 52:54:00:12:34:56, or such a target id) as a base-60 number, which
  netboot armed as a different MAC; quote it.
• repo / "sub" keeps a service's query string last
  (http://h/boot/sub/a?sig=1, not .../a?sig=1/sub), and leaves local
  unset when the repository had none instead of making it file:sub.
• A full-URI repository service no longer resolves the repository's address
  (nor warns that it has none); a relative one resolves it once, through
  netimps' shared cache, rather than on every URL.
• templates: dir given to Pixie directly is one root, not one per
  character, and parse_path reads every RFC 3986 scheme (svn+ssh://).
• pixie dhcp-config with no target warns about a target whose zone cannot
  be found and covers the rest, where it stopped at the first; a recipe
  several targets share is printed once. --as on Windows with standard
  input from NUL fails at once instead of waiting on a prompt.
• A global named like any member of the render context (version,
  render, searchpaths, ...) is dropped with a warning, as target and
  image already were, instead of replacing it or breaking the context.
• A zone with no dhcpservers warns that nothing was armed.
• hooks: None means no hooks; ctx.resource(Resource) for a resource with
  no path is None.
• A configuration key nothing reads (targetz:, a zone's gatway:) is
  logged as a warning naming it. Data for templates belongs under globals;
  a key starting with _ is never warned about.
• With DUHO_TRACEBACK set, a configuration error prints its traceback to
  standard error again: duho 0.7.6 leaves that to netboot, which now requires
  duho>=0.7.6.
• dnsmasq:// writes reservation and option files dnsmasq can read: each line
  is the value of --dhcp-host / --dhcp-option without the key, which
  dnsmasq rejected before, so no reservation took effect. A dnsmasq extras
  fragment is written the same way (tag:web01,252,http://x).
• dnsmasq:// in directory mode stops serving a completed target: dnsmasq
  keeps what --dhcp-hostsdir held until SIGHUP, so completion now signals
  pidfile= (default /run/dnsmasq/dnsmasq.pid), or needs reload= when the
  dnsmasq is remote. A dnsmasq target served options with no optsfile= is
  refused rather than armed without its boot file.
• dhcpd:// keeps a completed target (the host is deleted and created again,
  as dhcpd refuses to overwrite one), serves a conditions=group target its
  conditional boot file (the host's own no longer outranks the group's), and
  writes an option dhcpd has no name for in a form dhcpd parses.
• kea:// keeps a completed target with reservation-update (or a delete
  and add), where it sent a reservation-add Kea refuses for an existing
  reservation, and finds a zone's subnet inside a Kea shared-networks entry.
• windhcp:// completing a target that is removed also removes its own
  policies (netboot-<mac>-<condition>, netboot-<mac>), which went on serving
  its boot file; conditions=shared leaves an existing policy untouched;
  pixie dhcp-config prints the per-target policies target mode creates, every
  value quoted; method=netsh keeps a reservation through netsh and no longer
  fails its read-back for an option a condition serves.
• pixie initiate on a target that is already armed replaces its reservation
  on every DHCP server; dhcpd and Kea refused, and the Windows cmdlet threw.
• kea:// conditions are a client class of the target's own, so an iPXE
  client is served the chainload and any other client the reservation's boot
  file; on a real Kea both were handed the same one. dhcp-config prints the
  two classes per target (<name>-<mac> and netboot-<mac>). A Kea with no
  hosts database is armed with ?target=memory.
• Every DHCP option reaches dnsmasq, dhcpd and Kea in a form it reads: dhcpd
  refused the host's statements for most options netboot did not name, and
  dnsmasq read flags, number lists and octets as text.
• windhcp:// and a remote dnsmasq:// reload run through hostctl. ssh is
  asyncssh rather than the system ssh client (it reads ~/.ssh/config as
  ssh does), so windhcp:// over ssh now needs netboot[ssh]; ?timeout= bounds one
  Windows script (default 120 s).
• windhcp:// over its local and ssh transports runs the whole script and
  fails when any of it fails. It ran only the first line of each multi-line
  block and reported success, so a reservation could be reported armed and not
  exist.

Full Changelog: v0.3.2...v0.4.0