Adapter redesign

[michaeltlombardi]

Summary of the new feature / enhancement

As a DSC user, I want to be able to use adapted resources with a configurable adapter so that I can control how the adapter works on my systems.

As a DSC user, I want to be able to invoke adapted resources more quickly so that I'm not routinely waiting for long periods accounting for the overhead of an adapter performing discovery, setup, and clean up every time I need to invoke a resource interactively or in a configuration document.

As an adapter developer, I want to enable defining my adapter with configurable options and a long-running mode that integrates with DSC and higher order tools through well-defined contracts.

In DSC 3.2, we redesigned the implementation of adapters (see #1331) as resources to support always passing a single resource instance to the adapter for invocation. This simplified the handling for the adapters and made defining adapted resource instances in configuration documents without using the nested instance model much easier. This made configuration documents easier to read, reason about (particularly for the DAG), and made reviewing the output for operations against adapted resources equivalent to non-adapted resources.

However, this model has the drawbacks that caused us to initially implement the adapters as resources that accept nested instances - particularly that the overhead per invocation for adapted PSDSC resources (_especially with the Windows PowerShell adapter) is very slow.

When we originally designed adapters, we did so before we had any meaningful extension APIs. Now that we have decided to move away from supporting nested instances for adapters - adapters in the recommended single mode model never show up directly in the configuration document except when pinning to a specific adapter through the requireAdapter directive - we should fully reconsider the design for adapters.

In my opinion, adapters no longer really make sense as a special kind of resource. They are more similar effectively to extensions. However, rather than present adapter definitions through an extension manifest, I think it makes more sense to fully separate them from both resources and extensions into their own specific manifest.

Initially, we could make the new model of adapter manifests behave identically to the way the new PowerShell adapters work (expect to be invoked once for every resource, no long-running process).

The following list of enhancements to adapters is meant to provide an overview of how migrating from adapters-as-resources to adapters-as-own-domain-contract enables improvements that are much more difficult in the adapter-as-resource model. However, this functionality is explicitly out of scope for the initial implementation. Each feature described in the following list would need it's own issue and design considerations:

Potential features for adapters

• Add a discovery caching API.

  This would enable users to do things like warm and clear the cache programmatically through DSC instead of needing to review the adapter documentation or implementation to figure out how to manage the adapter's resource discovery cache.

  For example, users could call the following commands:

  • dsc adapter cache refresh --adapter Microsoft.Adapter/PowerShell - tell the adapter to search for resources and update its cache.
  • dsc adapter cache clear --all - tell every adapter with the cache capability to clear its cache.
  • dsc adapter cache init --adapter Microsoft.Adapter/PowerShell - tell the adapter to to discover and cache resources if the cache isn't already setup.

• Add configurable options for the adapter.

  This would address issues like being able to specify alternative locations for adapted resources, stricter requirements, and so on. In the current model, there isn't a coherent way to pass options to the adapter. An adapter developer could potentially make options work through passing metadata on resource invocations, but that's complex, fragile, difficult to discover, and would need to be done for every resource instance in a configuration document.

  These options could then be set by DSC resources (Microsoft.Adapter.PowerShell/Options), commands (dsc adapter options set), in a configuration document (directives.adapters at the top level, directives.adapterOptions at the resource instance level), or as a flag on resource invocation commands (--adapter-options).

• Add a long-running mode capability.

  Currently adapters are spun up and down on every adapted resource invocation. We could design a contract and API that would enable DSC to spin up an adapter at the beginning of a configuration and then send operations to the adapter for instances as needed. This would drastically help with invocation speed.

  Further, we could eventually support spinning up an adapter that continues to live external to specific DSC invocations. A user might run a command like dsc adapter start --adapter Microsoft.Adapter/PowerShell to spin up the adapter and cache the connection information, then have DSC use that adapter for subsequent operations.

  While I think it likely makes sense to provide a long-running mode for extensions (for example, secret extensions where you might retrieve a secret n times in a configuration) and resources (obviating the need for performance-centric *List resources), I think adapters are likely the highest return on development for this functionality and a good place to experimentally define this API before replicating it to other DSC components.

Finally, there are now components of the resource manifest that aren't coherent for adapters, like the required schema field - new-style adapters don't have an instance schema. Other resource-specific capabilities also don't always make sense for an adapter, like SetHandlesExist or Resolve. Separately defining adapters from resources is more semantically correct and will make defining the data contract (as well as documenting for adapter users and developers) simpler and more coherent.

Proposed technical implementation details (optional)

The following changes are a non-exhaustive list of updates needed to reimplement adapters as their own kind of DSC component:

• Define a new adapters module with the AdapterManifest and Adapter structs.
• Update discovery to handle (until 4.0) both adapter resources and adapters.
• Define a new adapters command with the list subcommand to return all known adapters. Initially, this should probably also return legacy adapter resources in the output, which will require implementing From<DscResource> for Adapter.
• Migrate the PowerShell adapters to the new design.

The following snippet is a (simplified) gesture towards the definition for an adapter manifest:

pub struct AdapterManifest {
    /// The version of the resource manifest schema.
    pub schema_version: String,
    // The fully qualified type name for the adapter that uniquely identifies it.
    pub type_name: FullyQualifiedTypeName,
    /// An optional condition for the adapter to be active.  If the condition
    /// evaluates to false, the adapter is skipped.
    pub condition: Option<String>,
    /// An optional message indicating the adapter is deprecated.  If provided,
    /// the message will be shown when the adapter is used.
    pub deprecation_message: Option<String>,
    /// The semantic version of the adapter.
    pub version: SemanticVersion,
    /// A short description for the adapter.
    pub description: Option<String>,
    /// Tags for the adapter.
    pub tags: TagList,
    /// Indicates how to invoke the adapter to find and list adapted resources.
    pub discovery: AdapterDiscovery,
    /// Indicates how to invoke the adapter to validate adapted resource
    /// instances.
    /// 
    /// This field is required for adapters that don't provide a resource schema
    /// for adapted resources.
    pub validation: Option<AdapterValidation>,
    /// Indicates how to invoke the adapter to retrieve the current state of an
    /// adapted resource instance.
    pub get: AdapterGetOperation,
    /// Indicates how to invoke the adapter to test whether an adapted resource 
    /// instance is in the desired state.
    /// 
    /// When an adapter doesn't define this operation, DSC _always_ uses
    /// synthetic testing for adapted resources through this adapter.
    pub test: Option<AdapterTestOperation>,
    /// Indicates how to invoke the adapter to idempotently enforce the desired
    /// state for an adapted resource instance.
    /// 
    /// When an adapter doesn't define this operation, users can't modify system
    /// state with adapted resources through this adapter.
    pub set: Option<AdapterSetOperation>,
    /// Indicates how to invoke the adapter to remove an adapted resource
    /// instance.
    /// 
    /// When an adapter doesn't define this operation, users can't call the
    /// `dsc resource delete` command for adapted resources through this
    /// adapter and DSC will always invoke the `set` operation to enforce the
    /// desired state for adapted resource instances, even when the adapted
    /// resource is implemented to support have the `setHandlesExist`
    /// capability.
    pub delete: Option<AdapterDeleteOperation>,
    /// Defines a mapping of exit codes to descriptions for the adapter.
    pub exit_codes: ExitCodesMap,
    /// Defines arbitrary metadata to include for the adapter.
    /// 
    /// DSC doesn't interpret this metadata; it's purely informational to the
    /// engine.
    pub metadata: Option<ManifestMetadata>,
}

Separately defining the struct lets us be much more specific about the contract for an adapter. For example, we could decide to make supporting all resource operations non-optional and require the adapter to define the what-if argument for set and delete invocations.

It also makes it easier for us to extend the manifest to support new adapter capabilities without affecting resource manifests.