Need a way to support export scenario

[SteveL-MSFT]

Summary of the new feature / enhancement

Users want to manually configure an application/workload and get all the relevant configuration out and apply it to many other instances. Currently, get uses the supplied config as a filter. Also some resources like registry and file wouldn't make sense to enumerate the whole registry or filesystem.

Proposed technical implementation details (optional)

Propose an optional export method in addition to get, set, and test. dsc config export would ensure all the resources in the config support this method otherwise it fails.

[NikCharlebois]

Is the idea to have return type of the export be an array of hashtables from the Get or a string representation of the resulting current configuration in JSON/YAML?

[SteveL-MSFT]

Intent is that the output could be turned around and reused for get/set/test so needs to be a valid resource config. This means if it's an array of values, then the resource needs to have it's config as an array already.

Thinking about this further, it may also make sense to support the case where export would return multiple instances and dsc.exe would wrap each one as a config for the resource. It'll be up to the resource to decide which model works best.

[Bpoe]

Modeling this as a Restful API would make enumerating all instances of a given resource feel natural

[michaeltlombardi]

Thinking about this again today and considering the following use cases:

1. I want to retrieve the current state for all instances of a single resource (what IIS sites exist on this machine?) for inspection.
2. I want to retrieve the current state for all instances of a list of resources (What IIS sites and application pools exist on this machine, which features are installed) for inspection.
3. I want to get the current state for all instances in a group of resources so I can use the output as a starting point for further configuration.

Case 1

For this case, I think that to maintain backward compatibility with existing resources, it makes the most sense to assume that a resource can only return a single instance from a get call.

If the resource's manifest has a key that opts into a return-all functionality, like _enumerable or _supportsGetAll, then the caller knows they can use the get method on the resource without any additional input to retrieve state for all instances of that resource. Alternatively, with the --all parameter on the subcommand.

For example:

# Get a single instance of an IIS site
dsc resource get --name Site --module Microsoft.IIS --properties "name=foo"
# Get all IIS sites (no parameters)
dsc resource get --name Site --module Microsoft.IIS
# Get all IIS sites (indicating flag)
dsc resource get --name Site --module Microsoft.IIS --all

Case 2

For retrieving all instances for multiple resources, I think it would be reasonable for the caller to just loop calls to dsc.

foreach ($Resource in @('Site', 'ApplicationPool', 'Feature')) {
    dsc resource get --name $Resource --module Microsoft.IIS --all
}

Case 3

To enumerate the current state of an arbitrary set of resources on the target and get the return output as a configuration document, I think you're correct that the dsc config command needs an export subcommand. I don't think that the DSC Resource API surface semantics should be extended for an export method. If the resource has marked itself as supporting the retrieval of all instances of itself on a target, dsc knows whether it can export that resource. Similarly, that information can be advertised when authoring a configuration file directly.

I wonder if another approach, re-using the existing grouping model, is to have a built-in ExportGroup. This group's validation could fail any entry in it that doesn't support full enumeration.

An example of a configuration for this purpose might be:

# ./exportable.config.yaml
resources:
- name: my resources
  type: DSC/ExportGroup
  properties:
    resources:
    - name: Websites
      type: Microsoft.IIS/Site
    - name: App Pools
      type: Microsoft.IIS/ApplicationPools
    - name: IIS Features
      type: Microsoft.IIS/Feature
  dependsOn:
    - '[DSC/AssertionGroup]IsWindows'
- name: IsWindows
  type: DSC/AssertionGroup
  properties:
    resources:
    - name: os
      type: Microsoft/OSInfo
      properties:
        family: Windows
- name: Example Variable
  type: Microsoft.Windows/EnvironmentVariable
  properties:
    name: EXAMPLE
    value: An arbitrary string
    _ensure: present

And a user would call it with:

# Output to terminal
dsc config export --file ./exportable.config.yaml
# Capture in new config file
dsc config export --file ./exportable.config.yaml > webfarm.config.yaml

When you call dsc config export, any resources not included in a DSC/ExportGroup get ignored but appended to the output config. I can also see wanting to apply the assertions before export.

Aside

Another reason to support the "enumerate all instances of this resource" key in a resource manifest is to enable the caller to decide whether it wants to enumerate all of the instances of a resource at once or one at a time. Some resources might functionally have to enumerate the full list every time and filter it to only return the specified instance.

If a configuration includes 2-3 of those resources, the performance impact might not be noticeable. If the configuration includes 20 of those resources, knowing whether it can intelligently decide to pre-cache the instances would be useful. That also enables improved integration with higher-order tools that also have a model for enumerating the full list of instances for a resource.

It also makes DSC Resources more useful as investigative tools on a system.

Conclusion

I think adding a new config key for a resource to advertise whether it supports enumerating every instance of itself, an ExportGroup resource, and an export subcommand to dsc config would be a relatively neat solution to this problem without increasing the complexity or cognitive load for users and resource authors too much. Resource authors just need to indicate if their resource supports the operation, config authors need to know about the subcommand and resource group.

For a user who wants to enumerate the list of instances without worrying about the return information being a usable configuration document, calling dsc resource get [--all] without defining properties seems like a coherent UX that is similar to behavior in PowerShell, only filtering the list of resources if provided additional information or failing if the query requires mandatory information.

[SteveL-MSFT]

From discussion today, it seems there are multiple parts here:

• ok to use --all with get sub-command for dsc.exe
• reject any input JSON when --all is used, we can always support filtering in the future if there's a valid scenario
• for resources, it makes sense to have resources return JSONLINES instead of a single JSON object containing an array so that the instances don't need to be cached in memory to emit the entire object
• dsc.exe would collect the results and emit a valid configuration doc. We have two options:

resources:
  name: IIS
   properties:
     - website: A
       foo: bar
     - website: B
       foo: a
     - website: C
       foo: C

resources:
  type: IIS
   properties:
     website: A
     foo: bar
  type: IIS
   properties:
     website: B
     foo: a
  type: IIS
   properties:
     website: C
     foo: C

We agreed to go with the 2nd option pending customer feedback.

[SteveL-MSFT]

Thinking about this further, I think it may be better to start with an export operation rather than an --all switch to get as it may make it more clear to users that this is distinct from get since the semantics are very different and export would make more sense to use as declaring support in a resource manifest.

[michaeltlombardi]

I think dsc resource|config export is a very different operation from dsc resource get --all - they're related, insofar as only resources that can support --all can also be used for export - but definitely different use cases.

I'm not sure I see how we can support export without supporting --all - but --all doesn't require anything (I think?) but an entry in the resource manifest indicating what argument to pass to the resource to return all instances and wiring dsc resource get to handle that flag / return an array of instances when the flag is used.

Edit for clarification:

By this I mean that there's two operations a caller might want to perform on a resource:

1. I want to retrieve all instances of some resource R - to cache or to view, whatever. I want just the array of instances as return output.
2. I want to export the current state of some resource R - to use for baseline or reporting, etc. I want the DSC configuration document representing those resources. I might even want to export multiple resources at once.

This implies to me that the two operations are dsc resource get --all and config | dsc resource export. The export call likely uses the same code path as resource get --all, but they're distinct operations.

[mgreenegit]

The specific requirement for dsc resource export is limited in scope to known scenarios that support it. Two partners would be WinGet and M365Dsc. For example, you have written a JSON configuration that is intended to get the configuration details for 10 known applications (WinGet must already have an install/configure script).

@michaeltlombardi, good description of get all. I am pinging a few stakeholders that used it in the past for inventory.

[SteveL-MSFT]

@michaeltlombardi just like get/set/test, the export operation can be done at the config level or resource level. With dsc config export, you would get a valid config that can immediately be applied elsewhere. With dsc resource export, you would get JSONLINES.

[denelon]

We're looking at the same thing over in WinGet for:

• Generate configuration from machine state  microsoft/winget-cli#3274

And over at Dev Home for:

• Create DSC file from existing computer microsoft/devhome#685
• Option to configure from a file, with no way to export current configuration microsoft/devhome#721
• Create a config file based on an existing machine microsoft/devhome#876
• Include existing apps into Machine Configuration microsoft/devhome#1065

[michaeltlombardi]

@michaeltlombardi just like get/set/test, the export operation can be done at the config level or resource level. With dsc config export, you would get a valid config that can immediately be applied elsewhere. With dsc resource export, you would get JSONLINES.

To clarify, I don't mean that there's no value in export at the resource level, just that it's a distinct use case from get --all, even though the underlying mechanism for retrieving the full set of instances for a resource is the same (call the command with the correct flags).

Another thought, there's no way for extant PowerShell DSC Resources to participate in this - would that require an overload for Get or a new method?

[anmenaga]

Another thought, there's no way for extant PowerShell DSC Resources to participate in this - would that require an overload for Get or a new method?

That would require:

1. updating PSDesiredStateConfiguration module to, for example, when 'Get' operation is invoked without any filters - treat it as 'get-all' and pass it down to the resource;
2. correspondingly update the classic PS resource to accept the whatever mechanism is used in 1st point.