Plan for existing DSC 1.1 / 2.0 users to migrate to DSC 3.0

[Borgquite]

Summary of the new feature / enhancement

So @anmenaga suggested that I raise the questions here that I was asking on the DSC community call last night - here goes!

Currently I have a number of nodes managed with DSC 1.1 using Azure Automation State Configuration - all on-premises Windows Server nodes. The question was really around:

• What benefits should I expect from moving to DSC 1.1 to 3.0 - I understand that newer DSC resources could make this a requirement - but what are the driving forces for someone with this setup, apart from 'staying ahead of the curve'?
• What will moving from DSC 1.1 to DSC 3.0 look like, especially when it comes to the deprecation of the LCM? I currently have a lot of config which is in the 'classic' PowerShell compliation / MOF format - including looping through data stored in .PSD1 files to automate standing up many resources with small differences, and composite resources. Will there be migration tools available, how challenging is the migration expected to be in my scenario.

I'm familiar with the desire to move from MOF to JSON / YAML and why that makes sense, and 'get' how the new model will make it easier for tools Chef, Puppet, Ansible to integrate with DSC. But what's the benefits that those of us using DSC 1.1 in pull mode with the LCM can expect to reap? Just trying to warn about the sense of mere fire and motion.

Proposed technical implementation details (optional)

Making the move DSC 1.1 (including using the LCM) to 3.0 as seamless and pain-free as possible.

[SteveL-MSFT]

@Borgquite I converted this to a discussion as it's not really an issue (yet).

My expectation is that eventually we'll see more DSCv3 resources simply because they can be written in the language the developer is already familiar with and ideally, the application exe itself serves as the DSC resource rather than a separate thing. However, this won't happen for some time and in the interim, people can still use existing PowerShell based resources or author new ones in PowerShell. The shift from MOF to JSON/YAML means you can use much more tooling that is available and alignment with ARM Template syntax means we'll eventually have Bicep support (for those that want it), but also learning one gives you access to both.

My recommendation is that any configuration you have currently that is working to not try to migrate to 3.0. New configurations should consider being authored in the new 3.0 model. There is a separate discussion here about a potential migration PowerShell module for existing configurations.

LCM is currently not in the DSC 3.0 plan, but something similar might be discussed for 3.1+. However, in many cases Azure Machine Config (once they integrate with DSC3) will serve as your LCM.

It would be helpful for our potential LCM plans if you can be more specific on what features of LCM you rely on and the scenarios on how they are used.

[Borgquite]

Thanks @SteveL-MSFT. I guess for me, using Azure Machine Configuration to fulfil the functions of an LCM (i.e. get a config, apply it, return the result to the service) would probably work, given that I hope the cost and features would remain equivalent to Azure State Configuration pull server (I hope they would). I think my main concern would be over how to move existing 1.1 configurations over to 3.0 easily. I looked into the process for Machine Configuration for 2.0 and it looked 'feasable' (if I remember correctly, I would just need to do something about my composite configurations which are server-role based, but the foundation is still PowerShell / MOF based). My concerns were more over what I'm seeing in 3.0 re YAML - what does the workflow / migration process look like when moving from 1.1 to 3.0 Azure Machine Configuration when using long-supported and documented constructs to create a configuration - what tools and support are/will there be for transitioning existing 1.1 PowerShell scripts which use e.g. the following functionality to generate MOFs, to create equivalent DSC 3.0 compatible configurations, in JSON/YAML?

• Parameters
• Conditional statements and loops
• Composite / nested configurations
• Configuration data
• Separate configuration and environment data

[michaeltlombardi]

Regarding those features:

• Parameters are available in DSCv3 configuration documents via the parameters section and can be accessed in resource instances with the parameters() configuration function.

• Conditional statements are possible (may require a custom resource) with the Microsoft.DSC/Assertion resource and the dependsOn keyword for a resource instance, as shown in this example.

• There are no built-in configuration functions or group resources for loops at this time, but they could be reasonably implemented as a group resource (which I would prefer compared to extending the engine, personally). Essentially, you would define a resource that looks something like:

  - type: MyOrganization.Logic/ForEach
    name: Services
    properties:
      resources:
        - type: MyOrganization.MyApp/Service
          name: <name> service
          properties: {}
      data:
        - name: MyServiceFoo
          startupType: automatic
        - name: MyServiceBar
          startupType: automatic
        - name: ForbiddenService
          startupType: disabled

  And that would resolve in the nodegraph and act as a group resource to configure each of the resources defined in the data. This model above is just a quick sketch to give you the idea, any actual implementation would require a lot of thought and care around its model and behavior, but I don't see any reason a group resource couldn't handle this sort of thing. Plus, if the data is defined as a configuration parameter, you could separate the data from the foreach group more readily.

• Composite/nested configurations can be mimicked with the Microsoft.DSC/Include resource, which resolves another configuration document into the current one.

• Configuration data is a little mixed - on the one hand, you can argue that parameters fulfill the role. On the other hand, because configuration documents are (arguably primarily) static, you can't compile different configuration graphs based on that data without a pre-processing step. Accessing different data per-node, definitely possible with parameters. Different resource instances would be more difficult.

  You could handle this with a pre-processing step, generating however many different configuration documents you need. You could also use assertion groups and dependsOn to get something similar, but the functionality would be a little messy.

  In my opinion, this feature is better handled by higher order tooling, like Machine Configuration - once I'm trying to represent many nodes with differing requirements, I probably want all the other things that come with higher order tooling. If I couldn't use a higher-order tool, I would instead likely decompose my configurations and use the Microsoft.Dsc/Include resource to define the documents that actually configure my machines¹.

Footnotes

1. Actually, I would probably do this anyway from the perspective of testing and maintaining my infrastructure. I'm a bit removed from ops now, but being able to separately version, document, and test chunks of my configuration model was a big deal to me and something that the folks I supported when I was at Puppet struggled with. ↩

[Borgquite]

OK - it's becoming clear to me that this discussion is mis-titled (now corrected) - it seems like my main issue with migration would be as a result of configuration documents needing to be defined in JSON or YAML - using PowerShell scripting / configuration / data files is no longer be supported.

What is the proposed migration path for someone currently using DSC 1.1, or 2.0, when moving to DSC 3.0? Is it basically 'rewrite every configuration in a third-party tooling, using YAML / JSON'? I'm really not clear what the value proposition for people with existing DSC 1.1 or 2.0 configurations lies here? Again, are you risking just consuming our time with fire and motion?

It makes sense to me to try to remove the MOF dependency - however what's wrong with maintaining the ability to create configuration documents using the existing PowerShell script syntax - and instead of generating MOF, the output is valid YAML/JSON instead? If you don't want to do that, take the MOF output of a 1.1 / 2.0 configuration document, and allow us to convert it into DSC 3.0 complaint YAML/JSON - surely that's also possible?

I am struggling to see the benefit of rewriting PowerShell configuration documents written for DSC 1.1 or even 2.0 (as currently recommended by Microsoft right now) in an entirely new language, when the only upside I'm being presented with is 'you'll be able to access new DSC resources written in non-PowerShell language, although none of them have actually been written yet'. How do you plan to get critical mass around this without risking losing / forking the existing community?

[kilasuit]

@SteveL-MSFT I know you likely haven't but it worth mentioning that we shouldn't be forgetting about the usecases including disconnected environments and non-azure accessible environments including AWS, GCP or other providers including private clouds especially where we want to sequence the order of events in such a way that allows for the best use of available resources in resource constrained environments (like Developer machines)

Also thanks for the comments in the DSCCommunity call on 19th March around this.