Dynamics 365 CE DevOps tools

Build and release tooling for Microsoft Dynamics 365 Customer Engagement and Dataverse — solution and configuration-data transport, plugin assembly and package updates, web resource deployment, JavaScript minification and assembly obfuscation.

Works against Dynamics 365 Online / Dataverse and Dynamics CRM on-premises.

The same engine ships three ways:

• Install
• Pipeline tasks
• Connection strings
• Shuffle in depth
• PowerShell cmdlets
• Build from source
• Credits

Install DevOps for Microsoft Dynamics 365 into your Azure DevOps organisation. The tasks then appear in the task picker prefixed D365CE DevOps:, and are referenced from YAML by their task name (see below).

Tasks run on the PowerShell3 handler and need a Windows agent.

These are not in the XrmToolBox Tool Library. Download the release from rappen/Xrm.Shuffle, unblock the zip before extracting (right-click → Properties → Unblock), and drop the .dll into the Plugins folder of your XrmToolBox installation. Both tools then appear in your toolbox.

• Shuffle Builder — a no-code editor for the definition file: which entities, which attributes, in what order, with which filters and relationships.
• Shuffle Runner — executes a definition against the connected environment, exporting or importing.

git clone --recurse-submodules https://github.com/imranakram/Innofactor.Crm.CI.git
cd Innofactor.Crm.CI

See Build from source.

All nine tasks, with their YAML task name and major version:

Store the connection string as a secret variable and pass it as $(CrmConnectionString). Never inline credentials in YAML.

Export from dev, then import into test:

trigger:
  branches:
    include: [ main ]

pool:
  vmImage: windows-latest

variables:
  - group: dynamics-connections   # holds DevConnectionString / TestConnectionString as secrets

steps:
  # Fail fast with a clear error if the connection is wrong
  - task: WhoAmI@9
    displayName: Verify DEV connection
    inputs:
      crmConnectionString: $(DevConnectionString)

  - task: ShuffleExport@9
    displayName: Export solution and config data from DEV
    inputs:
      crmConnectionString: $(DevConnectionString)
      definitionFile: $(Build.SourcesDirectory)/shuffle/release.xml
      dataFile: $(Build.ArtifactStagingDirectory)/release.data.xml
      setVersion: true

  - publish: $(Build.ArtifactStagingDirectory)
    artifact: shuffle

  - task: ShuffleImport@9
    displayName: Import into TEST
    inputs:
      crmConnectionString: $(TestConnectionString)
      definitionFile: $(Build.SourcesDirectory)/shuffle/release.xml
      dataFile: $(Build.ArtifactStagingDirectory)/release.data.xml

If dataFile is omitted it defaults to the definition file path with the extension changed to .data.xml — so release.xml pairs with release.data.xml.

steps:
  # Stamps the build number into every AssemblyInfo.cs it finds
  - task: ApplyVersionToAssemblies@9
    inputs:
      versionType: build            # 'build' parses $(Build.BuildNumber); 'file' reads versionFile
      workingDirectory: $(Build.SourcesDirectory)/src
      versionMatch: false           # true replaces the first two digits with the CRM SDK
                                    # version found in packages.config

  - task: VSBuild@1
    inputs:
      solution: '**/*.sln'
      configuration: Release

  # Minifies every *.maxi.js in the folder to the same name without '.maxi'
  - task: MinifyJS@8
    inputs:
      jsPath: $(Build.SourcesDirectory)/webresources/scripts

  - task: ObfuscateAssembly@9
    inputs:
      assembly: $(Build.SourcesDirectory)/src/Plugins/bin/Release/My.Plugins.dll
      level: '2'                    # 0 lightest .. 4 extreme; costs size and performance
      key: $(Build.SourcesDirectory)/src/Plugins/My.snk

  - task: UpdateAssembly@9
    inputs:
      crmConnectionString: $(CrmConnectionString)
      assembly: $(Build.SourcesDirectory)/src/Plugins/bin/Release/My.Plugins.dll
      updateManaged: false          # true to allow updating a managed assembly

  - task: UpdatePackage@9
    inputs:
      crmConnectionString: $(CrmConnectionString)
      packageName: My.Plugin.Package
      packageFile: $(Build.SourcesDirectory)/src/Package/bin/Release/My.Plugin.Package.nupkg
      updateManaged: false

UpdateAssembly and UpdatePackage update an existing registration — the assembly or package must already be registered in the target organisation.

  - task: UpdateWebResources@9
    inputs:
      crmConnectionString: $(CrmConnectionString)
      prefix: new                   # must match the publisher prefix of the existing resources
      rootPath: $(Build.SourcesDirectory)/webresources
      pattern: $(Build.SourcesDirectory)/webresources/pattern.txt
      updateManaged: false

rootPath is the local folder corresponding to the web resource root in CRM. pattern points at a text file of include/exclude globs, one per line, where ! negates. This pairs neatly with MinifyJS — ship the minified output and skip the sources:

scripts\*.js
!**\*.maxi.js
images\*.*
!scripts\test.js

Web resources must already be registered.

Every CRM-facing task takes a standard Dynamics 365 connection string. The AuthType decides which authentication stack is used underneath.

Online — service principal (recommended for pipelines):

AuthType=ClientSecret;Url=https://contoso.crm4.dynamics.com;ClientId=<app id>;ClientSecret=<secret>

Online — username and password:

AuthType=OAuth;Url=https://contoso.crm4.dynamics.com;Username=svc@contoso.com;Password=<pw>;AppId=51f81489-12ee-4a9e-aaae-a2591f45987d;RedirectUri=app://58145B91-0C36-4500-8554-080854F2AC97;LoginPrompt=Never

On-premises — Active Directory:

AuthType=AD;Url=http://crmserver/contoso;Domain=CONTOSO;Username=svcaccount;Password=<pw>

On-premises — internet-facing deployment:

AuthType=IFD;Url=https://contoso.crm.local/contoso;Domain=CONTOSO;Username=svcaccount;Password=<pw>

Why the ADAL version is pinned. Microsoft.Xrm.Tooling.Connector 4.0.0.0 — from Microsoft.CrmSdk.XrmTooling.CoreAssembly 9.1.1.65, the newest published — is compiled against Microsoft.IdentityModel.Clients.ActiveDirectory 3.19.8.16603 exactly. Strong-name binding then demands that exact version, and overriding it requires a binding redirect in the host process config (powershell.exe.config on the agent), which an extension cannot modify. A library .dll.config is ignored — that is why no .config ships in the package. So the pin is a requirement, not a workaround.

This is not a PowerShell limitation: ADAL 5.x loads into PS 5.1 perfectly well on its own. And since no newer XrmTooling binds a newer ADAL, the only way off ADAL 3.x is off CrmServiceClient altogether — which Microsoft's transition guidance advises against while on-prem AD/IFD support is required, since Microsoft.PowerPlatform.Dataverse.Client never implemented it.

Shuffle moves solutions and configuration data between environments, driven by a single declarative XML definition. Its stand-out capability is handling N:N (many-to-many) relationships and cross-entity lookups that most data-migration tools cannot.

A run has two halves:

• Export reads from the source environment and writes a data file (*.data.xml).
• Import reads that data file and writes into the target environment.

The same definition file drives both. The definition says what the data is and how to match it; the data file is only the payload.

<ShuffleDefinition Timeout="600" StopOnError="true">
  <Blocks>
    <SolutionBlock ... />
    <DataBlock ... />
    <!-- blocks execute top to bottom, in document order -->
  </Blocks>
</ShuffleDefinition>

Order matters. Blocks run in the order they appear, and later blocks can reference earlier ones. Export parents before children, and both participants of an N:N before the intersect.

<DataBlock Name="Categories" Entity="category" Type="Entity">
  <Export ActiveOnly="false"> ... </Export>
  <Import CreateWithId="true" Save="CreateUpdate"> ... </Import>
  <Relation Block="..." Attribute="..." />
</DataBlock>

A block may have Export, Import, or both. Without Export the block contributes nothing to the data file; without Import it is exported but never written to the target.

Either an explicit attribute list or raw FetchXML — not both.

<Export ActiveOnly="true">
  <Filter Attribute="statuscode" Operator="Equal" Type="int" Value="3" />
  <Sort Attribute="title" Type="Asc" />
  <Attributes>
    <Attribute Name="title" />
    <Attribute Name="content" />
    <Attribute Name="statecode" />
    <Attribute Name="statuscode" />
    <Attribute Name="keywords" IncludeNull="true" />
  </Attributes>
</Export>

Do not export system-maintained attributes such as createdon, createdby, modifiedon or modifiedby — the import will fail. If you export statuscode you must also export statecode, since the pair is validated together.

<Import CreateWithId="true" Save="CreateUpdate" Delete="None"
        UpdateInactive="false" UpdateIdentical="false">
  <Match PreRetrieveAll="true">
    <Attribute Name="title" />
  </Match>
</Import>

Match is what makes an import idempotent. It defines how a record in the data file is recognised in the target:

• Match on one or more attributes — a title, or a business key such as new_code.
• With CreateWithId="true" you can match on the primary id instead.
• No Match element means every run creates new records.
• PreRetrieveAll="true" pre-loads target records in one query rather than querying per record — much faster for large sets.

Save="Never" is the idiom for a block that exists only so other blocks can resolve references to it. The matching rules stay available but nothing is written — exactly what you want for the two sides of an N:N whose records already exist in the target.

Filters a block's export by its relationship to an already-exported block, so you take only the children of the parents you actually exported:

<DataBlock Name="Tasks" Entity="task">
  <Export> ... </Export>
  <Relation Block="Accounts" Attribute="regardingobjectid" />
</DataBlock>

The case most tools cannot handle. Three blocks, in this order:

1. Both participating entities, as ordinary Entity blocks.
2. An Intersect block for the relationship itself.

<!-- 1. Both sides first. Save="Never" if they already exist in the target -->
<DataBlock Name="Articles" Entity="knowledgearticle" Type="Entity">
  <Export>
    <Attributes><Attribute Name="title" /></Attributes>
  </Export>
  <Import CreateWithId="true" Save="Never">
    <Match><Attribute Name="title" /></Match>
  </Import>
</DataBlock>

<DataBlock Name="Categories" Entity="category" Type="Entity">
  <Export>
    <Attributes><Attribute Name="title" /></Attributes>
  </Export>
  <Import CreateWithId="true" Save="Never">
    <Match><Attribute Name="title" /></Match>
  </Import>
</DataBlock>

<!-- 2. Then the relationship. Export exactly the two id columns, nothing else -->
<DataBlock Name="ArticleCategories" Entity="knowledgearticlecategories"
           Type="Intersect" IntersectName="knowledgearticlecategories">
  <Export>
    <Attributes>
      <Attribute Name="knowledgearticleid" />
      <Attribute Name="categoryid" />
    </Attributes>
  </Export>
  <Import />
</DataBlock>

Two rules that catch people out:

• The intersect's Export must list exactly the two id attributes of the related entities.
• The intersect's Import takes no Match element. Shuffle Builder adds one automatically; delete it.

<SolutionBlock Name="MySolution" Path="solutions" File="MySolution.zip">
  <Export Type="Managed" PublishBeforeExport="true" SetVersion="{ShuffleVar:version}">
    <Settings Customization="true" General="true" />
  </Export>
  <Import Type="Managed"
          ActivateServersideCode="true"
          OverwriteCustomizations="true"
          PublishAll="true"
          OverwriteSameVersion="true"
          OverwriteNewerVersion="false">
    <PreRequisites>
      <Solution Name="BaseSolution" Comparer="ge" Version="2.1.0.0" />
    </PreRequisites>
  </Import>
</SolutionBlock>

Export — Type is Managed, Unmanaged, Both or None. PublishBeforeExport publishes customisations first. SetVersion stamps a version into the solution before export; TargetVersion sets the target CRM version. The optional Settings element selects which system settings travel with the solution (AutoNumbering, Calendar, Customization, EmailTracking, General, Marketing, OutlookSync, RelationshipRoles, IsvConfig — all default false).

Import — Type, ActivateServersideCode, OverwriteCustomizations and PublishAll are all required. OverwriteSameVersion defaults true, OverwriteNewerVersion defaults false. PreRequisites gates the import on other solutions being present, where Comparer is one of any, eq-this, ge-this, eq, ge. PostSuccessfulImportBlocks nests further blocks that run only when the import succeeded — handy for seeding configuration data right after a solution lands.

If a definition contains the literal placeholder {ShuffleVar:version} and the ShuffleExport task runs with setVersion: true (the default), the task substitutes it with the contents of version.txt from $(Agent.BuildDirectory) before exporting. That is how a pipeline stamps the build number into an exported solution:

<Export Type="Managed" SetVersion="{ShuffleVar:version}" />

The ShuffleExport pipeline task always writes SimpleWithValue: compact XML that also records clear-text labels for lookups and option sets, so a data file stays readable and reviewable in a pull request. The Export-CrmShuffle cmdlet lets you choose:

• Keep definitions small and focused — one per concern (10–15 entities is a comfortable ceiling) rather than one definition covering everything.
• Commit both the definition and the exported data file. With Sort set and SimpleWithValue serialization, configuration changes show up as reviewable diffs.
• Adding an attribute to a definition later is fine — re-export and the data file picks it up.
• Run WhoAmI as the first step of any pipeline that touches CRM, so a bad connection string fails immediately with a clear message.

The Innofactor.Crm.CI module underlies the pipeline tasks and can be used directly. Every cmdlet takes -ConnectionString and an optional -Timeout in seconds.

Import-Module .\Cmdlets\FindCrmUser\bin\Release\Innofactor.Crm.CI.dll

Import these under Windows PowerShell 5.1. They are .NET Framework 4.6.2 assemblies built against the PowerShell 5 reference assemblies — the same host an Azure DevOps agent uses.

$conn = 'AuthType=ClientSecret;Url=https://contoso.crm4.dynamics.com;ClientId=...;ClientSecret=...'

Find-CrmUser -ConnectionString $conn

[xml]$definition = Get-Content .\shuffle\config.xml
$data = Export-CrmShuffle -ConnectionString $conn `
                          -Definition $definition `
                          -Folder .\shuffle `
                          -Type SimpleWithValue
$data.Save('.\shuffle\config.data.xml')

Requires Visual Studio 2022 or later (or MSBuild), the .NET Framework 4.6.2 targeting pack, and NuGet. The Shuffle core arrives as a git submodule, so clone with --recurse-submodules.

nuget restore Innofactor.Crm.CI.sln
msbuild Innofactor.Crm.CI.sln /t:Rebuild /p:Configuration=Release

Always clean first — a stale bin/Release is how removed dependencies keep getting shipped, because pack.ps1 copies *.dll out of it wholesale.

Get-ChildItem Cmdlets -Directory | ForEach-Object {
  foreach ($sub in 'bin','obj') {
    $dir = Join-Path $_.FullName $sub
    if (Test-Path $dir) { Remove-Item $dir -Recurse -Force }
  }
}
msbuild Innofactor.Crm.CI.sln /t:Rebuild /p:Configuration=Release
.\Extension\pack.ps1

pack.ps1 stages each task's ps_modules, downloads the VstsTaskSdk, and calls tfx to build (and optionally publish) the .vsix into Extension/VSIX.

./tests/Run-Tests.ps1
./tests/Run-Tests.ps1 -VsixPath ./Extension/VSIX/<built>.vsix

The suite gates the packaged artifact — no removed dependencies riding along, ADAL still pinned, task ids and versions never regressing, and every documentation image still resolving. See tests/README.md. A green run against a freshly built .vsix is the release gate.

To verify a real connection (online or on-prem), run tests/Test-Connection.ps1 from a Windows PowerShell 5.1 window - see tests/README.md.

For publishing steps and the rollback procedure, see RELEASING.md.

Cmdlets/          PowerShell cmdlet projects (one assembly per task)
Extension/        Azure DevOps extension: manifest, task definitions, packaging
  Implementation/   one folder per task - task.json + script.ps1
  Documentation/    Marketplace overview and images
modules/          Xrm.Shuffle submodule - the Shuffle core, upstream at rappen/Xrm.Shuffle
tests/            Pester suite

Shuffle was created by Jonas Rapp, who maintains the core engine and the XrmToolBox tools at rappen/Xrm.Shuffle. The DevOps tooling around it grew out of work at Cinteros and later Innofactor, and is now maintained by Imran Akram.

Background reading:

• Jonas Rapp — DevOps I: background, II: build and release tasks, III: end-to-end demo, public preview
• Sara Lagerquist — Transport data between environments, a walkthrough of Shuffle Builder and Runner covering knowledge articles, categories and the N:N between them. Written against an older release, so the screenshots and download location differ from the current tools, but the approach still holds.

Changes to the Shuffle core (modules/Xrm.Shuffle) belong upstream in rappen/Xrm.Shuffle — that is the copy that compiles. Everything else goes here. Please run ./tests/Run-Tests.ps1 before opening a pull request.

See LICENCE.