feat: comprehensive CMT support, schema management, changeset staging, and performance optimizations#25
Conversation
Phase 1: Enhanced txc data package import with optional parameters: - --batch-mode: enable ExecuteMultiple/UpsertMultiple batching - --batch-size: records per batch (default 600) - --override-safety-checks: skip duplicate detection, always create - --prefetch-limit: max records to preload into cache (default 4000) - --delete-before-import: delete existing records before importing Phase 2: New txc data package export command: - txc data package export --schema <path> --output <path> - --export-files: include binary file/image columns - --overwrite: overwrite existing output file - Full subprocess isolation via LegacyAssemblyHostSubprocess - ExportCrmDataHandler accessed via reflection (runtime-resolved assembly) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add ConfigurationMigration.Wpf NuGet PackageDownload for ExportProcessor DLL - Copy ExportProcessor and BatchedTelemetry DLLs as content (not compile-time refs) - Patch PE32+ (x64) assemblies to AnyCPU via Cecil for ARM64 compatibility - Add deferred assembly loading in LegacyAssemblyRuntime for assemblies whose eager load fails due to unresolved dependencies - Use NoInlining on RunCmtExportCoreAsync to defer JIT resolution of CmtExportRunner until after LegacyAssemblyRuntime is initialized - CmtExportRunner uses pure reflection (no compile-time ExportProcessor dependency) Tested: export + import round-trip against org2928f636.crm.dynamics.com Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both CmtImportRunner and CmtExportRunner now call SetTraceLevel on the CMT TraceLogger — Verbose when --verbose is used, Information otherwise. Without this, CMT's internal logger may filter out detailed diagnostic messages. Mirrors the PAC CLI pattern in DataVerbBase.ListenToDataMigrationLogging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The --connection-count option was passing a List<object> to the ImportConnections property which expects Dictionary<int, CrmServiceClient> with 1-based connection indices. Fixed to build the correct dictionary type. Tested: --connection-count 2 now works correctly against live org. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ch original behavior Two bugs fixed: 1. GetDataByKeyFromResultsSet<T> — the original CrmServiceClient does an 'is T' check and returns default(T) when the types don't match. Our shim did a direct (T)cast which threw InvalidCastException when CMT called GetDataByKeyFromResultsSet<Guid> on a string field (e.g. primarynamefield). Now matches the original: is-check, PICKLIST handling, _Property fallback, swallowed exceptions. 2. EntityToDictionary — the original stores two entries per attribute: 'key' → FormattedValue (string) or raw value, and 'key_Property' → KeyValuePair<string,object> with the raw SDK type. GetDataByKeyFromResultsSet uses the _Property fallback to get typed values when the primary entry is a formatted string. Our shim only stored raw values without _Property entries, breaking the fallback chain. Also removed --delete-before-import flag — decompilation confirmed that CMT's ImportDataToCrm accepts deleteBeforeAdd as a parameter but never uses it internally (the delete functionality was never implemented). Improved CLI help descriptions to be clearer and warn about dangerous options. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…riginal CrmServiceClient Systematic comparison against decompiled original CrmServiceClient from Microsoft.Xrm.Tooling.Connector revealed 17 bugs (2 critical, 7 high, 5 medium, 2 low). All fixed: CRITICAL: - LogicalSearchOperator enum: And/Or values were swapped (And=1→2, Or=2→1) causing ALL search queries to use inverted filter logic - EntityCollectionToDictionary: used entity.Id as dict key instead of Guid.NewGuid() — duplicates silently lost in M2M queries HIGH: - Missing ReturnProperty_EntityName and ReturnProperty_Id entries in entity dicts - Empty dict returned instead of null for empty results (CMT null-checks these) - Customer/Lookup fields ignored ReferencedEntity — created invalid EntityReferences - CrmFieldType enum missing Raw value - BuildQueryExpression missing wildcard (%→Like) and null (→Null) support - GetEntityDataById returned empty dict instead of null on failure - GetEntityDataByLinkedSearch incorrect M2M link chain + missing reflexive support MEDIUM: - EntityToDictionary synthetic ID entry removed (replaced by ReturnProperty_Id) - ImportStatus enum missing Failed value - NoLock=true added to all query expressions - CreateNewRecord/UpdateEntity now use Request objects with SuppressDuplicateDetection - InjectPagingIntoFetchXml rewritten to use XmlDocument (no double-encoding) LOW: - GetMyCrmUserId now caches result - ImportFileItem.RecordOwner changed from string to Guid Tested: import (standard + batch), export all pass against live org. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The original CrmServiceClient.AddValueToPropertyList skips File, Image and Key types (no-op break). CMT handles file uploads separately via UpdateFileColumns → UploadFileAttribute after record create/update. Our shim was passing FileData objects through to the entity which would cause Dataverse to reject the request. Now skips these types to match the original behavior. Also removed Key from ConvertCrmDataTypeValue since it's now skipped at the PopulateEntityFromDataTypeWrappers level. Data type analysis confirmed all 15 CMT-supported types work correctly: string, number, datetime, decimal, float, money, bool, guid, optionsetvalue, lookup/customer/owner/entityreference (all → Lookup), and Raw types (partylist, status, state, optionsetvaluecollection, imagedata). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add two new subcommands to 'txc environment data record': - download-file: downloads a file/image column value to a local file - upload-file: uploads a local file to a file/image column on a record Both use the Dataverse block-based file API (InitializeFileBlocksDownload/ Upload, DownloadBlock, UploadBlock, CommitFileBlocksUpload) with 4MB chunks. New files: - IDataverseFileService interface in Core/Contracts - DataverseFileService implementation in Platform.Dataverse.Data - EnvDataRecordDownloadFileCliCommand - EnvDataRecordUploadFileCliCommand Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add docs/configuration-migration.md covering all txc data package commands: - Overview, quick start, and full command reference (export/import/convert) - Schema file (data_schema.xml) and data file (data.xml) XML reference - Supported data types with serialization formats - File and image column support - Import tuning options (connections, batching, prefetch, safety checks) - Deduplication deep-dive (preprocessing, tiered matching, two-pass import) - Date handling modes (absolute, relative, relativeDaily) - Known limitations and gotchas from decompilation analysis - Troubleshooting guide with common errors and performance checklist Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ExportCrmDataHandler.ExportData() re-reads _isExportFilesEnabled from AppSettings["ExportFiles"] internally, overwriting any value set via reflection. Without an app.config the value defaults to false, silently ignoring the --export-files flag. Fix: set ConfigurationManager.AppSettings before calling ExportData/ ImportDataToCrm so the handlers pick up the correct values. Remove the now-unnecessary TrySetExportFilesEnabled reflection method. Keep the existing handler property sets in CmtImportRunner as belt-and-suspenders. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New commands for many-to-many (N:N) relationship operations: - txc environment data record associate <id> --entity --target --target-entity --relationship - txc environment data record disassociate <id> --entity --target --target-entity --relationship Uses modern Dataverse SDK Associate/Disassociate directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…hip list commands Implement three new CLI commands: - txc environment entity attribute create: Creates columns of various types (lookup, choice, multichoice, string, number, money, bool, datetime, decimal, float, image, file) with automatic entity publishing - txc environment entity relationship create: Creates N:N relationships between two entities with associated menu configuration - txc environment entity relationship list: Lists all relationships (1:N, N:1, N:N) for an entity with table/JSON output Extends IDataverseEntityMetadataService with CreateAttributeAsync, CreateManyToManyRelationshipAsync, and ListRelationshipsAsync methods. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…not 'lookup' Testing revealed that CMT's schema validator rejects type="lookup" and type="customer". The correct schema type for all lookup-like fields (lookup, customer, owner, entityreference) is type="entityreference". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… delete commands Add five new CLI commands: - txc environment entity create - txc environment entity delete - txc environment entity attribute update - txc environment entity attribute delete - txc environment entity relationship delete Service layer methods added to IDataverseEntityMetadataService and DataverseEntityMetadataService with publish-after-modify pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add four new commands under 'txc environment entity optionset': - create-global: Create a new global option set with parsed options CSV - add-option: Insert an option value into a local or global option set - delete-option: Remove an option value from a local or global option set - list-global: List all global option sets with table/JSON output New IDataverseOptionSetService interface and DataverseOptionSetService implementation using CreateOptionSetRequest, InsertOptionValueRequest, DeleteOptionValueRequest, and RetrieveAllOptionSetsRequest SDK calls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nd options object - Replace --type string param with AttributeTypeArg enum (DotMake auto-parses) - Add all type-specific CLI options with [DefaultValue] attributes: string/memo, numeric, bool, datetime, choice, lookup, image/file params - Create CreateAttributeOptions record in Core (primitive types only, no SDK dep) - Refactor DataverseEntityMetadataService.CreateAttributeAsync to accept options object - Add support for: memo, polymorphic-lookup, customer, global optionsets, string formats, datetime behavior, cascade delete, solution targeting - Add runtime validation per type in RunAsync (ValidateTypeSpecificParams) - Add input validation for all enum-like string params before service call Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add single-attribute deep-dive command that retrieves detailed metadata for a specific entity attribute using RetrieveAttributeRequest. - New EntityAttributeGetCliCommand with --entity, --name, --json options - New GetAttributeDetailAsync method on IDataverseEntityMetadataService - Service implementation with type-specific metadata extraction (string, memo, integer, decimal, double, money, boolean, datetime, picklist, multi-select picklist, lookup, image, file) - Vertical key:value text output with JSON alternative via --json flag - Registered as child of EntityAttributeCliCommand Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DotMake treats nullable string? CLI options as required unless explicitly annotated with Required=false. This caused 'attribute create' to reject commands with 'Option X is required' errors for ALL type-specific params regardless of the selected --type. Fixed by adding Required=false to all 12 type-specific optional params. Known issue: 'attribute update --required recommended' reports success but RequiredLevel stays None. Under investigation — may be a Dataverse managed property restriction or a publish timing issue. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…consistency The Dataverse SDK's UpdateAttributeRequest silently drops RequiredLevel (ManagedProperty) changes — the request succeeds but the change is not persisted server-side. This is a known limitation; the Power Apps maker portal uses direct Web API PUT instead. Changes: - Add ForceServerMetadataCacheConsistency=true on metadata reads and update flows to ensure fresh data after publish - Add post-update verification that detects when RequiredLevel wasn't applied, and throws a clear error message directing users to make.powerapps.com for this specific change - Fix DotMake Required=false on all optional attribute create params - Add 'Required Level Can Change' to attribute get output - Remove debug Console.Error lines Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ound) The SDK's UpdateAttributeRequest does not serialize ManagedProperty changes (RequiredLevel, IsAuditEnabled, etc.) regardless of whether you use a fresh object, modify .Value, or replace the entire property. This is a confirmed SDK limitation — the Power Apps maker portal uses direct Web API PUT instead. Solution: For RequiredLevel changes, we now: 1. GET the full typed attribute JSON from the Web API 2. Modify the RequiredLevel object in the JSON 3. PUT back the full definition (same approach as make.powerapps.com) Added DataverseConnection.CreateWebApiClientAsync() that exposes an HttpClient with OAuth bearer auth for direct Web API calls. Other attribute changes (DisplayName, Description) continue to use the SDK path which works correctly for those properties. Tested: RequiredLevel change from None → Recommended verified via OData metadata query against live org. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…, changeset commands) - Add StagedCliCommand base class with --apply/--stage execution modes - Add StagedOperation model for representing staged operations - Add IChangesetStore interface and InMemoryChangesetStore implementation - Register InMemoryChangesetStore as singleton in DI container - Add changeset status and discard CLI commands under 'txc environment changeset' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement the changeset apply command with three data execution strategies: - batch (ExecuteMultiple) with optional --continue-on-error - transaction (ExecuteTransaction) with automatic rollback on failure - bulk (CreateMultiple/UpdateMultiple) grouped by entity and operation type The apply pipeline runs in phases: 1. Schema operations (sequential via existing service methods) 2. Publish affected entities 3. Data operations (strategy-dependent) 4. Summary with per-operation results New files: - IChangesetApplier contract and result types in Core - ChangesetApplyCliCommand (CLI surface, inherits ProfiledCliCommand) - ChangesetApplier implementation in Platform.Dataverse.Data The changeset store is cleared automatically after a fully successful apply. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…mand Change base class of 17 mutating commands to StagedCliCommand, adding --apply/--stage execution mode support with ValidateExecutionMode() validation and IChangesetStore staging logic. Schema commands (entity/attribute/relationship/optionset): category=schema Data commands (record create/update/delete/associate/disassociate/upload): category=data Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DataverseEntityMetadataService, DataverseRelationshipService, and DataverseOptionSetService perform DDL-style schema operations (entity/ attribute/relationship create/update/delete) which belong in the Application plane, not the Data plane. The Data plane (Platform.Dataverse.Data) retains data-focused services: DataverseFileService, DataverseRecordService, DataverseBulkService, ChangesetApplier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…sage Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add EntityOptionSetDeleteGlobalCliCommand extending StagedCliCommand - Add DeleteGlobalOptionSetAsync to IDataverseOptionSetService interface - Implement DeleteGlobalOptionSetAsync in DataverseOptionSetService using DeleteOptionSetRequest - Register delete-global in EntityOptionSetCliCommand Children array - Fix stale metadata in list-global by setting ForceServerMetadataCacheConsistency = true Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…a validation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Set ServiceClient.MaxConnectionTimeout to 2 minutes in DataverseConnectionFactory to prevent indefinite hangs when token acquisition fails silently. Document known limitations: - JSON data record create does not auto-coerce OptionSetValue/Money types - Connection timeout behavior on heavily loaded environments - Choice and Money field type conversion for bulk data loading Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Entity create now supports: - --ownership user|organization (default: user) - --type standard|activity|elastic (default: standard) - --has-notes — enable notes/attachments - --has-activities — enable activity association - --enable-audit — enable auditing - --enable-change-tracking — enable change tracking Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pe to attribute create Shared properties on all attribute types: - --is-auditable: enable per-column audit tracking - --is-searchable: control Advanced Find visibility (default: true) - --is-secured: enable field-level security New type: bigint (BigIntAttributeMetadata) — 64-bit integer for large numbers, timestamps, versioning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR adds broad Dataverse automation capabilities to txc, including Configuration Migration Tool (CMT) import/export, schema management commands, and a cross-process changeset staging/apply workflow with multiple execution strategies.
Changes:
- Adds CMT export support (subprocess-hosted) and import tuning options.
- Introduces schema mutation commands (entity/attribute/relationship/optionset) plus record file transfer and N:N associate/disassociate.
- Implements changeset staging/persistence and a changeset applier with schema batching + publish optimization.
Reviewed changes
Copilot reviewed 77 out of 78 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| src/TALXIS.CLI.Platform.XrmShim/CrmServiceClient.cs | XrmShim behavior alignment + FetchXML paging + result dictionary shaping |
| src/TALXIS.CLI.Platform.XrmShim/CrmFieldType.cs | Adds Raw field type |
| src/TALXIS.CLI.Platform.Xrm/TALXIS.CLI.Platform.Xrm.csproj | Pulls CMT WPF package + copies ExportProcessor DLLs |
| src/TALXIS.CLI.Platform.Xrm/LegacyAssemblyRuntime.cs | Preload/patch legacy assemblies (ExportProcessor) + deferred resolver |
| src/TALXIS.CLI.Platform.Xrm/CmtImportRunner.cs | Adds import tuning + trace level wiring |
| src/TALXIS.CLI.Platform.Xrm/CmtImportRequest.cs | Extends request with tuning flags |
| src/TALXIS.CLI.Platform.Xrm/CmtExportRunner.cs | New standalone export runner (reflection + resolvers) |
| src/TALXIS.CLI.Platform.Xrm/CmtExportResult.cs | Export result DTO |
| src/TALXIS.CLI.Platform.Xrm/CmtExportRequest.cs | Export request DTO |
| src/TALXIS.CLI.Platform.Dataverse.Runtime/Runtime/DataverseConnectionFactory.cs | Sets connection timeout + captures token provider |
| src/TALXIS.CLI.Platform.Dataverse.Runtime/Domain/DataverseConnection.cs | Adds Web API HttpClient factory using token provider |
| src/TALXIS.CLI.Platform.Dataverse.Data/DependencyInjection/DataverseDataServiceCollectionExtensions.cs | Registers file service + changeset applier |
| src/TALXIS.CLI.Platform.Dataverse.Data/DataverseFileService.cs | Block-based file/image upload/download service |
| src/TALXIS.CLI.Platform.Dataverse.Data/ChangesetApplier.cs | Applies staged schema/data operations with multiple strategies |
| src/TALXIS.CLI.Platform.Dataverse.Application/Services/DataverseRelationshipService.cs | N:N associate/disassociate service |
| src/TALXIS.CLI.Platform.Dataverse.Application/Services/DataverseOptionSetService.cs | Global/local optionset management service |
| src/TALXIS.CLI.Platform.Dataverse.Application/Services/DataverseDataPackageService.cs | Wires new CMT export + import tuning into feature layer |
| src/TALXIS.CLI.Platform.Dataverse.Application/Sdk/LegacyAssemblyHostSubprocess.cs | Adds CMT export subprocess IPC endpoint |
| src/TALXIS.CLI.Platform.Dataverse.Application/Sdk/CmtExportJob.cs | Export IPC envelope record |
| src/TALXIS.CLI.Platform.Dataverse.Application/DependencyInjection/DataverseApplicationServiceCollectionExtensions.cs | Registers schema/relationship/optionset services |
| src/TALXIS.CLI.Features.Environment/EnvironmentCliCommand.cs | Adds environment changeset command group |
| src/TALXIS.CLI.Features.Environment/Entity/EntityCliCommand.cs | Adds entity CRUD + attribute/relationship/optionset subcommands |
| src/TALXIS.CLI.Features.Environment/Entity/EntityCreateCliCommand.cs | New entity create command with staging |
| src/TALXIS.CLI.Features.Environment/Entity/EntityUpdateCliCommand.cs | New entity update command with staging |
| src/TALXIS.CLI.Features.Environment/Entity/EntityDeleteCliCommand.cs | New entity delete command with staging |
| src/TALXIS.CLI.Features.Environment/Entity/EntityGetCliCommand.cs | New entity get command |
| src/TALXIS.CLI.Features.Environment/Entity/EntityAttributeCliCommand.cs | Parent for attribute commands |
| src/TALXIS.CLI.Features.Environment/Entity/EntityAttributeCreateCliCommand.cs | New typed attribute create command + validation |
| src/TALXIS.CLI.Features.Environment/Entity/EntityAttributeUpdateCliCommand.cs | Attribute update with staging |
| src/TALXIS.CLI.Features.Environment/Entity/EntityAttributeDeleteCliCommand.cs | Attribute delete with staging |
| src/TALXIS.CLI.Features.Environment/Entity/EntityAttributeGetCliCommand.cs | Attribute detail retrieval command |
| src/TALXIS.CLI.Features.Environment/Entity/EntityAttributeTypeCliCommand.cs | Parent for attribute type introspection |
| src/TALXIS.CLI.Features.Environment/Entity/EntityAttributeTypeListCliCommand.cs | Lists supported attribute types |
| src/TALXIS.CLI.Features.Environment/Entity/EntityAttributeTypeDescribeCliCommand.cs | JSON schema for attribute type parameters |
| src/TALXIS.CLI.Features.Environment/Entity/AttributeTypeRegistry.cs | Static registry of supported attribute types/params |
| src/TALXIS.CLI.Features.Environment/Entity/EntityRelationshipCliCommand.cs | Parent for relationship commands |
| src/TALXIS.CLI.Features.Environment/Entity/EntityRelationshipCreateCliCommand.cs | Relationship create with staging |
| src/TALXIS.CLI.Features.Environment/Entity/EntityRelationshipListCliCommand.cs | Relationship list command |
| src/TALXIS.CLI.Features.Environment/Entity/EntityRelationshipDeleteCliCommand.cs | Relationship delete with staging |
| src/TALXIS.CLI.Features.Environment/Entity/EntityOptionSetCliCommand.cs | Parent for optionset commands |
| src/TALXIS.CLI.Features.Environment/Entity/EntityOptionSetCreateGlobalCliCommand.cs | Create global optionset with staging |
| src/TALXIS.CLI.Features.Environment/Entity/EntityOptionSetDeleteGlobalCliCommand.cs | Delete global optionset with staging |
| src/TALXIS.CLI.Features.Environment/Entity/EntityOptionSetAddOptionCliCommand.cs | Add option to local/global optionset with staging |
| src/TALXIS.CLI.Features.Environment/Entity/EntityOptionSetDeleteOptionCliCommand.cs | Delete option from local/global optionset with staging |
| src/TALXIS.CLI.Features.Environment/Entity/EntityOptionSetListGlobalCliCommand.cs | List global optionsets command |
| src/TALXIS.CLI.Features.Environment/Data/Record/EnvDataRecordCliCommand.cs | Adds record file + association commands |
| src/TALXIS.CLI.Features.Environment/Data/Record/EnvDataRecordCreateCliCommand.cs | Record create converted to staging-capable |
| src/TALXIS.CLI.Features.Environment/Data/Record/EnvDataRecordUpdateCliCommand.cs | Record update converted to staging-capable |
| src/TALXIS.CLI.Features.Environment/Data/Record/EnvDataRecordDeleteCliCommand.cs | Record delete adds apply/stage flags |
| src/TALXIS.CLI.Features.Environment/Data/Record/EnvDataRecordDownloadFileCliCommand.cs | New record file download command |
| src/TALXIS.CLI.Features.Environment/Data/Record/EnvDataRecordUploadFileCliCommand.cs | New record file upload command (staged) |
| src/TALXIS.CLI.Features.Environment/Data/Record/EnvDataRecordAssociateCliCommand.cs | New associate command (staged) |
| src/TALXIS.CLI.Features.Environment/Data/Record/EnvDataRecordDisassociateCliCommand.cs | New disassociate command (staged) |
| src/TALXIS.CLI.Features.Environment/Changeset/ChangesetCliCommand.cs | Parent changeset command group |
| src/TALXIS.CLI.Features.Environment/Changeset/ChangesetStatusCliCommand.cs | Shows staged operations |
| src/TALXIS.CLI.Features.Environment/Changeset/ChangesetApplyCliCommand.cs | Applies staged operations via strategy |
| src/TALXIS.CLI.Features.Environment/Changeset/ChangesetDiscardCliCommand.cs | Clears staged operations |
| src/TALXIS.CLI.Features.Data/DataPackageCliCommand.cs | Adds data package export command |
| src/TALXIS.CLI.Features.Data/DataPackageImportCliCommand.cs | Adds import tuning flags |
| src/TALXIS.CLI.Features.Data/DataPackageExportCliCommand.cs | New CMT export command |
| src/TALXIS.CLI.Core/Shared/StagedCliCommand.cs | New base class for --apply/--stage mutating commands |
| src/TALXIS.CLI.Core/DependencyInjection/ConfigServiceCollectionExtensions.cs | Registers changeset store singleton |
| src/TALXIS.CLI.Core/Changeset/InMemoryChangesetStore.cs | Adds persisted changeset store at .txc/changeset.json |
| src/TALXIS.CLI.Core/Contracts/Dataverse/StagedOperation.cs | Staged operation contract |
| src/TALXIS.CLI.Core/Contracts/Dataverse/IChangesetStore.cs | Changeset store interface |
| src/TALXIS.CLI.Core/Contracts/Dataverse/IChangesetApplier.cs | Changeset applier interface + result DTOs |
| src/TALXIS.CLI.Core/Contracts/Dataverse/IDataverseFileService.cs | File service contract |
| src/TALXIS.CLI.Core/Contracts/Dataverse/IDataverseRelationshipService.cs | Relationship service contract |
| src/TALXIS.CLI.Core/Contracts/Dataverse/IDataverseOptionSetService.cs | Optionset service contract |
| src/TALXIS.CLI.Core/Contracts/Dataverse/IDataverseEntityMetadataService.cs | Expands schema service contract (CRUD + relationships) |
| src/TALXIS.CLI.Core/Contracts/Dataverse/IDataPackageService.cs | Adds CMT export + import tuning in contract |
| src/TALXIS.CLI.Core/Contracts/Dataverse/CreateEntityOptions.cs | DTO for entity creation |
| src/TALXIS.CLI.Core/Contracts/Dataverse/CreateAttributeOptions.cs | DTO for attribute creation |
| docs/dataverse-metadata-performance.md | New performance guide |
| .gitignore | Ignores .txc/ changeset persistence folder |
…turns, help text - Fix optionset CSV parsing in changeset applier - Add ASSOCIATE/DISASSOCIATE support to BuildOrganizationRequest - Handle file uploads separately (chunked API, not batchable) - Normalize parameter keys between staging and dispatch - Fix status help text to match actual apply options - RecordDelete inherits StagedCliCommand + IDestructiveCommand - Only inject paging-cookie when non-empty - Return empty dict instead of null! on GetEntityDataById/LinkedSearch - Add explanatory comments for AppSettings mutation and ReturnProperty_Id trailing space - Remove duplicate doc comment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
@copilot is this addition ok? does it introduce any inconsistencies to the CLI codebase? is there any dead code, unnecessary complexity etc? give me your expert opinion |
Agent-Logs-Url: https://github.com/TALXIS/tools-cli/sessions/09500588-6f3a-4cb3-937e-83321fa274ed Co-authored-by: TomProkop <4339668+TomProkop@users.noreply.github.com>
Agent-Logs-Url: https://github.com/TALXIS/tools-cli/sessions/09500588-6f3a-4cb3-937e-83321fa274ed Co-authored-by: TomProkop <4339668+TomProkop@users.noreply.github.com>
Agent-Logs-Url: https://github.com/TALXIS/tools-cli/sessions/09500588-6f3a-4cb3-937e-83321fa274ed Co-authored-by: TomProkop <4339668+TomProkop@users.noreply.github.com>
I reviewed it as a codebase-consistency pass. Broadly the direction is OK, but I found and fixed a few real inconsistencies: staged attribute create now normalizes/parses the same shape as immediate apply, the new read/list commands now use global |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…forcement New analyzer rules: - TXC005: No raw integer returns in ExecuteAsync (use ExitSuccess/ExitError) - TXC006: No try-catch in ExecuteAsync (base class handles errors) - TXC007: No --json CLI option (use base class --format) - TXC008: Must override Logger property, not shadow with field - TXC009: Public enum members must have explicit values Config changes: - Nullable warnings CS8600-CS8625 escalated to warning via .editorconfig - Namespace-folder mismatch IDE0130 enabled as warning - TXC004 added to WarningsAsErrors in Directory.Build.props - Banned old ProfiledCliCommand location in BannedSymbols.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- TXC005: replace raw integer returns with named exit constants - TXC006: remove try-catch from ExecuteAsync, extract fallible helpers - TXC007: (no violations) - TXC008: convert ILogger fields to auto-properties - TXC009: add explicit values to all public enum members Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nalyzers - New: docs/changeset-staging.md (staging workflow guide) - New: docs/schema-management.md (entity/attribute/relationship/optionset CRUD) - Updated: README.md (new command groups, links) - Updated: CONTRIBUTING.md (safety annotations, staging pattern) - Updated: docs/architecture.md (Dataverse.Data active, Analyzers project) - Updated: docs/research/proposed-roslyn-rules.md (all rules implemented) - Updated: docs/dataverse-metadata-performance.md (cross-references) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Schema belongs to Application plane, not a separate plane - Changeset staging shown inline as workflow, not a plane - Data plane examples grouped and tightened - CMT section highlights cross-platform advantage Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Default output is now an extracted folder for direct repo use. Pass --zip to produce a .zip archive instead (previous behavior). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…EADME These are key differentiators vs PAC CLI — give them proper spotlight. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reword documentation to present technical knowledge as expertise without revealing reverse engineering methodology. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Workspace (local, instant, agent-friendly) leads the narrative. Environment commands follow as the sync layer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ledge base (#31) * feat: Milestone 1 — foundation for MCP progressive disclosure - Improve 31 tool descriptions for AI steering (LOCAL/LIVE context, profile requirements, cross-references for confusable pairs) - Add ToolCatalog: pre-built schemas, workflow tagging, sampling prompt generation, category derivation - Add ActiveToolSet: session-scoped tool management with always-on tools, LRU eviction for injected tools, thread safety - Refactor McpToolRegistry to populate ToolCatalog at startup with pre-cached schemas (no longer rebuilt on every ListTools call) - Add static BuildToolDefinition helper for guide tool injection Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Milestone 2 — guide tools, execute_operation bridge, progressive disclosure Replace static 97-tool list_tools with 8 always-on tools: - guide: cross-domain tool discovery via sampling/createMessage - guide_workspace: LOCAL development (scaffolding, workspace understanding) - guide_environment: LIVE environment inspection and mutation - guide_deployment: deployment lifecycle (pack, import, publish) - guide_data: LIVE data operations (queries, CRUD, bulk, CMT) - guide_config: CLI configuration (auth, profiles, connections) - execute_operation: bridge for same-turn execution of discovered tools - copilot-instructions: unchanged Guide tools use MCP sampling to delegate tool selection to the client's LLM with local-first development steering. Falls back to keyword matching when sampling is unavailable. Discovered tools are injected into ActiveToolSet for direct calling on subsequent turns (client re-fetches list_tools between turns). execute_operation validates against the full internal catalog (not just active set) enabling same-turn execution of guide-discovered tools. ServerInstructions rewritten with workflow map and local-first guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Milestone 3 — skills knowledge base with internal reasoning and public guidance Internal reasoning skills (src/TALXIS.CLI.MCP/Skills/Internal/): - 6 proprietary markdown files NOT exposed to clients - local-first-philosophy, schema-workflow, deployment-sequence, troubleshooting-patterns, data-migration-workflow, solution-management - Loaded by GuideReasoningEngine at startup, injected into guide sampling prompts for context-aware tool selection Public skills (src/TALXIS.CLI.Features.Docs/Skills/): - 8 developer-facing skill files + index.json manifest - project-structure, component-creation, solution-layering, deployment-workflow, data-migration, troubleshooting, environment-management, schema-management - Accessible via get_skill_details MCP tool (new always-on tool) - Embedded as assembly resources, shipped via NuGet Infrastructure: - GuideReasoningEngine: loads internal skills, maps them to domain guides - PublicSkillLoader: loads public skills from Features.Docs assembly - get_skill_details handler with skill index in description - Guide sampling prompts now include relevant internal skills - Integration test updated to verify get_skill_details in always-on set Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: activate txc docs command group for skill knowledge base Wire DocsCliCommand into root command tree with two subcommands: - txc docs list: lists available skills from embedded index.json - txc docs show <skill-id>: prints full skill markdown content Both commands read from the same embedded resources as the MCP get_skill_details tool, providing dual delivery (CLI + MCP). Uses project conventions: TxcLeafCommand, OutputFormatter, ExitSuccess/ExitError/ExitValidationError, TxcJsonOptions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Milestone 4 — tests and documentation for progressive disclosure Tests (32 new, 509 total passing): - ToolCatalogTests: 12 tests — registration, workflow filtering, catalog prompts, category/workflow derivation - ActiveToolSetTests: 10 tests — always-on, injection, LRU eviction, thread safety, IsActive - GuideReasoningEngineTests: 5 tests — skill loading, context mapping - McpToolRegistryTests: 4 new tests — catalog population, pre-built schemas, workflow tags, prompt generation Documentation: - README.md: progressive disclosure architecture, 9 always-on tools, workflow, skills tiers, local-first philosophy - architecture.md: MCP server progressive disclosure behavior, ActiveToolSet, sampling, execute_operation bridge, skills Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: code review findings — task args passthrough and environment guide scoping Bug 1 (High): execute_operation passed outer params to task-augmented execution instead of parsed inner tool arguments. Added overrideArguments parameter to ExecuteAsTaskAsync so long-running tools (solution import, package import) receive correct arguments when invoked via the bridge. Bug 2 (Medium): guide_environment with a query dispatched to the generic unscoped handler, making it identical to guide. Now searches both environment-inspection and environment-mutation workflows and merges results, maintaining proper domain scoping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix code review findings: attributes, dead code, parallelization Fix 1: Replace hardcoded _longRunningCommandTypes with [CliLongRunning] attribute Fix 2: Add [CliWorkflow] attribute with heuristic fallback for workflow classification Fix 3: Eliminate McpToolDescriptorProvider (dead parallel state) Fix 4: Remove dead McpToolRegistry.ListTools() method Fix 5: Remove dead PublicSkillLoader.GetIndex() method Fix 6: Consolidate duplicate BuildGuidanceFromEntries into GuideHandler Fix 7: Parallelize guide_environment and remove format-dependent filtering Fix 8: Set ListChanged = false (honest about capabilities) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Roslyn analyzers TXC010-TXC013 for description quality and workflow enforcement TXC010 (Error): Leaf [CliCommand] Description must be >= 20 chars. Prevents empty/terse descriptions that degrade AI tool discovery. TXC011 (Warning): ProfiledCliCommand subclass description should mention 'profile' or 'environment'. Ensures AI harnesses know prerequisites. Currently flags ~20 commands from PRs #25/#29 that need description updates. TXC012 (Warning): [CliDestructive] description should contain danger words (delete, remove, uninstall, etc.). Ensures both humans and AI see risk. TXC013 (Warning): Commands in TALXIS.CLI.Features.Data without [CliWorkflow] get a warning. Prevents workflow misclassification by name heuristic. Also: add [CliWorkflow("local-development")] to DataModelConvert and DataPackageConvert to resolve TXC013 warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove listChanged dead code, fix schemas and error handling - Remove ListChanged property from ToolsCapability - Remove (CallToolResult, bool ToolsInjected) tuple pattern from guide handlers - Change ActiveToolSet.InjectTools return type from bool to void - Remove all listChanged/toolsInjected references from code and docs - Change top parameter schema type from number to integer with defensive parsing - Remove type constraint from arguments schema in execute_operation - Catch JsonException specifically and exclude raw input from error messages - Update BuildGuidanceResponse trailing message for eviction awareness - Update tests to match new void InjectTools signature Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: skills overhaul — dedup, new skills, PR feedback round 2 Skills deduplication: - Internal skills rewritten as pure decision trees (no duplicated prose) - Public skills enriched with concrete examples, common scenarios, anti-patterns New skills (5 public + 1 internal): - plugin-development: project structure, registration, stages, testing - form-xml-reference: XML hierarchy, ClassID table, dialog forms, fragments - custom-api-development: XML structure, type codes, plugin backing - bpf-development: BPF entities, stages, branching, workflow metadata - pcf-controls: manifest, lifecycle methods, dataset vs field controls - component-composition-chains: multi-step scaffolding decision trees PR feedback round 2: - execute_operation rejects MCP-specific in-process tools with clear error - Compact listing (no schemas) for workflow browse / empty query responses - Full schemas only for specific tool matches via query/sampling - ProfiledDescriptionAnalyzer message lists all 4 accepted keywords - GuideReasoningEngine test uses >= instead of exact count Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Milestone 5 — recipe-based guide intelligence + skill trimming Recipe generation: - Guide sampling prompt now produces multi-step recipes with concrete execute_operation calls, parameter values, and validation checkpoints - SampleToolSelectionAsync returns (tools, recipeText) tuple - BuildGuidanceResponse prepends RECIPE section when recipe available - MaxTokens increased to 1500 for recipe content - Keyword fallback still works (no recipe, just tools) Skill trimming (-728 lines net): - Removed template parameter listings from public skills (AI discovers via workspace_component_parameter_list at runtime) - Removed XML schema details, ClassID tables, form hierarchy details - Kept: composition chains, decision trees, architecture decisions - Skills now 30-60 lines each (was 70-150) Copilot-instructions slimmed: - Kept: Project Structure and Naming Conventions (user-customizable defaults) - Removed: verbose MANDATORY operational rules (now in ServerInstructions) - Replaced with brief pointer to MCP guides Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Milestone 6 — build error recovery skill + internal skill trimming New public skill: build-errors.md - Maps TALXISXSD001, TALXISGUID001, TALXISQF001, TALXISPCF001, TALXISJSONSCHEMA001 to causes and fixes - Includes structural rules (name lengths, GUID patterns, required elements) Internal skill trimming: - component-composition-chains.md: 194 → 35 lines - Removed parameter listings, XML details, tool-discoverable content - All 7 internal skills verified under 80 lines, pure decision trees Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: remove keyword matching fallback — sampling is required Keyword matching was a low-quality fallback that produced wrong results (e.g., environment_entity_create ranked above workspace_component_create for 'create a table'). Sampling is required since Copilot CLI v1.0.13. Changes: - DiscoverToolsAsync/DiscoverToolsWithScopedCatalogAsync now throw InvalidOperationException if sampling returns no results - Removed KeywordMatch method entirely - Removed word-by-word fallback extraction from sampling response parser - Clients without sampling support get a clear error message Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: hide data_package_convert (unfinished) from CLI and MCP Remove DataPackageConvertCliCommand from parent Children array so it's unreachable from both CLI and MCP. Clean all references from: - data-migration.md (public skill) - data-migration-workflow.md (internal skill) The command class is kept for future completion but not wired. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address PR #31 review round 3 — 7 unresolved comments - Remove stale 'keyword matching fallback' references from GuideHandler - Rewrite ParseToolNamesAndRecipeFromSamplingResponse with robust line-by-line JSON array scanning (no more fragile first-bracket grab) - Clamp 'top' parameter to [1, 20] range in HandleGuideToolAsync - Fix JsonDocument leak in ToolCatalogTests.CreateTestSchema - Clarify sampling prompt: MCP in-process tools should be called directly, not through execute_operation - Mirror production ListChanged=false in McpServerProtocolTests; add comments explaining test vs production tool listing divergence Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: PR review round 3 — parsing robustness, input clamping, stale references - GuideHandler: remove stale keyword-matching references in comments - GuideHandler: rewrite JSON parsing to line-by-line progressive scan - GuideHandler: clarify sampling prompt re MCP in-process tools - Program.cs: clamp 'top' parameter to 1-20 range - ToolCatalogTests: dispose JsonDocument properly - McpServerProtocolTests: align with progressive disclosure (ListChanged=false) - McpToolRegistryTests: remove data_package_convert assertion (command hidden) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: fix stale keyword-matching reference in architecture.md Sampling is required since keyword matching was removed. Updated architecture.md to reflect that sampling is mandatory, not a fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Major feature PR adding Configuration Migration Tool (CMT) support, full schema management commands, changeset staging with batch apply, and Dataverse metadata performance optimizations.
37 commits, 78 files, +8,296 lines
New Commands (27)
CMT Data Package
txc data package export— export data with--export-files,--overwritetxc data package import— enhanced with--batch-mode,--batch-size,--override-safety-checks,--prefetch-limitEntity Schema Management
entity create— with--ownership,--type(standard/activity/elastic),--has-notes,--has-activities,--enable-audit,--enable-change-trackingentity get/update/deleteattribute create— 16 types (string, memo, number, decimal, float, money, bool, datetime, choice, multichoice, lookup, polymorphic-lookup, customer, image, file, bigint) with typed parametersattribute get/update(Web API PUT for RequiredLevel) /deleteattribute type list/type describe— JSON schema introspection for MCPrelationship create/list/deleteoptionset create-global/delete-global/add-option/delete-option/list-globalRecord Operations
record download-file/upload-file— chunked block APIrecord associate/disassociate— N:N relationshipsChangeset Staging
--apply/--stageon all 17 mutating commands (StagedCliCommandbase class)changeset status/apply/discard--strategy batch(ExecuteMultiple),--strategy transaction(ExecuteTransaction),--strategy bulk(CreateMultiple/UpdateMultiple).txc/changeset.json) for cross-process CLI usageCreateEntitiesbatch API integration + singlePublishXmlXrmShim Fixes
Documentation
docs/configuration-migration.md— 915-line comprehensive CMT guide (schema reference, 18 data types, file columns, import tuning, dedup internals, date modes)docs/dataverse-metadata-performance.md— 331-line performance guide (CreateEntities API, publish optimization, metadata cache strategies)Key Technical Discoveries
CreateEntitiesbatch SDK action: 12s for entity creation vs 60s+ standardUpdateAttributeRequestsilently drops RequiredLevel — workaround: Web API PUTExportFilesAppSetting must betruefor both export AND importPublishXmlwith multiple entities is ~10x faster than per-entity publishTesting
Architecture
StagedCliCommandin Core (inheritsProfiledCliCommand)[CliDestructive],[CliReadOnly],[CliIdempotent])