feat(training): support Foundry storage connections

[Yeshwanth N (yeshsurya)]

Fixes #9569

Summary

• support SDK-compatible asset_name and asset_version for training job outputs
• support Foundry storage connections through storage_connection_name and --storage-connection-name
• preserve azureai:// code and input references instead of treating them as local paths
• scope local-upload deduplication by the effective storage connection
• document the new YAML and CLI behavior

Cross-connection deduplication issue

Previously, local upload versions were derived only from the content hash. If content had already been uploaded using default storage, submitting the same content with --storage-connection-name could reuse the default-storage dataset and never call startPendingUpload for the requested connection. This can reproduce the original storage-resolution failure even after the user selects a working BYOS connection.

The dataset version remains content-hash based for default storage. For an explicit connection, it is deterministically derived from both the content hash and connection name, preserving deduplication within a connection while preventing reuse across connections.

Validation

• go test ./... -count=1
• go build ./...
• go vet ./...
• git diff --check
• go fix -diff ./...
• regression test covers default-storage content followed by an identical BYOS upload
• built and installed a local extension bundle; YAML validation and command help smoke tests passed

[azure-pipelines]

Azure Pipelines:
7 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds Foundry storage connection support and named training output assets.

Changes:

• Adds YAML/CLI storage connection configuration.
• Preserves azureai:// code and input references.
• Maps asset_name and asset_version into job outputs with tests and docs.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
README.md	Documents new job options.
pkg/models/dataset.go	Adds upload connection serialization.
pkg/models/dataset_test.go	Tests request wire format.
pkg/client/datasets.go	Sends the storage connection.
internal/utils/yaml_parser.go	Parses new fields and URI scheme.
internal/utils/yaml_parser_test.go	Tests YAML parsing and URI preservation.
internal/service/upload_service.go	Propagates connection configuration.
internal/service/upload_service_test.go	Tests connection propagation.
internal/service/resolver_test.go	Covers remote azureai:// references.
internal/cmd/job_submit.go	Adds CLI override and output mapping.
internal/cmd/job_submit_test.go	Tests flags, precedence, and outputs.
internal/cmd/init_template.go	Adds storage configuration guidance.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (2)

cli/azd/extensions/azure.ai.training/internal/cmd/init_template.go:76

• [azd-code-reviewer] The generated file still declares the Azure ML command-job schema, whose top-level additionalProperties: false does not include this extension-only property. As soon as a user uncomments this line, YAML tooling reports storage_connection_name as invalid even though the CLI accepts it. Update the template to use a schema that includes the extension fields, remove the incompatible $schema, or avoid advertising the field in this schema-bound template.

# storage_connection_name: <optional Foundry storage connection for local uploads>

cli/azd/extensions/azure.ai.training/README.md:20

• [azd-code-reviewer] This is not the complete default output: job submit always calls PrintObject, and its default --output format is JSON (job_submit.go:139,160). Label this as the initial status output and mention the following JSON result so users do not mistake the example for the command's full stdout.

Expected output:

[Yeshwanth N (yeshsurya)]

Addressed the latest suppressed review feedback in 565f01b0b:

• Removed storage_connection_name from the Azure ML schema-bound generated starter template; the extension-specific field remains documented in the extension README.
• Updated the README output example to show that the status lines are followed by the default JSON job result.
• Updated the linked design documentation to reflect supported named output registration and the asset_name / asset_version mapping.

The earlier cross-connection deduplication comment was addressed in 00f8a4a98.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (1)

cli/azd/extensions/azure.ai.training/pkg/models/dataset.go:23

• azd-code-reviewer: The linked design documentation still describes this request as containing only pendingUploadType (design/design.md:500-504, design/execution-plan.md:411-414) and shows StartPendingUpload without a connection argument (design/design.md:981, design/execution-plan.md:518). Update those request examples and signatures to include the optional connectionName, otherwise the extension's design reference contradicts the new API behavior.

	ConnectionName    string `json:"connectionName,omitempty"` // Optional Foundry storage connection

[Yeshwanth N (yeshsurya)]

Addressed the final multi-agent Standards findings in 8c16aedad:

• renamed the test mock to mock_upload_client.go
• documented every exported mock method
• removed the duplicate handwritten upload client fake and migrated the upload-service tests to the single testify mock

Full extension tests, build, vet, go fix -diff, and git diff --check pass.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

[saanikaguptamicrosoft]

/check-enforcer override