fix: make `sources-files` update `sources` by PawelWMS · Pull Request #69 · microsoft/azure-linux-dev-tools

PawelWMS · 2026-04-07T01:24:38Z

Adding a modification to how source-files config affects the output sources file:

New files will be appended to the sources file.
The source-files entries must have their hash and hash type provided unless the user passes the --allow-no-hashes flag, which will then retrieve the sources and calculate the missing hashes.
If a file name already exists in the sources file, it's considered an error. The user is told to either remove the original entry through an overlay (I'll add it in a separate PR, if it doesn't exist yet) or to re-name the file from source-files.

I've also added config validation for the source-files section of the TOML and update the code to work only with the hash types defined in the HashType* constants.

Copilot

Pull request overview

This PR updates source preparation so that a component’s source-files configuration can append new entries into the Fedora/RHEL sources file, requiring both hash and hash-type and erroring on filename conflicts.

Changes:

Refactors Fedora sources parsing to expose a reusable ParseSourcesFile and adds a FormatSourcesEntry helper.
Extends source preparation to append configured source-files entries into the output sources file (with validation and conflict detection).
Updates/adds unit tests for the new formatting and sources-file update behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
internal/providers/sourceproviders/fedorasource/fedorasource.go	Introduces exported parsing/formatting helpers and refactors URL-building to use parsed entries.
internal/providers/sourceproviders/fedorasource/fedorasource_test.go	Updates hash type assertions and adds coverage for `FormatSourcesEntry`.
internal/app/azldev/core/sources/sourceprep.go	Adds `sources` file update step during `PrepareSources`, parsing existing entries and appending new ones from config.
internal/app/azldev/core/sources/sourceprep_test.go	Adds tests verifying `sources` file creation/appending and validation errors.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 7 comments.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

…elwi/sources_update

PawelWMS · 2026-04-08T22:50:02Z


 	// HashTypeSHA256 represents the SHA-256 hash algorithm.
-	HashTypeSHA256 HashType = "sha256"
+	HashTypeSHA256 HashType = "SHA256"


Note: all current TOML configs use the upper case version, so this should not break them.

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

…types.

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Copilot · 2026-04-09T00:27:27Z

+				"invalid 'uri' for source file %#q, component %#q; "+
+					"URI %#q is missing a scheme (e.g. 'https://')",
+				filename, componentName, origin.Uri)
+		}


For 'download' origins, validation currently only checks that the URI has some scheme. Since source downloading uses the HTTP downloader (net/http), non-HTTP schemes (e.g. file://, ftp://) and even malformed HTTP(S) URLs with an empty host (e.g. https:///path) will pass config validation but fail later at runtime. Consider restricting schemes to http/https (and requiring a non-empty host for those schemes) so config errors are caught early with a clear message.

Suggested change

}

}

if parsed.Scheme != "http" && parsed.Scheme != "https" {

return fmt.Errorf(

"invalid 'uri' for source file %#q, component %#q; "+

"URI %#q must use 'http' or 'https'",

filename, componentName, origin.Uri)

}

if parsed.Host == "" {

return fmt.Errorf(

"invalid 'uri' for source file %#q, component %#q; "+

"URI %#q must include a host",

filename, componentName, origin.Uri)

}

Copilot · 2026-04-09T00:27:27Z

+		if err := fileutils.ValidateFilename(ref.Filename); err != nil {
+			return nil, fmt.Errorf("invalid filename %#q in 'source-files' configuration:\n%w", ref.Filename, err)
+		}


FormatSourcesEntry writes the filename verbatim inside parentheses. Filenames containing a closing parenthesis ) (or newlines) will produce a 'sources' entry that ReadSourcesFileEntries cannot parse (its regex stops at )), and will likely break downstream tooling. Since ValidateFilename doesn’t reject these characters, consider adding an explicit validation in buildSourceEntries (or config validation) to reject filenames that can’t be represented in the modern sources-file format, with a clear error.

fix: make 'sources-files' update 'sources'

5416b5c

Copilot AI review requested due to automatic review settings April 7, 2026 01:24

Copilot started reviewing on behalf of PawelWMS April 7, 2026 01:25 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

Comment thread internal/providers/sourceproviders/fedorasource/fedorasource.go Outdated

Comment thread internal/app/azldev/core/sources/sourceprep.go Outdated

Comment thread internal/app/azldev/core/sources/sourceprep.go

Comment thread internal/app/azldev/core/sources/sourceprep.go Outdated

reubeno reviewed Apr 7, 2026

View reviewed changes

Comment thread internal/app/azldev/core/sources/sourceprep.go Outdated

Comment thread internal/app/azldev/core/sources/sourceprep.go Outdated

Comment thread internal/providers/sourceproviders/fedorasource/fedorasource.go

PawelWMS added 2 commits April 7, 2026 17:55

Addressing comments

644ca5b

Validate 'source-files' duplicates.

2670239

Copilot AI review requested due to automatic review settings April 7, 2026 18:45

Copilot AI reviewed Apr 7, 2026

View reviewed changes

Copilot started reviewing on behalf of PawelWMS April 7, 2026 19:09 View session

PawelWMS added 3 commits April 7, 2026 22:25

Make MD5 invalid.

5d86e3f

Adding the '--allow-no-hashes' argument. Failing tests.

85aa5e1

Adding the '--allow-no-hashes' argument. Updated tests and comments.

164d414

Copilot AI review requested due to automatic review settings April 8, 2026 19:05

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Comment thread internal/projectconfig/configfile.go

Comment thread internal/app/azldev/core/sources/sourceprep.go

Comment thread docs/user/reference/config/components.md Outdated

Comment thread internal/app/azldev/core/sources/sourceprep.go

Copilot started reviewing on behalf of PawelWMS April 8, 2026 19:23 View session

PawelWMS added 2 commits April 8, 2026 19:41

Updating 'origin' validation and comments.

bbc0167

Addressing Copilot's feedback and updating comments.

cf4a31f

Copilot AI review requested due to automatic review settings April 8, 2026 19:57

Copilot started reviewing on behalf of PawelWMS April 8, 2026 19:58 View session

PawelWMS added 2 commits April 8, 2026 20:00

More feedback updates.

6ebb317

Merge branch 'main' into pawelwi/sources_update

75fc5d4

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Update WithSkipLookaside comment

1cce895

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 8, 2026 20:54

Copilot started reviewing on behalf of PawelWMS April 8, 2026 20:56 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Comment thread internal/providers/sourceproviders/fedorasource/fedorasource.go

Comment thread internal/projectconfig/configfile.go

Comment thread internal/app/azldev/core/sources/sourceprep.go

PawelWMS added 3 commits April 8, 2026 21:40

Update hash type requirements and validation.

e371246

Merge remote-tracking branch 'origin/pawelwi/sources_update' into paw…

4275768

…elwi/sources_update

Adding test cases for invalid hash type strings.

7223a7a

Addressing Copilot feedback.

63d077c

Copilot AI review requested due to automatic review settings April 8, 2026 22:43

Extending user error message for missing hashes.

3d52ce6

PawelWMS force-pushed the pawelwi/sources_update branch from fa1fd8a to 3d52ce6 Compare April 8, 2026 22:48

PawelWMS commented Apr 8, 2026

View reviewed changes

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Comment thread docs/user/reference/config/components.md Outdated

Comment thread internal/projectconfig/configfile.go

Comment thread internal/app/azldev/core/sources/sourceprep.go

Comment thread internal/app/azldev/core/sources/sourceprep_test.go

Copilot started reviewing on behalf of PawelWMS April 8, 2026 22:55 View session

PawelWMS added 2 commits April 8, 2026 23:24

Updating tests and docs with feedback.

60ee98b

Updating docs.

d5d448a

Copilot AI review requested due to automatic review settings April 8, 2026 23:31

Copilot started reviewing on behalf of PawelWMS April 8, 2026 23:32 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Comment thread internal/projectconfig/configfile.go

Comment thread internal/app/azldev/core/sources/sourceprep.go

Comment thread internal/providers/sourceproviders/fedorasource/fedorasource.go

reubeno approved these changes Apr 8, 2026

View reviewed changes

PawelWMS added 2 commits April 8, 2026 23:58

Add a test to ensure JSON schema enums always match the allowed hash …

af7405b

…types.

Fixing comment.

25dac04

Copilot AI review requested due to automatic review settings April 9, 2026 00:07

Copilot started reviewing on behalf of PawelWMS April 9, 2026 00:08 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Comment thread internal/utils/fileutils/hash.go

Comment thread internal/projectconfig/component.go Outdated

Comment thread internal/app/azldev/core/sources/sourceprep.go Outdated

PawelWMS added 2 commits April 9, 2026 00:20

Comment updates.

4cef9c3

Fix linting.

af5abb1

Copilot AI review requested due to automatic review settings April 9, 2026 00:22

Copilot started reviewing on behalf of PawelWMS April 9, 2026 00:23 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Update scenario tests.

15faead

reubeno merged commit b6a9106 into microsoft:main Apr 9, 2026
15 checks passed

PawelWMS deleted the pawelwi/sources_update branch April 9, 2026 16:21

-		}
+		}
+		if parsed.Scheme != "http" && parsed.Scheme != "https" {
+			return fmt.Errorf(
+				"invalid 'uri' for source file %#q, component %#q; "+
+					"URI %#q must use 'http' or 'https'",
+				filename, componentName, origin.Uri)
+		}
+		if parsed.Host == "" {
+			return fmt.Errorf(
+				"invalid 'uri' for source file %#q, component %#q; "+
+					"URI %#q must include a host",
+				filename, componentName, origin.Uri)
+		}

Conversation

PawelWMS commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PawelWMS Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

PawelWMS commented Apr 7, 2026 •

edited

Loading