file-based apps: dotnet add file.cs package <package> changes encoding of file.cs

[johnkors]

Describe the bug

The SDK changes encoding of file-based app files when adding packages.

To Reproduce

1 - Create a a file.cs (UTF-8 encoded)
2 - Add a shebang : #!/usr/bin/env dotnet run and chmod +x file.cs
3- Run it: ./file.cs
4 - Add any package dotnet add file.cs AnyNugetPackageIdHere => file.cs is converted to UTF-8 with BOM
5 - Running ./file.cs again after the package is added results in failure:

> ./file.cs
./file.cs: line 1: #!/usr/bin/env: No such file or directory

Further technical details

.NET 10 SDK, 1.0.100

[DamianEdwards]

Analysis from Copilot:

Root Cause Analysis

The issue is in the SourceFile struct in [src/Cli/Microsoft.DotNet.FileBasedPrograms/FileLevelDirectiveHelpers. cs](https://github.com/dotnet/sdk/blob/e300dea82e0248b0801f706914c96073c1bd5350/src/Cli/Microsoft.DotNet. FileBasedPrograms/FileLevelDirectiveHelpers.cs#L256-L274):

public void Save()
{
    using var stream = File.Open(Path, FileMode.Create, FileAccess. Write);
    using var writer = new StreamWriter(stream, Encoding.UTF8);
    Text.Write(writer);
}

Problem: Encoding. UTF8 (the static property) emits a UTF-8 BOM by default. When dotnet add file. cs package <package> modifies the file and saves it, the BOM (0xEF 0xBB 0xBF) is prepended to the file content.

On Unix-like systems, the shebang #!/usr/bin/env dotnet run becomes <BOM>#!/usr/bin/env dotnet run, which the kernel doesn't recognize as a valid interpreter directive, causing:

./file.cs: line 1: #!/usr/bin/env: No such file or directory

Fix: The Save() method should preserve the original file's encoding/BOM characteristics. Specifically, for files without a BOM, it should use:

new UTF8Encoding(encoderShouldEmitUTF8Identifier: false)

The Load() method should also detect and store whether the original file had a BOM so it can be preserved on save.

[johnkors]

Found the BOM header "bug" in several places

• dotnet new: creates UTF-8 BOM files (csprojs, sln, cs) - is this intentional?
• dotnet sln add/remove : changes the sln encoding, registered in dontet sln add keeps changing encoding and line endings #50066

However, dotnet add package on a csproj does not change the encoding of the csproj.

[baronfel]

@johnkors re: dotnet new's use of BOM, that's tracked in #39187 - in general new should adhere to editorconfig here for generated content, but the overall architecture of the template engine may need to be reworked to allow for editorconfig to be interpreted and applied to the output. We'd been looking at accomplishing this by invoking dotnet format on the generated files, for example, since format is kind of the 'canonical' source of how editorconfig should be applied to C# source files - it didn't make sense to me to duplicate this logic.