[#14] Marshal native paths as UTF-8 to support non-ASCII paths#122
Merged
Conversation
The native UnityFileSystemApi expects char* paths in UTF-8, but DllWrapper marshalled them as LPStr (the system ANSI code page). Non-ASCII paths (e.g. Japanese) were mangled before reaching native, causing FileNotFound/IOException on Windows. Switch path/string inputs to LPUTF8Str; CreateArchive marshals its string arrays to UTF-8 by hand since LPUTF8Str can't be an array subtype.
SkowronskiAndrew
marked this pull request as ready for review
July 24, 2026 20:56
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes Unicode/non-ASCII filesystem path handling in the native interop layer (UnityFileSystemApi) by ensuring path strings are marshalled as UTF-8 instead of the Windows ANSI code page, addressing failures when working from directories like テスト or emoji-named folders.
Changes:
- Updated P/Invoke signatures in
DllWrapper.csto marshal path/string inputs as UTF-8 (LPUTF8Str) and introduced a manual UTF-8 marshaling wrapper forCreateArchivestring arrays. - Added regression tests that copy test files into temp directories/files with Japanese characters and verify native
MountArchiveandOpenFilework correctly.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| UnityFileSystem/DllWrapper.cs | Switches native string/path input marshalling to UTF-8 and adds manual UTF-8 marshalling for CreateArchive string arrays. |
| UnityFileSystem.Tests/UnityFileSystemTests.cs | Adds regression tests and helper for exercising non-ASCII path scenarios through the native API. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+198
to
+201
| // Strings returned from native (here and in the other StringBuilder-based calls) come back as | ||
| // UTF-8 bytes but are marshalled as the system ANSI code page, so only ASCII round-trips correctly. | ||
| // These are internal archive/serialized-file names, which Unity keeps ASCII, so it's not an issue in | ||
| // practice. Input paths, by contrast, are marshalled as UTF-8 (LPUTF8Str) to support non-ASCII paths. |
Comment on lines
+215
to
+236
| public static ReturnCode CreateArchive(string[] sourceFiles, string[] aliases, bool[] isSerializedFile, int count, | ||
| string archiveFile, CompressionType compression, out int crc) | ||
| { | ||
| var sourcePtrs = new IntPtr[sourceFiles.Length]; | ||
| var aliasPtrs = new IntPtr[aliases.Length]; | ||
| try | ||
| { | ||
| for (int i = 0; i < sourceFiles.Length; ++i) | ||
| sourcePtrs[i] = Marshal.StringToCoTaskMemUTF8(sourceFiles[i]); | ||
| for (int i = 0; i < aliases.Length; ++i) | ||
| aliasPtrs[i] = Marshal.StringToCoTaskMemUTF8(aliases[i]); | ||
|
|
||
| return CreateArchiveNative(sourcePtrs, aliasPtrs, isSerializedFile, count, archiveFile, compression, out crc); | ||
| } | ||
| finally | ||
| { | ||
| foreach (var p in sourcePtrs) | ||
| Marshal.FreeCoTaskMem(p); | ||
| foreach (var p in aliasPtrs) | ||
| Marshal.FreeCoTaskMem(p); | ||
| } | ||
| } |
Comment on lines
+28
to
+31
| finally | ||
| { | ||
| Directory.Delete(dir, true); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #14.
UnityDataToolthrewFileNotFoundException/IOExceptionwhen a path contained non-ASCII characters (Japanese, emoji, etc.), e.g. runninganalyzeordumpfrom a directory likeテスト. Commands that stay entirely in C# (such asserialized-file) worked fine, which pointed at the native interop layer.The native
UnityFileSystemApiexpectschar*paths encoded as UTF-8 (on Windows it decodes them viaMultiByteToWideChar(CP_UTF8, …)beforeCreateFileW; on macOS/Linux the bytes pass straight to the POSIX file APIs, where UTF-8 is the convention). ButDllWrappermarshalled every path with[MarshalAs(UnmanagedType.LPStr)], which is the system ANSI code page. On a non-Japanese Windows install,テストhas no ANSI representation and was converted to?before ever reaching native, so the file couldn't be found.LPUTF8Stris the platform-independent, universally-correct choice.Changes
DllWrapper.cs: marshal path/string inputs to the native library as UTF-8.MountArchive,OpenFile,OpenSerializedFile,AddTypeTreeSourceFromFile,GetRefTypeTypeTree:LPStr→LPUTF8Str.CreateArchive:outputFile→LPUTF8Str; the twochar**string arrays are marshalled to UTF-8 by hand (viaMarshal.StringToCoTaskMemUTF8) sinceLPUTF8Strcan't be used as anLPArraysubtype. The public signature is unchanged.StringBuilder-based outputs (e.g.GetArchiveNode) only round-trip ASCII. These carry Unity-controlled internal archive/serialized-file names, which are always ASCII, so this is left as-is.Testing
dotnet test UnityFileSystem.Tests— full suite green (435 passed, 10 pre-existing skips).MountArchive_NonAsciiPath_ReturnsArchiveandOpenFile_NonAsciiPath_ReadsExpectedData, which copy test files into a temp directory whose directory and file names contain Japanese characters and open them through the native API. These would fail on Windows with the old ANSI marshalling.dump --stdout level0from aテストdirectory — which now succeeds where it previously threw.Note: the bug was Windows-specific (only there does
LPStrmean the lossy ANSI code page; .NET on macOS/Linux already marshalsLPStras UTF-8), so the new tests pass on all platforms and the macOS CI run validates that the change doesn't regress Unix.