Add tilde expansion for windows native executables

[domsleee]

PR Summary

Attempted implementation for #20031.

Tilde expansion here is the same on windows as it is on unix, except that it also expands back slash.
i.e. ~, ~/...., and ~\.... are expanded.

PR Context

This comment: #20031 (comment)

My understanding is that tilde expansion was briefly introduced as part of a larger PR.
But that PR didn't handle double quotes properly, specifically "~" should not be expanded on windows, so it was removed.

There were also other issues with the PSNativePSPathResolution that lead to its removal, specifically around determining if the PSDrive argument was a file or not.

The failing example (from the previous PR) is:

$arg = 'temp:foo'; /bin/echo $arg | Should -be $arg

Expected: 'temp:foo' But was: '/var/folde...'

For tilde expansion, we don't need to consider this information. Tilde (with no quotes) should be always expanded.

We can use an experimental feature here if required, not sure if it is needed 👍

PR Checklist

• PR has a meaningful title
  • Use the present tense and imperative mood when describing your changes
• Summarized changes
• Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
• This PR is ready to merge and is not Work in Progress.
  • If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.
• Breaking changes
  • None
  • OR
  • Experimental feature(s) needed
    • Experimental feature name(s): Perhaps PSNativeWindowsTildeExpansion? It is similar to PSNativePSPathResolution, but with reduced scope.
• User-facing changes
  • Not Applicable: Maybe? It is debatable if this sort of behaviour is a breaking change, it would be strange if people expect tilde expansion to not occur I guess?
  • OR
  • Documentation needed
    • Issue filed:
• Testing - New and feature
  • N/A or can only be tested interactively
  • OR
  • Make sure you've added a new test if existing tests do not effectively test the code changed
• Tooling
  • I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.
  • OR
  • I have considered the user experience from a tooling perspective and opened an issue in the relevant tool repository. This may include:
    • Impact on PowerShell Editor Services which is used in the PowerShell extension for VSCode
      (which runs in a different PS Host).
      • Issue filed:
    • Impact on Completions (both in the console and in editors) - one of PowerShell's most powerful features.
      • Issue filed:
    • Impact on PSScriptAnalyzer (which provides linting & formatting in the editor extensions).
      • Issue filed:
    • Impact on EditorSyntax (which provides syntax highlighting with in VSCode, GitHub, and many other editors).
      • Issue filed:

[domsleee]

@microsoft-github-policy-service agree

[domsleee]

The place it is done for cmdlets like Get-ChildItem is here, and it supports ~, startsWith("~/") and startsWith("~\"):

PowerShell/src/System.Management.Automation/namespaces/LocationGlobber.cs

Lines 4542 to 4552 in dab6ca2

	if (path.IndexOf(StringLiterals.HomePath, StringComparison.Ordinal) == 0)
	{
	// Support the single "~"
	if (path.Length == 1)
	result = true;
	// Support "~/" or "~\"
	else if ((path.Length > 1) &&
	(path[1] == '\\' ||
	path[1] == '/'))
	result = true;
	}

[SteveL-MSFT]

Thanks for the contribution! Please wrap as an ExperimentalFeature as it also helps advertise the new capability and allow changes on feedback not to be a breaking change.

[SteveL-MSFT]

This is where this feature gets a bit complicated and the original implementation was abandoned. Double quotes are expected to be expanded (like if your path has whitespace), but single quotes are treated as literal and never expanded. So this needs to be differentiated.

[domsleee]

Would you expect tilde expansion if it was surrounded with double quotes? Currently pwsh on linux doesn't expand tilde for double quotes:

PowerShell/test/powershell/Language/Scripting/NativeExecution/NativeUnixGlobbing.Tests.ps1

Lines 130 to 133 in ac84c35

	/bin/echo '~' | Should -BeExactly '~'
	/bin/echo "~" | Should -BeExactly '~'
	/bin/echo '~/foo' | Should -BeExactly '~/foo'
	/bin/echo "~/foo" | Should -BeExactly '~/foo'

Also I checked bash, fish, zsh with /bin/echo "~/" and none of them expand it (whereas /bin/echo ~ is expanded). So I think it's reasonable to just follow every other shell here, are there other considerations for how tilde should behave on windows?

Also in a related area, the tab completion around tilde doesn't seem quite right - the behaviour from powershell 5 seems more correct to me.
Some cases I tried out, where $ is cursor position:

Case	bash --norc (5.2)	pwsh -NoProfile (7.4.0)	powershell -NoProfile (5.1)
ls "~/my<tab>	ls "/Users/user/my folder"/$	ls "~\my folder\$"	ls "C:\Users\user\my folder\$"
ls ~/my<tab>	ls ~/my\ folder/$	ls '~\my folder\$'	ls 'C:\Users\user\my folder\$'

Seems like it is new (undesirable) behaviour in 7.4.0, some discussion about it here: #20750

[mklement0]

POSIX-like shells only ever expand an unquoted argument-initial ~, either in isolation or if followed by a an unquoted username (not implemented in PowerShell - see #12387) or an unquoted /, and then possibly followed by an mix of unquoted and quoted strings (any mix of '...', "...", with their usual semantics), so I think that's what PowerShell should do as well - with the addition of also recognizing unquoted \, as stated in the initial post.

Thus, to ensure expansion of ~-prefixed paths parts of which require quoting, only the part after ~/ must be quoted; e.g. /bin/echo ~/'a test' (ditto in PowerShell) or /bin/echo ~/a\ test (/bin/echo ~/a` test in PowerShell)

Note that these cases are currently broken in PowerShell (which currently surface on Unix only):

• Unix: Emulation of ~ (tilde) expansion for native utilities is syntactically incomplete #20754
• They're also broken with respect to tab-completion, which inappropriately turns ~/'a test' into '~/a test', for instance: Broken tab-completion with file or directory names with spaces in the context of tilde-prefixed paths (~) #20851

Indeed, the problem described in #20750 effectively broke tab-completion of ~ on Windows for native-program calls, due to - by well-meaning design - no longer instantly expanding ~ to the value of $HOME.

This PR would make that problem go away - but to resolve #20750 this way (which I personally prefer) requires it to become a stable feature right away.

Find a summary of the solution options at:

• 7.4.0 Breaking Change: Tab expansion no longer converts tildes to full paths, breaking path passing to executables on Windows #20750 (comment)

[JamesWTruher]

@SteveL-MSFT i think this change just makes the UNIX and Windows behaviors similar, where ~ or ~/ would now behave consistently regardless of platform (it's cribbed from the code in lines 399-418). I would probably suggest #else rather than #endif at line 419 which might be clearer. We disambiguate between quoted and non-quoted ~ in lines after 344, so that would remain consistent. We don't currently treat ~ with regard to single and double quotes differently; if it's quoted, it's not expanded.

Now that I think on it, the concern I do have is that we would now have duplicated code which means a fix in one spot may not happen in the other. Perhaps this method should be refactored where we don't duplicate the code.

[pull-request-quantifier-deprecated]

This PR has 59 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

---

Quantification details

Label      : Small
Size       : +48 -11
Percentile : 23.6%

Total files changed: 4

Change summary by file extension:
.cs : +23 -10
.ps1 : +23 -0
.json : +2 -1

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

• Fast and predictable releases to production:
  • Optimal size changes are more likely to be reviewed faster with fewer
    iterations.
  • Similarity in low PR complexity drives similar review times.
• Review quality is likely higher as complexity is lower:
  • Bugs are more likely to be detected.
  • Code inconsistencies are more likely to be detected.
• Knowledge sharing is improved within the participants:
  • Small portions can be assimilated better.
• Better engineering practices are exercised:
  • Solving big problems by dividing them in well contained, smaller problems.
  • Exercising separation of concerns within the code changes.

What can I do to optimize my changes

• Use the PullRequestQuantifier to quantify your PR accurately
  • Create a context profile for your repo using the context generator
  • Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
  • Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
  • Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
• Change your engineering behaviors
  • For PRs that fall outside of the desired spectrum, review the details and check if:
    • Your PR could be split in smaller, self-contained PRs instead
    • Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

• One line was added: +1 -0
• One line was deleted: +0 -1
• One line was modified: +1 -1 (git diff doesn't know about modified, it will
  interpret that line like one addition plus one deletion)
• Change percentiles: Change characteristics (addition, deletion, modification)
  of this PR in relation to all other PRs within the repository.

---

Was this comment helpful? 👍  :ok_hand:  :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

[SteveL-MSFT]

LGTM

[JamesWTruher]

nice!

[bradwilson]

@SteveL-MSFT @JamesWTruher @domsleee The fact that PowerShell on Linux quotes space-containing paths during tab expansion is broken behavior since 7.4.0, because it breaks translation of ~ when calling executables.

This PR will now break Windows in the same way, because of the same tab expansion behavior.

Additionally, trying to use PowerShell's native escaping for spaces to work around this behavior on Linux is also currently broken:

In my unvarnished opinion, the dumpster fire around tab expansion behavior with ~ introduced in 7.4.0 was ill thought out, and this one PR is not remotely a complete fix, even on Windows. Frankly: where are the test cases for all of this? How was this feature allowed out into the wild without realizing how many things were being broken, and how is the team working to resolve them before 7.5.0? The fact that the team has been 100% silent on #20750 is extremely disappointing, given that the issue has been open since November 2023. Will anybody ever respond?

[mklement0]

While I share your frustration around the lack of responses / a decision on #20750,
what you're describing are known bugs that must be fixed separately and their existence is not an argument against the value or implementation of this PR:

• Broken tab-completion with file or directory names with spaces in the context of tilde-prefixed paths (~) #20851
• Unix: Emulation of ~ (tilde) expansion for native utilities is syntactically incomplete #20754

[bradwilson]

@mklement0 Not to be overly pessimistic, but neither of your issues have any comment from anybody on the team, either. Right now you're we're basically hoping they agree and create (or allow) a fix in time for 7.5.0.

[mklement0]

Fair point, @bradwilson - let's hope that this exchange helps.

[MatejKafka]

The behavior of this implementation of tilde expansion seems utterly unintuitive to me and breaks basic assumptions that I (and I assume many other PowerShell users) have:

$Path = "~"
cmd /c echo $Path # C:\Users\admin
cmd /c echo "$Path" # ~

One of the things I like about PowerShell is that I don't have to think about quoting rules – if I want to pass a variable as a parameter, I just pass a parameter and don't worry about quoting. This feature breaks that, and means that whenever I'm passing a variable to a native command, I have to actively think whether I want to expand ~ and quote accordingly. PowerShell is complex enough as it is, and I don't see any way this feature can pull its weight. I disagree with the original implementation of this feature, where ~ was always expanded, but at least it was consistent, unlike this implementation.

I first encountered this feature when doing something like

ssh <host> cp ... ~/...

as part of a longer command and receiving mysterious errors that C:Usersadmin... is not an existing directory, which took me quite a lot of time to debug. I would not intuitively expect that ~ will be expanded in this context, much less so when it wrapped in a PowerShell function and the ~ is passed as a parameter.

---

In POSIX shells, ~ is expanded immediately when written, and not when passed to a native command:

x1=~
x2="~"
echo $x1 # /home/...
echo "$x1" # /home/...
echo $x2 # ~
echo "$x2" # ~

This is not ideal, but it is much more intuitive than expanding ~ in the command invocation. I understand why PowerShell cannot resolve ~ in the same way.

I'd personally happily sacrifice the resolution of ~ when calling native commands if it means that PowerShell quoting rules will remain simple and interact nicely with remote ssh calls and other places where literal ~ is passed. I understand that this is a bigger problem on Linux compared to Windows, but tab-expanding ~ seems like a safer, simpler and well-tested alternative.

[mklement0]

The ssh example is an interesting one, as it mixes the worlds of Windows and Unix-like platforms.

Were you to run your example command on Unix, you would have to escape or quote the ~ to prevent instant expansion or - and with this feature you'll now need to do it on Windows too.

• In POSIX-compatible shells, you'd have to do echo \~ or echo '~', for instance.
• While /bin/echo '~' / cmd /c echo '~' also prevents instant expansion from PowerShell, the echo \~ equivalent, /bin/echo `~/ cmd /c echo `~ currently does not: see Unix-like platforms: native globbing and tilde expansion cannot be prevented with backtick-escaping #21304

Similarly, expansion still happening via variables - $p = "~"; /usr/bin/echo $p / $p = "~"; cmd /c echo $p should be considered a bug too, so I've created:

• Unix-like platforms, Windows with the experimental PSNativeWindowsTildeExpansion feature: Native tilde expansion unexpectedly expands tilde in variable references #21323

[MatejKafka]

Were you to run your example command on Unix, you would have to escape or quote the ~ to prevent instant expansion or - and with this feature you'll now need to do it on Windows too.

Yes, but I use PowerShell because it's better than Unix shells. :)

More seriously, PowerShell takes different approach than Unix-style shells, and not having to quote variables is an important part of what makes PowerShell more intuitive and comfortable for daily usage. In Unix shells, you're expected to quote more-or-less everything, which is not expected in PowerShell, and this PR makes PowerShell quoting more complex, for imo lesser benefit.

I understand the desire for consistent ~ behavior between cmdlets and native commands, but adding special cases for quoting (a much more commonly used and core feature of PowerShell) to achieve that seems too great of a sacrifice to me.