incorrect highlight of command with variable

[msftrncs]

The syntax highlighting often gives the user the idea of what is going to happen, but in a few situations it seems to indicate the wrong things.

$a='goodbye'
hello$a # 'hello' is yellow, '$a' is green like its going to be expanded, but it doesn't
# instead, "The term 'hello$a' is not recognized as the name of a …" is returned.

I think it is safe to say, everything highlighted yellow, does not allow expansion

function hello () {echo $args}
$a= @{hello=3;goodbye=4}
$a[hello] # 'hello]' is yellow, like it could be a function call, but function calls seem not to be permitted in array index
$a[hello goodbye] # as you type, it looks like its going to be valid, but the prompt knows, its still red at the end.
$a[(hello goodbye)] # this is accepted

This seems to really confuse the syntax highlighting:

$a[(hello$b goodbye)]

Environment data

PS version: 6.1.1
PSReadline version: 2.0.0-beta3
os: 10.0.18298.1000 (WinBuild.160101.0800)
PS file version: 6.1.1

[lzybkr]

The first case is easy to fix, PSReadLine shouldn't look at nested tokens when the TokenFlag CommandName is set on the token.

Your other examples are accurately representing how the PowerShell parser works. When the parser doesn't find an expression for an index, it assumes you're missing a closing ]. It inserts an artificial ] and keeps parsing, finding a valid command invocation.

[msftrncs]

Quickly realized this occurs in a few other situations as well. Its like the syntax highlighting is eager to find a command.

(assume 'hello' is the function defined in the earlier post)

[int]hello 3 # prompt remains red, but 'hello' is yellow
[invalid hello 3 # eager to have a command, 'hello' is yellow
[$b]3 # all the text except $b is yellow, $b is green, but no expansion is allowed here, and the whole thing is actually tokenized as a command name
345+65.com # 65.com becomes yellow, but this is an expression, so commands are no longer possible.
$a = hello 3 # this is valid
$a = 4 + hello 3 # but this is not, yet 'hello' is highlighted yellow
$a = , hello 3 # a command does not seem to work after a comma (its an unary expression operator), yet 'hello' is yellow

In the cases were a command is indicated that shouldn't be, there seems to be no treatment for the explicit case of being in expression mode. It would appear that the syntax highlighter is confident, that the first thing on a command line that is not 'something else' must be a command, where, in reality, the only thing that can be a command, is the first token, and only if its not valid as an expression first.

And so that last statement is not misinterpreted, I understand that expressions, sub-expressions, terminators, and pipelines all return back to argument mode, thus another command could appear on the same statement line; I am only showing statements on a new line for simplicity.

[lzybkr]

All of your subsequent examples are similar - the parser sees something not allowed in an expression and recovers by ending parsing as expression.

It might make a lot more sense if you change the formatting:

# The parser sees this
$a = 4 + hello 3
# no differently than
$a = 4 +
hello 3

And in the later case, the error recovery seems very reasonable.
You can confirm this by looking at the Ast, e.g. with Show-Ast

At any rate, if you feel like discussing the error recovery behavior, that belongs in the PowerShell repo, not here.

[msftrncs]

The first case is easy to fix, PSReadLine shouldn't look at nested tokens when the TokenFlag CommandName is set on the token.

That is, unless the token also includes, ExpandableStringExpression, such as happens with the . (dot source) operator. Looking at the AST, I am not sure how you differentiate that the item following the (dot) is a command, but that is how its being highlighted.

There is also an issue that seems to highlight anything that starts with a '-' as a parameter name, even though it might be a command name. Interestingly enough a '-' on the front eliminates the ExpandableStringExpression. I might have to wonder if that is an error, one way or the other (In the PS tokenizer).

$a='World"
. -hello$a -hello
. hello$a -hello
-hello$a -hello
hello$a - hello
& -hello$a -hello
& hello$a -hello # -hello parameter now highlights as a command

[msftrncs]

For follow up on this, its not possible to prevent the nested tokens using the token flag CommandName because its also set on the expandable argument token following the & and . operators. So it would seem to me that the only changes PSReadLine can make is to handle parameter tokens that have the CommandName flag set.

[msftrncs]

I almost missed my last comment from Feb 2, the last line of code, showing the command name highlight bleeding the rest of the statement:

(image might be from 2.0.0beta1)

At the time, I only entered the comment part here in the markdown, and not on the command line. Revisiting it made the issue that much larger when the comment was included.

[msftrncs]

Also see PowerShell/PowerShell#10275.

[msftrncs]

Closing, as the items PSReadLine is responsible for have been fixed in #989.