Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PowerShell ErrorToken: : #1682

jborean93 opened this issue Jan 14, 2021 · 4 comments · Fixed by #1758

PowerShell ErrorToken: : #1682

jborean93 opened this issue Jan 14, 2021 · 4 comments · Fixed by #1758
A-lexing area: changes to individual lexers T-bug type: a bug


Copy link

Running Pygments==2.7.4 I'm having issues highlighting PowerShell syntax.

It seems like the PowerShell syntax highlighter is failing when it encounters : in the code. A very simple example

echo '$item = Get-ChildItem C:\temp' | pygmentize -l powershell -F raiseonerror -v

Outputs the following

An unhandled exception occurred while highlighting.
Please report the whole traceback to the issue tracker at

$item = Get-ChildItem CTraceback (most recent call last):
  File "/home/jborean/venvs/psrp-39/bin/pygmentize", line 8, in <module>
  File "/home/jborean/venvs/psrp-39/lib/python3.9/site-packages/pygments/", line 557, in main
    return main_inner(popts, args, usage)
  File "/home/jborean/venvs/psrp-39/lib/python3.9/site-packages/pygments/", line 519, in main_inner
    highlight(code, lexer, fmter, outfile)
  File "/home/jborean/venvs/psrp-39/lib/python3.9/site-packages/pygments/", line 84, in highlight
    return format(lex(code, lexer), formatter, outfile)
  File "/home/jborean/venvs/psrp-39/lib/python3.9/site-packages/pygments/", line 66, in format
    formatter.format(tokens, outfile)
  File "/home/jborean/venvs/psrp-39/lib/python3.9/site-packages/pygments/formatters/", line 95, in format
    return Formatter.format(self, tokensource, outfile)
  File "/home/jborean/venvs/psrp-39/lib/python3.9/site-packages/pygments/", line 95, in format
    return self.format_unencoded(tokensource, outfile)
  File "/home/jborean/venvs/psrp-39/lib/python3.9/site-packages/pygments/formatters/", line 115, in format_unencoded
    for ttype, value in tokensource:
  File "/home/jborean/venvs/psrp-39/lib/python3.9/site-packages/pygments/", line 20, in _apply
    yield from filter_.filter(lexer, stream)
  File "/home/jborean/venvs/psrp-39/lib/python3.9/site-packages/pygments/filters/", line 783, in filter
    raise self.exception(value)
pygments.filters.ErrorToken: :

Doing just C\temp works just fine but that's not the code I want to be demonstrating. A colon is used quite a bit in PowerShell so I'm hoping I just have some config option set incorrectly.

Copy link

jborean93 commented Jan 14, 2021

Looks like quoting the string containing the : will work e.g. "C:\temp" or 'C:\temp' so I have a workaround but it would be nice to not have to do so.

It seems like $<scope>:var works without quoting as per

but there are a few other cases I can think off just off the top of my head which does not work currently.

# As mentioned above, using a PSProvider path qualifier, e.g. 'C:', 'cert:', 'wsman:' (can be any user defined prefix
Get-Item -Path C:\temp
Get-Item -Path WSMan:\localhost\service\AllowUnencrypted

# Using ${} when getting a var with path qualifier or special chars, e.g. '${function:Test-Path}'
Invoke-Command computer -ScriptBlock ${function:Test-Path}

I'm just getting started with pygments and PowerShell so I might come across some more. What's the best way to report these issues?

Copy link

Anteru commented Jan 14, 2021

Keep them coming here, ideally with an associated pull request :) There have been a few people looking into Powershell recently so hopefully someone can pick this up.

@Anteru Anteru added the A-lexing area: changes to individual lexers label Jan 14, 2021
Copy link

Thanks, I might play around with it a bit more and see what I can do. Regex isn't my thing but hopefully I can still help a bit.

Copy link

Anteru commented Jan 17, 2021

Thanks! It's hopefully possible to reuse some of the DOS/cmd shell rules for this.

@Anteru Anteru added the T-bug type: a bug label Jan 17, 2021
@Anteru Anteru added this to the 2.8 milestone Jan 17, 2021
@Anteru Anteru modified the milestones: 2.8, 2.9 Feb 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
A-lexing area: changes to individual lexers T-bug type: a bug
None yet

Successfully merging a pull request may close this issue.

2 participants