New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Unkown token when directory contains multibyte characters #1422

SIkebe opened this Issue Dec 27, 2016 · 1 comment


None yet
2 participants

SIkebe commented Dec 27, 2016

What You Are Seeing?

I upgraded cake version 0.15.2 to 0.17.0
When I run build.ps1 I get the error "Error: Unknown token".
This error occured in 0.16.0 or later version.
Probably because Cake.Core.IO.Globbing.GlobTokenizer.ScanToken() doesn't allow multibyte characters in the directory.

What is Expected?

Cake scripts should work.

What version of Cake are you using?


Are you running on a 32 or 64 bit system?

tested both

What environment are you running on? Windows? Linux? Mac?


Are you running on a CI Server? If so, which one?


How Did You Get This To Happen? (Steps to Reproduce)

Put any src in a directory that has multibyte characters in the path.
Then build the src using cake scripts.

mkdir C:\マルチバイト
cd C:\マルチバイト
git clone
cd example

Output Log

PS C:\マルチバイト\example> .\build.ps1
Preparing to run build script...
Running build script...
Module directory does not exist.
Analyzing build script...
Analyzing C:/マルチバイト/example/build.cake...
Processing build script...
Installing tools...
Error: System.NotSupportedException: Unknown token
   場所 Cake.Core.IO.Globbing.GlobTokenizer.ScanToken()
   場所 Cake.Core.IO.Globbing.GlobTokenizer.Scan()
   場所 Cake.Core.IO.Globbing.GlobParserContext.Accept()
   場所 Cake.Core.IO.Globbing.GlobParser.Parse(GlobParserContext context)
   場所 Cake.Core.IO.Globbing.GlobParser.Parse(String pattern, Boolean caseSensitive)
   場所 Cake.Core.IO.Globber.Match(String pattern, Func`2 predicate)
   場所 Cake.Core.IO.GlobberExtensions.Match(IGlobber globber, String pattern)
   場所 Cake.Core.IO.GlobberExtensions.GetFiles(IGlobber globber, String pattern)
   場所 Cake.NuGet.NuGetContentResolver.GetFiles(DirectoryPath path, PackageReference package)
   場所 Cake.NuGet.NuGetContentResolver.GetToolFiles(DirectoryPath path, PackageReference package)
   場所 Cake.NuGet.NuGetContentResolver.GetFiles(DirectoryPath path, PackageReference package, PackageType type)
   場所 Cake.NuGet.NuGetPackageInstaller.Install(PackageReference package, PackageType type, DirectoryPath path)
   場所 Cake.Core.Scripting.ScriptProcessor.InstallTools(ScriptAnalyzerResult analyzerResult, DirectoryPath installPath)

   場所 Cake.Core.Scripting.ScriptRunner.Run(IScriptHost host, FilePath scriptPath, IDictionary`2 arguments)
   場所 Cake.Commands.BuildCommand.Execute(CakeOptions options)
   場所 Cake.CakeApplication.Run(CakeOptions options)
   場所 Cake.Program.Main()

This comment has been minimized.


SIkebe commented Apr 12, 2017

This error occurs when cake script uses some tools and tries to get them by default pattern(.exe, .dll).
Cake.Core.IO.Globbing.GlobTokenizer.ScanToken() allows characters that match following Regex pattern.
^[0-9a-zA-Z\\+&%!@(). _-]$
Why are multibyte characters not allowed here?
Non-Western people often uses multibyte characters as a repository name. So very confused with this breaking change...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment