Skip to content

Tokenizer shouldn't throw for invalid UTF-8 #153

@c42f

Description

@c42f

For example,

julia> JuliaSyntax.parse(Expr, "+\xc7")
ERROR: Base.InvalidCharError{Char}('\xc7')
Stacktrace:
  [1] throw_invalid_char(c::Char)
    @ Base ./char.jl:86
  [2] UInt32
    @ ./char.jl:133 [inlined]
  [3] isopsuffix
    @ ~/.julia/dev/JuliaSyntax/src/tokenize_utils.jl:177 [inlined]
  [4] emit(l::JuliaSyntax.Tokenize.Lexer{IOBuffer}, kind::JuliaSyntax.Kind, maybe_op::Bool)
    @ JuliaSyntax.Tokenize ~/.julia/dev/JuliaSyntax/src/tokenize.jl:268
  [5] emit
    @ ~/.julia/dev/JuliaSyntax/src/tokenize.jl:266 [inlined]
  [6] lex_plus(l::JuliaSyntax.Tokenize.Lexer{IOBuffer})
    @ JuliaSyntax.Tokenize ~/.julia/dev/JuliaSyntax/src/tokenize.jl:691
  [7] _next_token(l::JuliaSyntax.Tokenize.Lexer{IOBuffer}, c::Char)
    @ JuliaSyntax.Tokenize ~/.julia/dev/JuliaSyntax/src/tokenize.jl:372
  [8] next_token(l::JuliaSyntax.Tokenize.Lexer{IOBuffer}, start::Bool)
    @ JuliaSyntax.Tokenize ~/.julia/dev/JuliaSyntax/src/tokenize.jl:302
  [9] next_token
    @ ~/.julia/dev/JuliaSyntax/src/tokenize.jl:298 [inlined]
...

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions