Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unicode in code causes parsing to stop early #214

Closed
Oblynx opened this issue Jun 4, 2019 · 0 comments
Closed

Unicode in code causes parsing to stop early #214

Oblynx opened this issue Jun 4, 2019 · 0 comments

Comments

@Oblynx
Copy link
Contributor

Oblynx commented Jun 4, 2019

Given a code chunk

Weave.CodeChunk("\nΝᵢₙ= 2\nNₛₚ= 2\nszᵢₙ= (5,5)\nszₛₚ= (6,9)\n", 1, 0, 16, "", Dict{Symbol,Any}(:dpi=>96,:fig=>true,:fig_pos=>nothing,:out_width=>nothing,:label=>nothing,:eval=>true,:wrap=>true,:fig_env=>nothing,:term=>false,:hold=>false…), "", "", AbstractString[], Weave.ChunkOutput[])

which contains Unicode characters, calling Weave.(chunk,report,Main) causes parsing to stop early:

Weave.CodeChunk("\nΝᵢₙ= 2\nNₛₚ= 2\nszᵢₙ= (5,5)\nszₛₚ= (6,9)\n", 1, 0, 16, "", Dict{Symbol,Any}(:dpi=>96,:fig=>true,:fig_pos=>nothing,:out_width=>"\\linewidth",:label=>nothing,:eval=>true,:wrap=>true,:fig_env=>nothing,:term=>false,:hold=>false…), "", "", AbstractString[], Weave.ChunkOutput[ChunkOutput("Νᵢₙ= 2\n", "", "", "", AbstractString[]), ChunkOutput("Nₛₚ= 2\n", "", "", "", AbstractString[]), ChunkOutput("szᵢₙ= (5,5)\n", "", "", "", AbstractString[])])

This is due to incorrect UTF-8 string handling in https://github.com/mpastell/Weave.jl/blob/master/src/run.jl#L259
UTF-8 strings don't always encode each character in a single byte.

length counts the number of UTF-8 characters in the string, whereas pos as updated by Meta.parse counts the number of bytes.
Therefore, https://github.com/mpastell/Weave.jl/blob/master/src/run.jl#L259
should be n = sizeof(input).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant