Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #1124 #1278

Merged
merged 1 commit into from
Nov 27, 2019
Merged

Fixes #1124 #1278

merged 1 commit into from
Nov 27, 2019

Conversation

wallysalami
Copy link
Contributor

Highlights Matlab double quoted string

@birkenfeld
Copy link
Member

Is there any escaping of double quotes inside these strings possible?

@wallysalami
Copy link
Contributor Author

Matlab escapes double quotes using two of them consecutively, like this:

text = "Total of ""Hello there"" memes"; % → Total of "Hello there" memes

I still don't know how that regex accepts this case (I copied it from Octave's lexer), but it works!

@Anteru
Copy link
Collaborator

Anteru commented Nov 26, 2019

That should get highlighted as three separate tokens -- "Total of", "Hello there", " memes", instead of one token with the "" being part of the string. It will not break though with a single "" inside the string, but it's not the correct solution either :)

@wallysalami
Copy link
Contributor Author

I thought that would happen, but I tested in my website via python-markdown2 and I got just one token:

Captura de Tela 2019-11-26 às 12 03 13

Maybe markdown2 made some magic there? I don't know. How can I test it using only pygments?

@Anteru
Copy link
Collaborator

Anteru commented Nov 26, 2019

Assuming you have it working locally, you can check using:

python -m pygments -v -l matlab -f raw <<<"your matlab code here"

That will use the matlab lexer and output the raw token stream.

@wallysalami
Copy link
Contributor Author

python3 -m pygments -v -l console -f raw <<< 'text = "Total of ""Hello there"" memes'

results in

Token.Generic.Output 'text = "Total of ""Hello there"" memes\n'

I should define the language as "Matlab" somehow, right?

@Anteru
Copy link
Collaborator

Anteru commented Nov 26, 2019

Sorry, yes, I edited my comment. console should have been matlab, -l foo sets the lexer to foo.

@wallysalami
Copy link
Contributor Author

wallysalami commented Nov 26, 2019

Ok, got it. The following command

python3 -m pygments -v -l matlab -f raw <<< 'text = "Total of ""Hello there"" memes"'

results in

Token.Name	'text'
Token.Text	' '
Token.Punctuation	'='
Token.Text	' '
Token.Literal.String	'"Total of "'
Token.Literal.String	'"Hello there"'
Token.Literal.String	'" memes"'
Token.Text	'\n'

It looks like it was some magic of markdown2 after all. Bummer.

@birkenfeld
Copy link
Member

birkenfeld commented Nov 26, 2019

Well, the HTML formatter coalesces parts with the same token type (actually, with the same CSS style; multiple token types can map to the same CSS style). The raw formatter does not.

In the end, I don't think it matters too much since we're a highlighter, not a lexer whose tokens are fed into a parser.

@birkenfeld birkenfeld merged commit f39476c into pygments:master Nov 27, 2019
@birkenfeld
Copy link
Member

I've fixed up the regex to recognize escapes now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants