refactor: implement lexing and multi-mode parsing engine

[polvalente]

The core idea here is that we'll be able to parse list comprehensions differently than normal code, and comments are now dealt with as they should be. In both cases, I relied on the concept of parsing modes, which ultimately change the way certain tokens are parsed. For example, in default mode, an ìftoken would expect anendtoken to close its block, while incomprehensionmode, it wouldn't, and while incomment` mode, it would be ignored altogether.

[julien-duponchelle]

Impressive ! Le mar. 8 oct. 2019 à 06:45, Paulo Valente <notifications@github.com> a écrit :

…

This is still work in progress, but the main idea is already implemented. TODO.md cites what is still missing (and should be done by the end of the week). The core idea here is that we'll be able to parse list comprehensions differently than normal code, and comments are now dealt with as they should be. In both cases, I relied on the concept of parsing modes, which ultimately change the way certain tokens are parsed. For example, in default mode, an ìftoken would expect anendtoken to close its block, while incomprehensionmode, it wouldn't, and while incomment` mode, it would be ignored altogether. ------------------------------ You can view, comment on, or merge this pull request online at: #21 Commit Summary - refactor: clean-up string matching engine - fix: single line ignore - fix: match elixir properly - wip File Changes - *A* TODO.md <https://github.com/noplay/vscode-rainbow-end/pull/21/files#diff-0> (6) - *M* package.json <https://github.com/noplay/vscode-rainbow-end/pull/21/files#diff-1> (4) - *M* src/extension.ts <https://github.com/noplay/vscode-rainbow-end/pull/21/files#diff-2> (238) - *M* src/languages.ts <https://github.com/noplay/vscode-rainbow-end/pull/21/files#diff-3> (53) - *A* src/tokenizer.ts <https://github.com/noplay/vscode-rainbow-end/pull/21/files#diff-4> (116) Patch Links: - https://github.com/noplay/vscode-rainbow-end/pull/21.patch - https://github.com/noplay/vscode-rainbow-end/pull/21.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#21>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACUKXKIWO2W2I7Z7MRLPQDQNQF45ANCNFSM4I6M3TXA> .

[polvalente]

@noplay I'm gonna need some help with the language definitions. I think that maybe the inlineOpenToken category is not needed anymore, but I reckon you know more of the languages that use it than me.

Other than that, I'm opening the PR to review!

[polvalente]

One improvement that needs addressing before merging is that the nesting for comprehensions is not implemented properly (only counting instead of using stacks).

[singularitti]

Any update? :)

[polvalente]

Any update? :)

I've been kinda busy (and sick) these last weeks, but I intend to get back to this as soon as possible!

[polvalente]

Sorry it took so long, guys. I got sick, and then was busy with other things. Code is now ready for review (and possible merging).

[julien-duponchelle]

Finally got time to test sorry for the long time .

I found two issue. One for ruby. if and unless are not highlited correctly . It's normal because they where implemented in a special list ignoreInDelimiters to support:

        puts "Hello" if a == 3

def test
    if a == 9000 || b < 3
        if c > 5
            if b == 7
            elsif x == 3
            end
        end

        todo.each do |dd|
        end

        puts "Hello" if a == 3
        puts "World" if a == 3
    end
end

The second issue is after changing multiple time to different tab the extension stop working. I'm not sure why. My demo env use a lot of different language (verilog, vhdl...). I'm still trying to figure what is wrong.

[julien-duponchelle]

Awesome work !

[polvalente]

Finally got time to test sorry for the long time .

I found two issue. One for ruby. if and unless are not highlited correctly . It's normal because they where implemented in a special list ignoreInDelimiters to support:

        puts "Hello" if a == 3

def test
    if a == 9000 || b < 3
        if c > 5
            if b == 7
            elsif x == 3
            end
        end

        todo.each do |dd|
        end

        puts "Hello" if a == 3
        puts "World" if a == 3
    end
end

The second issue is after changing multiple time to different tab the extension stop working. I'm not sure why. My demo env use a lot of different language (verilog, vhdl...). I'm still trying to figure what is wrong.

• When in ignoreInDelimiters, the words are ignored by our extension (but should be picked up by another syntax highlighter normally).

• Could you provide some test code for those multiple languages? Maybe there is some memory leaking happening (I don't know exactly how vscode handles extensions with multiple tabs).

In other news, I was thinking of looking into LSP. This should solve most of our problems, if we are able to fetch code block starters and such -- however, I'd still need to look if LSP provides all needed info.

[singularitti]

Any news?