Add more syntaxes to language Julia #16

singularitti · 2019-09-01T22:08:22Z

Hi @noplay. Thank you so much for such a good extension!

In my last issue (#15), I forgot to mention some other syntaxes that also require ends. I have added them here.

One fix is that Julia accepts what in ' and ' as a character, so there is no possibility for an end inside ''. Besides, Julia does support multi-line comment, starting with #= and ending with =#. So I replaced ' with them.

The only thing that I cannot handle is that Julia also supports Python-like generator expressions, which means you could write for without an end:

julia> sum(1/n^2 for n=1:1000)
1.6439345666815615

julia> [(i,j) for i=1:3 for j=1:i if i+j == 4]
2-element Array{Tuple{Int64,Int64},1}:
 (2, 2)
 (3, 1)

That breaks the highlighting. The for will match another end. Is there a way to solve this problem?

polvalente · 2019-09-01T22:25:03Z

@singularitti in version 0.7, there are now options for "ignore blocks", such as comments (both single and multiline) and strings!

polvalente · 2019-09-01T22:27:52Z

src/languages.ts

      }
    ],
    inlineOpenTokens: [],
    openTokens: [
      "if",
      "struct",
+      "abstract type",
+      "primitive type",
      "begin",
      "let",
      "for",


@singularitti Also, if non-generator-like for requires do or another token, you could change this to:

Suggested change

"for",

"for(?=.*do)",

This should now only match expressions like: for ... do

Hi, I am trying to use this pattern "^\s*for" (A for with only blank characters before it) to replace "for", but it does not seem to succeed. The below generator expression

[getfield(x, :a) for x in object.b.c],

is still treated as a for loop.

As an example, I only want the first for be matched.

I'm not familiar with Julia. Why the for preceeded by @SoftScope shouldn't be matched? This might bring problems with the matching end for it.

Check if this token works well enough for for:

(?<=\\[[^\\]]*)for(?![^\\]]*\\]))

It uses negative lookahead and positive lookbehind to try and match those generators.
I don't know if it can handle nested or multiline generators.

Another option is adding [ and ] as open and close tokens for an ignore block. This is not ideal, but would avoid ugly breakage for the rest of the file

Hi @polvalente, thank you for your reply. I am not an expert in regex, so I just wrote the simplest example. Yes, indeed, that will break the highlighting.

I cannot figure out a good regex to distinguish the for loops and generator expressions. Because Julia is so flexible that it allows a lot of ways to write a for loop. The only difference between them just seems to be for loops have an end. I will list some test examples below. I hope you can figure out a working way to distinguish them. Thank you!

Below are normal for loops so should be paired with ends.

# Those should be matched. for i = 1:10 s += i end @softscope for i in 1:10 s += i end softscope(Main, :(for i = 1:10 s += i end)) for i in 1:10; println(i); end for i in [1, 2, 3] println(i) end for i in collect(1:10) println(i) end

Below are generator expressions and are not paired with ends:

# Those should not be matched. sum(1/n^2 for n=1:1000) map(tuple, (1/(i+j) for i=1:2, j=1:2), [1 3; 2 4]) [(i,j) for i=1:3 for j=1:i if i+j == 4]

Oh wow. I think there is a problem with how I implemented ignore blocks. I'll refactor it in the coming weekend.

Basically, I didn't think to support nesting. As such, ((x) for) will be passed to the coloring function as for) (because the first closing paren will close the first open paren).

Luckily, I already have an idea on how to solve it (by using stacks).

Nevermind the weekend: #19
HAHA

Cool! Cannot wait to have a try!

I have tried this method. It seems that with the following test text:

sum(1/n^2 for n=1:1000) map(tuple, (1/(i+j) for i=1:2, j=1:2), [1 3; 2 4]) [(i,j) for i=1:3 for j=1:i if i+j == 4] for i = 1:10 s += i end @softscope for i in 1:10 s += i end softscope(Main, :(for i = 1:10 s += i end)) for i in 1:10; println(i); end for i in [1, 2, 3] println(i) end for i in collect(1:10) println(i) end

The map(tuple, (1/(i+j) for i=1:2, j=1:2), [1 3; 2 4]) is not working because there is one ) before the for.

This test now works:

src/languages.ts

singularitti · 2019-09-02T05:13:50Z

@singularitti in version 0.7, there are now options for "ignore blocks", such as comments (both single and multiline) and strings!

Hi, could you tell me how to use it? Thank you!

polvalente · 2019-09-02T15:21:26Z

@singularitti in version 0.7, there are now options for "ignore blocks", such as comments (both single and multiline) and strings!

Hi, could you tell me how to use it? Thank you!

There are some examples in other languages' specs. It's the ignoreInDelimiters parameter. In short, you always specify an open token and a close token, and for single-line blocks, you just set the closing token to \n

example: https://github.com/noplay/vscode-rainbow-end/blob/aa46bde39c02c1004731a8b05479c457ac5bb762/src/languages.ts#L47-L64

singularitti · 2019-09-02T18:19:38Z

@singularitti in version 0.7, there are now options for "ignore blocks", such as comments (both single and multiline) and strings!

Hi, could you tell me how to use it? Thank you!

There are some examples in other languages' specs. It's the ignoreInDelimiters parameter. In short, you always specify an open token and a close token, and for single-line blocks, you just set the closing token to \n

example:

https://github.com/noplay/vscode-rainbow-end/blob/aa46bde39c02c1004731a8b05479c457ac5bb762/src/languages.ts#L47-L64

Thank you. I am already using this syntax.

src/languages.ts

test to see what would happen Co-Authored-By: Paulo Valente <polvalente@gmail.com>

julien-duponchelle · 2019-09-03T20:51:03Z

0.7.2 releases with the nested block improve.

singularitti · 2019-09-04T06:50:28Z

I found a very strange behavior:
The following code will break the pairing:

"""
# module Test



# Examples
"""
module Test

function aa()
    open(x, "") do io

    end
end # function aa

end

Specifically, once I type in the "" in function aa, the pairing intermediately breaks.

Before

After

However, if I remove the """...""" block, everything works fine.

Before

After

I suppose that the "" accidentally matches the """, or it might have something to do with the nesting. Could you reproduce it?

If I only input 1 ", it works fine:

polvalente · 2019-09-04T07:13:49Z

Maybe there is an "off-by-one" error happening with the new ignore-blocks implementation. I might only be able to look into that in the weekend. If you'd like to give it a go, I suspect the problem is during the slicing to replace the text with spaces.

polvalente · 2019-09-07T01:52:17Z

@singularitti

Maybe there is an "off-by-one" error happening with the new ignore-blocks implementation. I might only be able to look into that in the weekend. If you'd like to give it a go, I suspect the problem is during the slicing to replace the text with spaces.

Turns out the problem was the way the delimiter pairing was implemented in the new version. It broke when both the opening and closing delimiters were equal to each other. #20 attempts to fix this problem.

singularitti · 2019-09-07T03:04:00Z

Thank you. I just found another example that breaks the highlighting:

using Parameters

macro xxx(dict)
    quote
        replace("$(dict...)", "=>" => "=")
    end
end

It seems also caused by ":

polvalente · 2019-09-07T12:41:18Z

Thank you. I just found another example that breaks the highlighting:
using Parameters

macro xxx(dict)
    quote
        replace("$(dict...)", "=>" => "=")
    end
end
It seems also caused by ":

It seems my PR also fixed this example:

singularitti · 2019-09-09T20:39:08Z

The following example is broken after I sync PR #20

module XXXX

primitive type Float16 <: AbstractFloat 16 end
primitive type Float32 <: AbstractFloat 32 end
primitive type Float64 <: AbstractFloat 64 end

primitive type Bool <: Integer 8 end
primitive type Char <: AbstractChar 32 end

primitive type Int8    <: Signed   8 end
primitive type UInt8   <: Unsigned 8 end
primitive type Int16   <: Signed   16 end
primitive type UInt16  <: Unsigned 16 end
primitive type Int32   <: Signed   32 end
primitive type UInt32  <: Unsigned 32 end
primitive type Int64   <: Signed   64 end
primitive type UInt64  <: Unsigned 64 end
primitive type Int128  <: Signed   128 end
primitive type UInt128 <: Unsigned 128 end

abstract type Number end
abstract type Real <: Number end

struct Point end
struct Point <: AbstractPoint; x; y; end

end

As you can see, all the abstract types and primitive types are not matching their ends. This is not a problem of writing them in one line since structs match their ends. I guess there might be a problem with the space inside the tokens. I change the source code to

openTokens: [
      ...,
      "abstract\s+type",
      "primitive\s+type",
      ...
    ],

but it does not work.

singularitti · 2019-09-09T21:18:50Z

Another example that will break the highlighting:

module PWscf

# =============================== AAA ============================== #
struct AAA{A<:AbstractString,B<:Real,C<:AbstractString}
    a::A
    b::B
    c::C
end

end

It seems to be caused by the comment line:

# =============================== AAA ============================== #

The second # seems to be responsible:

However, this is often a template commenting style, so I hope it could be fixed.

polvalente · 2019-09-10T00:25:45Z

Another example that will break the highlighting:
module PWscf

# =============================== AAA ============================== #
struct AAA{A<:AbstractString,B<:Real,C<:AbstractString}
    a::A
    b::B
    c::C
end

end
It seems to be caused by the comment line:
# =============================== AAA ============================== #
The second # seems to be responsible:

However, this is often a template commenting style, so I hope it could be fixed.

@noplay Maybe we could refactor the token matching spec to accept whole regexes and match each one individually, instead of building the regex from strings. I've encountered some problems in dealing with some of Elixir's syntax that have arisen from this too.

polvalente · 2019-09-10T00:26:43Z

Another example that will break the highlighting:
module PWscf

# =============================== AAA ============================== #
struct AAA{A<:AbstractString,B<:Real,C<:AbstractString}
    a::A
    b::B
    c::C
end

end
It seems to be caused by the comment line:
# =============================== AAA ============================== #
The second # seems to be responsible:

However, this is often a template commenting style, so I hope it could be fixed.

I guess the \s spec should have been abstract\\s+type instead. All backslashes in the spec need to be escaped.

singularitti · 2019-09-12T22:58:52Z

Another example that will break the highlighting:
module PWscf

# =============================== AAA ============================== #
struct AAA{A<:AbstractString,B<:Real,C<:AbstractString}
    a::A
    b::B
    c::C
end

end
It seems to be caused by the comment line:
# =============================== AAA ============================== #
The second # seems to be responsible:

However, this is often a template commenting style, so I hope it could be fixed.
I guess the \s spec should have been abstract\\s+type instead. All backslashes in the spec need to be escaped.

Yes, adding \\s+ fixes this bug:

as mentioned in #16 (comment)

singularitti · 2019-09-12T23:03:17Z

Another example that will break the highlighting:
module PWscf

# =============================== AAA ============================== #
struct AAA{A<:AbstractString,B<:Real,C<:AbstractString}
    a::A
    b::B
    c::C
end

end
It seems to be caused by the comment line:
# =============================== AAA ============================== #
The second # seems to be responsible:

However, this is often a template commenting style, so I hope it could be fixed.
@noplay Maybe we could refactor the token matching spec to accept whole regexes and match each one individually, instead of building the regex from strings. I've encountered some problems in dealing with some of Elixir's syntax that have arisen from this too.

We still have 1 bug to fix.

polvalente · 2019-09-13T14:27:55Z

@singularitti I know. I'm kinda overwhelmed with things at work, but I'll get to it as soon as possible.

My mention to @noplay was intended as a long term bug fix, because there are some things that can't really be matched by regexes as of now.

I already have some ideas, such as implementing an option to specify raw regexes instead of building an option regex from words.

However, this will demand some work.

Anyway, I'll try to fix this specific problem independently of this

singularitti · 2019-09-13T19:45:22Z

@singularitti I know. I'm kinda overwhelmed with things at work, but I'll get to it as soon as possible.

My mention to @noplay was intended as a long term bug fix, because there are some things that can't really be matched by regexes as of now.

I already have some ideas, such as implementing an option to specify raw regexes instead of building an option regex from words.

However, this will demand some work.

Anyway, I'll try to fix this specific problem independently of this

No hurry. I was just writing down a "TODO". Take your time!

polvalente · 2019-10-04T02:22:21Z

@singularitti and @noplay, just an update on this:
Next week I'll be working on a proposal for a new matching system to avoid some workarounds and problems we've been having. Hopefully, this will solve the issues we have with the current framework.

I'll draft a proposal as an issue and link this PR there.

julien-duponchelle · 2019-10-06T10:05:39Z

Sorry to be unresponsive, I'm super busy. But fully support your initiatives. Le ven. 4 oct. 2019 à 04:22, Paulo Valente <notifications@github.com> a écrit :

…

@singularitti <https://github.com/singularitti> and @noplay <https://github.com/noplay>, just an update on this: Next week I'll be working on a proposal for a new matching system to avoid some workarounds and problems we've been having. Hopefully, this will solve the issues we have with the current framework. I'll draft a proposal as an issue and link this PR there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACUKXLJXIXCMDURUGPFBXLQM2SF5ANCNFSM4ISYEOMQ> .

polvalente · 2019-10-07T05:12:37Z

Sorry to be unresponsive, I'm super busy. But fully support your initiatives. Le ven. 4 oct. 2019 à 04:22, Paulo Valente notifications@github.com a écrit :
…
@singularitti https://github.com/singularitti and @noplay https://github.com/noplay, just an update on this: Next week I'll be working on a proposal for a new matching system to avoid some workarounds and problems we've been having. Hopefully, this will solve the issues we have with the current framework. I'll draft a proposal as an issue and link this PR there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16?email_source=notifications&email_token=AACUKXILM4ZOV3T4EHT6BZTQM2SF5A5CNFSM4ISYEOM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAKEWNI#issuecomment-538200885>, or mute the thread https://github.com/notifications/unsubscribe-auth/AACUKXLJXIXCMDURUGPFBXLQM2SF5ANCNFSM4ISYEOMQ .

No worries! I was pretty busy as well, but things have settled by now. I hope to have something to show by Friday!

polvalente · 2019-10-08T04:46:34Z

@singularitti @noplay #21
New features incoming (still WIP), to deal with the list comprehension issue.

xtalax · 2021-12-09T19:09:43Z

What is the status of this PR? Even with the outstanding bugs it seems to be an improvement. I support this being merged and will be happy to contribute to further Julia support.

julien-duponchelle · 2022-01-17T07:19:22Z

@xtalax you are right I will merge like this

Add more syntaxes to language Julia

aa46bde

polvalente reviewed Sep 1, 2019

View reviewed changes

src/languages.ts Show resolved Hide resolved

polvalente reviewed Sep 3, 2019

View reviewed changes

src/languages.ts Show resolved Hide resolved

Update src/languages.ts

d42b527

test to see what would happen Co-Authored-By: Paulo Valente <polvalente@gmail.com>

This was referenced Sep 3, 2019

[BUG] Ignore Blocks do not support nesting #18

Closed

fix: support ignore block nesting #19

Merged

Merge remote-tracking branch 'upstream/master'

5f3ab57

polvalente mentioned this pull request Sep 7, 2019

fix: only allow nesting if delimiters are different #20

Merged

Merge remote-tracking branch 'upstream/master'

ce3d7f9

Fix a breaking caused by whitespace

ed99e87

as mentioned in #16 (comment)

polvalente mentioned this pull request Oct 4, 2019

Replace regex parser by a more robust implementation #1

Open

singularitti mentioned this pull request Aug 28, 2020

[feature request] Rainbow ends support? julia-vscode/julia-vscode#1635

Open

Merge branch 'master' into master

2ae6d94

julien-duponchelle merged commit d593646 into julien-duponchelle:master Jan 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more syntaxes to language Julia #16

Add more syntaxes to language Julia #16

singularitti commented Sep 1, 2019 •

edited

Loading

polvalente commented Sep 1, 2019

polvalente Sep 1, 2019

singularitti Sep 2, 2019

singularitti Sep 2, 2019

polvalente Sep 2, 2019 •

edited

Loading

singularitti Sep 3, 2019 •

edited

Loading

polvalente Sep 3, 2019

polvalente Sep 3, 2019

polvalente Sep 3, 2019

singularitti Sep 3, 2019

singularitti Sep 3, 2019

singularitti commented Sep 2, 2019

polvalente commented Sep 2, 2019 •

edited

Loading

singularitti commented Sep 2, 2019

julien-duponchelle commented Sep 3, 2019

singularitti commented Sep 4, 2019 •

edited

Loading

polvalente commented Sep 4, 2019

polvalente commented Sep 7, 2019

singularitti commented Sep 7, 2019 •

edited

Loading

polvalente commented Sep 7, 2019 •

edited

Loading

singularitti commented Sep 9, 2019

singularitti commented Sep 9, 2019

polvalente commented Sep 10, 2019

polvalente commented Sep 10, 2019

singularitti commented Sep 12, 2019 •

edited

Loading

singularitti commented Sep 12, 2019

polvalente commented Sep 13, 2019

singularitti commented Sep 13, 2019

polvalente commented Oct 4, 2019

julien-duponchelle commented Oct 6, 2019 via email

polvalente commented Oct 7, 2019

polvalente commented Oct 8, 2019

xtalax commented Dec 9, 2021

julien-duponchelle commented Jan 17, 2022

Add more syntaxes to language Julia #16

Add more syntaxes to language Julia #16

Conversation

singularitti commented Sep 1, 2019 • edited Loading

polvalente commented Sep 1, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

polvalente Sep 2, 2019 • edited Loading

Choose a reason for hiding this comment

singularitti Sep 3, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

singularitti commented Sep 2, 2019

polvalente commented Sep 2, 2019 • edited Loading

singularitti commented Sep 2, 2019

julien-duponchelle commented Sep 3, 2019

singularitti commented Sep 4, 2019 • edited Loading

Before

After

Before

After

polvalente commented Sep 4, 2019

polvalente commented Sep 7, 2019

singularitti commented Sep 7, 2019 • edited Loading

polvalente commented Sep 7, 2019 • edited Loading

singularitti commented Sep 9, 2019

singularitti commented Sep 9, 2019

polvalente commented Sep 10, 2019

polvalente commented Sep 10, 2019

singularitti commented Sep 12, 2019 • edited Loading

singularitti commented Sep 12, 2019

polvalente commented Sep 13, 2019

singularitti commented Sep 13, 2019

polvalente commented Oct 4, 2019

julien-duponchelle commented Oct 6, 2019 via email

polvalente commented Oct 7, 2019

polvalente commented Oct 8, 2019

xtalax commented Dec 9, 2021

julien-duponchelle commented Jan 17, 2022

singularitti commented Sep 1, 2019 •

edited

Loading

polvalente Sep 2, 2019 •

edited

Loading

singularitti Sep 3, 2019 •

edited

Loading

polvalente commented Sep 2, 2019 •

edited

Loading

singularitti commented Sep 4, 2019 •

edited

Loading

singularitti commented Sep 7, 2019 •

edited

Loading

polvalente commented Sep 7, 2019 •

edited

Loading

singularitti commented Sep 12, 2019 •

edited

Loading