Skip to content
This repository has been archived by the owner on Dec 15, 2022. It is now read-only.

Tokenize punctuation (and generics) #22

Merged
merged 24 commits into from
Apr 19, 2016
Merged

Tokenize punctuation (and generics) #22

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
3273608
Tokenize braces
50Wliu Oct 1, 2015
78562b2
Tokenize brackets in arrays
50Wliu Nov 3, 2015
ebfe99f
Merge remote-tracking branch 'refs/remotes/origin/master' into wl-braces
50Wliu Nov 3, 2015
9312a2c
:green_heart: Fix specs!
50Wliu Dec 9, 2015
72bd720
Tokenize generics
50Wliu Dec 10, 2015
9a7343a
Merge remote-tracking branch 'master' into wl-braces
50Wliu Dec 10, 2015
af7535f
Tokenize commas in method headers
50Wliu Dec 10, 2015
a67ef17
Commas are now punctuation.separator.delimiter
50Wliu Dec 10, 2015
1c594bb
Tokenize commas everywhere
50Wliu Dec 10, 2015
2d33c68
:green_heart: Add basic punctuation specs
50Wliu Dec 11, 2015
1b319c0
Tokenize periods and ampersands
50Wliu Dec 12, 2015
1bf118e
Tokenize generic object types
50Wliu Jan 5, 2016
fade67f
:fire: Extra newline
50Wliu Jan 5, 2016
dc42c5c
Square brackets can be inside generics
50Wliu Jan 13, 2016
f424930
Fix types being matched incorrectly
50Wliu Jan 13, 2016
626e4be
Tokenize commas after `implements`
50Wliu Jan 14, 2016
19cbf02
Rework package tokenization
50Wliu Feb 1, 2016
0e2a197
Add specs!
50Wliu Feb 1, 2016
2cc799c
Tokenize multi-dimensional arrays
50Wliu Feb 2, 2016
2ce4d1f
Also tokenize object multidimensional arrays
50Wliu Feb 2, 2016
ccba156
Remove unneeded comma match
50Wliu Feb 2, 2016
6b9bcb1
Rewrite import regexes
50Wliu Feb 2, 2016
91476f4
punctuation.wildcard -> variable.language.wildcard
50Wliu Feb 3, 2016
e44ccc5
Update tokenizations
50Wliu Feb 5, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
204 changes: 141 additions & 63 deletions grammars/java.cson
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,13 @@
'patterns': [
{
'begin': '\\['
'beginCaptures':
'0':
'name': 'meta.brace.square.java'
'end': '\\]'
'endCaptures':
'0':
'name': 'meta.brace.square.java'
'patterns': [
{
'include': '#code'
Expand Down Expand Up @@ -178,9 +184,12 @@
{
'begin': '\\('
'beginCaptures':
'1':
'name': 'storage.type.java'
'0':
'name': 'meta.brace.round.java'
'end': '\\)'
'endCaptures':
'0':
'name': 'meta.brace.round.java'
'patterns': [
{
'include': '#code'
Expand Down Expand Up @@ -237,6 +246,9 @@
{
'include': '#storage-modifiers'
}
{
'include': '#generics'
}
{
'include': '#comments'
}
Expand Down Expand Up @@ -287,7 +299,7 @@
'0':
'name': 'punctuation.section.class.begin.java'
'end': '(?=})'
'name': 'meta.class.body.java'
'contentName': 'meta.class.body.java'
'patterns': [
{
'include': '#class-body'
Expand All @@ -309,6 +321,9 @@
{
'include': '#variables'
}
{
'include': '#generics'
}
{
'include': '#methods'
}
Expand Down Expand Up @@ -452,6 +467,45 @@
'name': 'constant.other.java'
}
]
'generics':
'begin': '<'
'beginCaptures':
'0':
'name': 'meta.brace.angle.java'
'end': '>'
'endCaptures':
'0':
'name': 'meta.brace.angle.java'
'patterns': [
{
'match': '\\b(extends|super)\\b'
'name': 'storage.modifier.$1.java'
}
{
'match': '([a-zA-Z$_][a-zA-Z0-9$_]*)(?=\\s*<)'
'captures':
'1':
'name': 'storage.type.java'
}
{
'match': '[a-zA-Z$_][a-zA-Z0-9$_]*'
'name': 'storage.type.generic.java'
}
{
'match': '\\?'
'name': 'storage.type.generic.wildcard.java'
}
{
'match': ','
'name': 'punctuation.definition.separator.parameters.java'
}
{
'include': '#generics'
}
{
'include': '#comments'
}
]
'enums':
'begin': '^\\s*(enum)\\s+(\\w+)'
'beginCaptures':
Expand All @@ -465,9 +519,6 @@
'name': 'punctuation.section.enum.end.java'
'name': 'meta.enum.java'
'patterns': [
{
'include': '#parens'
}
{
'begin': '{'
'beginCaptures':
Expand Down Expand Up @@ -591,11 +642,16 @@
'include': '#storage-modifiers'
}
{
'begin': '(\\w+)\\s*\\('
'begin': '(\\w+)\\s*(\\()'
'beginCaptures':
'1':
'name': 'entity.name.function.java'
'2':
'name': 'punctuation.definition.parameters.begin.java'
'end': '\\)'
'endCaptures':
'0':
'name': 'punctuation.definition.parameters.end.java'
'name': 'meta.method.identifier.java'
'patterns': [
{
Expand All @@ -604,20 +660,7 @@
]
}
{
'begin': '<'
'end': '>'
'name': 'storage.type.token.java'
'patterns': [
{
'include': '#object-types'
}
{
'begin': '<'
'comment': 'This is just to support <>\'s with no actual type prefix'
'end': '>|[^\\w\\s,\\[\\]<]'
'name': 'storage.type.generic.java'
}
]
'include': '#generics'
}
{
'begin': '(?=\\w.*\\s+\\w+\\s*\\()'
Expand Down Expand Up @@ -652,34 +695,21 @@
'object-types':
'patterns': [
{
'begin': '\\b((?:[a-z]\\w*\\.)*[A-Z]+\\w*)<'
'end': '>|[^\\w\\s,\\?<\\[\\]]'
'name': 'storage.type.generic.java'
'patterns': [
{
'include': '#object-types'
}
{
'begin': '<'
'comment': 'This is just to support <>\'s with no actual type prefix'
'end': '>|[^\\w\\s,\\[\\]<]'
'name': 'storage.type.generic.java'
}
]
'include': '#generics'
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I couldn't find a way to keep this name with the new generics include. @kevinsawicki do you know if this should be possible, or was it intentionally not allowed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you know if this should be possible, or was it intentionally not allowed?

At first glance, it does appear this would have to be removed to get the new generics patterns working. Nothing jumps out to my as to why this scope would be helpful so I think it is okay to remove it.

{
'begin': '\\b((?:[a-z]\\w*\\.)*[A-Z]+\\w*)(?=\\[)'
'end': '(?=[^\\]\\s])'
'begin': '\\b((?:[a-z]\\w*\\.)*[A-Z]+\\w*)(\\[)'
'beginCaptures':
'2':
'name': 'meta.brace.square.java'
'end': '\\]'
'endCaptures':
'0':
'name': 'meta.brace.square.java'
'name': 'storage.type.object.array.java'
'patterns': [
{
'begin': '\\['
'end': '\\]'
'patterns': [
{
'include': '#code'
}
]
'include': '#code'
}
]
}
Expand All @@ -694,20 +724,7 @@
'object-types-inherited':
'patterns': [
{
'begin': '\\b((?:[a-z]\\w*\\.)*[A-Z]+\\w*)<'
'end': '>|[^\\w\\s,<]'
'name': 'entity.other.inherited-class.java'
'patterns': [
{
'include': '#object-types'
}
{
'begin': '<'
'comment': 'This is just to support <>\'s with no actual type prefix'
'end': '>|[^\\w\\s,<]'
'name': 'storage.type.generic.java'
}
]
'include': '#generics'
}
{
'captures':
Expand Down Expand Up @@ -741,18 +758,63 @@
}
]
'parens':
'begin': '\\('
'end': '\\)'
'patterns': [
{
'include': '#code'
'begin': '\\('
'beginCaptures':
'0':
'name': 'meta.brace.round.java'
'end': '\\)'
'endCaptures':
'0':
'name': 'meta.brace.round.java'
'patterns': [
{
'include': '#code'
}
]
}
{
'begin': '\\['
'beginCaptures':
'0':
'name': 'meta.brace.square.java'
'end': '\\]'
'endCaptures':
'0':
'name': 'meta.brace.square.java'
'patterns': [
{
'include': '#code'
}
]
}
{
'begin': '{'
'beginCaptures':
'0':
'name': 'meta.brace.curly.java'
'end': '}'
'endCaptures':
'0':
'name': 'meta.brace.curly.java'
'patterns': [
{
'include': '#code'
}
]
}
]
'primitive-arrays':
'patterns': [
{
'match': '\\b(?:void|boolean|byte|char|short|int|float|long|double)(\\[\\])*\\b'
'match': '\\b(?:void|boolean|byte|char|short|int|float|long|double)(?:(\\[)|(\\]))*\\b'
'name': 'storage.type.primitive.array.java'
'captures':
'1':
'name': 'meta.brace.square.java'
'2':
'name': 'meta.brace.square.java'
}
]
'primitive-types':
Expand Down Expand Up @@ -832,7 +894,23 @@
'applyEndPatternLast': 1
'patterns': [
{
'begin': '(?x:(?=\n (?:\n (?:private|protected|public|native|synchronized|abstract|threadsafe|transient|static|final) # visibility/modifier\n |\n (?:def)\n |\n (?:void|boolean|byte|char|short|int|float|long|double)\n |\n (?:(?:[a-z]\\w*\\.)*[A-Z]+\\w*) # object type\n )\n \\s+\n (?!private|protected|public|native|synchronized|abstract|threadsafe|transient|static|final|def|void|boolean|byte|char|short|int|float|long|double)\n [\\w\\d_<>\\[\\],\\?][\\w\\d_<>\\[\\],\\? \\t]*\n (?:=|$)\n \n\t\t\t\t\t))'
'begin': '''
(?x:(?=
(?:
(?:private|protected|public|native|synchronized|abstract|threadsafe|transient|static|final) # visibility/modifier
|
(?:def)
|
(?:void|boolean|byte|char|short|int|float|long|double)
|
(?:(?:[a-z]\\w*\\.)*[A-Z]+\\w*) # object type
)
\\s+
(?!private|protected|public|native|synchronized|abstract|threadsafe|transient|static|final|def|void|boolean|byte|char|short|int|float|long|double)
[\\w\\d_<>\\[\\],\\?][\\w\\d_<>\\[\\],\\?\\s]*
(?:=|$)
))
'''
'end': '(?=;)'
'name': 'meta.definition.variable.java'
'patterns': [
Expand Down