Skip to content

Commit 98e3552

Browse files
kddnewtonmatzbot
kddnewton
authored and
matzbot
committed
[ruby/prism] Add necessary encoding flags for symbols and regex
This doesn't actually fix the encodings for symbols and regex, unfortunately. But I wanted to get this change in because it is the last AST change we're going to make before 3.3 is released. So, if consumers want, they can start to check these flags to determine the encoding, even though it will be wrong. Then once we actually set them correctly, everything should work. ruby/prism@9b35f7e891
1 parent a94a2ee commit 98e3552

File tree

148 files changed

+645
-13
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

148 files changed

+645
-13
lines changed

‎prism/config.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -392,6 +392,12 @@ flags:
392392
comment: "s - forces the Windows-31J encoding"
393393
- name: UTF_8
394394
comment: "u - forces the UTF-8 encoding"
395+
- name: FORCED_UTF8_ENCODING
396+
comment: "internal bytes forced the encoding to UTF-8"
397+
- name: FORCED_BINARY_ENCODING
398+
comment: "internal bytes forced the encoding to binary"
399+
- name: FORCED_US_ASCII_ENCODING
400+
comment: "internal bytes forced the encoding to US-ASCII"
395401
comment: Flags for regular expression and match last line nodes.
396402
- name: StringFlags
397403
values:
@@ -402,6 +408,15 @@ flags:
402408
- name: FROZEN
403409
comment: "frozen by virtue of a `frozen_string_literal` comment"
404410
comment: Flags for string nodes.
411+
- name: SymbolFlags
412+
values:
413+
- name: FORCED_UTF8_ENCODING
414+
comment: "internal bytes forced the encoding to UTF-8"
415+
- name: FORCED_BINARY_ENCODING
416+
comment: "internal bytes forced the encoding to binary"
417+
- name: FORCED_US_ASCII_ENCODING
418+
comment: "internal bytes forced the encoding to US-ASCII"
419+
comment: Flags for symbol nodes.
405420
nodes:
406421
- name: AliasGlobalVariableNode
407422
fields:
@@ -2465,6 +2480,9 @@ nodes:
24652480
^^^^^^^^^^^^^^
24662481
- name: SymbolNode
24672482
fields:
2483+
- name: flags
2484+
type: flags
2485+
kind: SymbolFlags
24682486
- name: opening_loc
24692487
type: location?
24702488
- name: value_loc

‎test/prism/errors_test.rb‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -407,7 +407,7 @@ def test_splat_argument_after_keyword_argument
407407
ArgumentsNode(0, [
408408
KeywordHashNode(
409409
[AssocNode(
410-
SymbolNode(nil, Location(), Location(), "foo"),
410+
SymbolNode(0, nil, Location(), Location(), "foo"),
411411
expression("bar"),
412412
nil
413413
)]
@@ -585,16 +585,16 @@ def test_cannot_assign_to_a_reserved_numbered_parameter
585585
expected = BeginNode(
586586
Location(),
587587
StatementsNode([
588-
LocalVariableWriteNode(:_1, 0, Location(), SymbolNode(Location(), Location(), nil, "a"), Location()),
589-
LocalVariableWriteNode(:_2, 0, Location(), SymbolNode(Location(), Location(), nil, "a"), Location()),
590-
LocalVariableWriteNode(:_3, 0, Location(), SymbolNode(Location(), Location(), nil, "a"), Location()),
591-
LocalVariableWriteNode(:_4, 0, Location(), SymbolNode(Location(), Location(), nil, "a"), Location()),
592-
LocalVariableWriteNode(:_5, 0, Location(), SymbolNode(Location(), Location(), nil, "a"), Location()),
593-
LocalVariableWriteNode(:_6, 0, Location(), SymbolNode(Location(), Location(), nil, "a"), Location()),
594-
LocalVariableWriteNode(:_7, 0, Location(), SymbolNode(Location(), Location(), nil, "a"), Location()),
595-
LocalVariableWriteNode(:_8, 0, Location(), SymbolNode(Location(), Location(), nil, "a"), Location()),
596-
LocalVariableWriteNode(:_9, 0, Location(), SymbolNode(Location(), Location(), nil, "a"), Location()),
597-
LocalVariableWriteNode(:_10, 0, Location(), SymbolNode(Location(), Location(), nil, "a"), Location())
588+
LocalVariableWriteNode(:_1, 0, Location(), SymbolNode(0, Location(), Location(), nil, "a"), Location()),
589+
LocalVariableWriteNode(:_2, 0, Location(), SymbolNode(0, Location(), Location(), nil, "a"), Location()),
590+
LocalVariableWriteNode(:_3, 0, Location(), SymbolNode(0, Location(), Location(), nil, "a"), Location()),
591+
LocalVariableWriteNode(:_4, 0, Location(), SymbolNode(0, Location(), Location(), nil, "a"), Location()),
592+
LocalVariableWriteNode(:_5, 0, Location(), SymbolNode(0, Location(), Location(), nil, "a"), Location()),
593+
LocalVariableWriteNode(:_6, 0, Location(), SymbolNode(0, Location(), Location(), nil, "a"), Location()),
594+
LocalVariableWriteNode(:_7, 0, Location(), SymbolNode(0, Location(), Location(), nil, "a"), Location()),
595+
LocalVariableWriteNode(:_8, 0, Location(), SymbolNode(0, Location(), Location(), nil, "a"), Location()),
596+
LocalVariableWriteNode(:_9, 0, Location(), SymbolNode(0, Location(), Location(), nil, "a"), Location()),
597+
LocalVariableWriteNode(:_10, 0, Location(), SymbolNode(0, Location(), Location(), nil, "a"), Location())
598598
]),
599599
nil,
600600
nil,
@@ -995,7 +995,7 @@ def test_returning_to_optional_parameters_multiple_times
995995

996996
def test_case_without_when_clauses_errors_on_else_clause
997997
expected = CaseMatchNode(
998-
SymbolNode(Location(), Location(), nil, "a"),
998+
SymbolNode(0, Location(), Location(), nil, "a"),
999999
[],
10001000
ElseNode(Location(), nil, Location()),
10011001
Location(),
@@ -1009,7 +1009,7 @@ def test_case_without_when_clauses_errors_on_else_clause
10091009

10101010
def test_case_without_clauses
10111011
expected = CaseNode(
1012-
SymbolNode(Location(), Location(), nil, "a"),
1012+
SymbolNode(0, Location(), Location(), nil, "a"),
10131013
[],
10141014
nil,
10151015
Location(),

‎test/prism/snapshots/alias.txt‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,14 @@
66
├── @ AliasMethodNode (location: (1,0)-(1,15))
77
│ ├── new_name:
88
│ │ @ SymbolNode (location: (1,6)-(1,10))
9+
│ │ ├── flags: ∅
910
│ │ ├── opening_loc: (1,6)-(1,7) = ":"
1011
│ │ ├── value_loc: (1,7)-(1,10) = "foo"
1112
│ │ ├── closing_loc: ∅
1213
│ │ └── unescaped: "foo"
1314
│ ├── old_name:
1415
│ │ @ SymbolNode (location: (1,11)-(1,15))
16+
│ │ ├── flags: ∅
1517
│ │ ├── opening_loc: (1,11)-(1,12) = ":"
1618
│ │ ├── value_loc: (1,12)-(1,15) = "bar"
1719
│ │ ├── closing_loc: ∅
@@ -20,12 +22,14 @@
2022
├── @ AliasMethodNode (location: (3,0)-(3,21))
2123
│ ├── new_name:
2224
│ │ @ SymbolNode (location: (3,6)-(3,13))
25+
│ │ ├── flags: ∅
2326
│ │ ├── opening_loc: (3,6)-(3,9) = "%s["
2427
│ │ ├── value_loc: (3,9)-(3,12) = "abc"
2528
│ │ ├── closing_loc: (3,12)-(3,13) = "]"
2629
│ │ └── unescaped: "abc"
2730
│ ├── old_name:
2831
│ │ @ SymbolNode (location: (3,14)-(3,21))
32+
│ │ ├── flags: ∅
2933
│ │ ├── opening_loc: (3,14)-(3,17) = "%s["
3034
│ │ ├── value_loc: (3,17)-(3,20) = "def"
3135
│ │ ├── closing_loc: (3,20)-(3,21) = "]"
@@ -34,12 +38,14 @@
3438
├── @ AliasMethodNode (location: (5,0)-(5,19))
3539
│ ├── new_name:
3640
│ │ @ SymbolNode (location: (5,6)-(5,12))
41+
│ │ ├── flags: ∅
3742
│ │ ├── opening_loc: (5,6)-(5,8) = ":'"
3843
│ │ ├── value_loc: (5,8)-(5,11) = "abc"
3944
│ │ ├── closing_loc: (5,11)-(5,12) = "'"
4045
│ │ └── unescaped: "abc"
4146
│ ├── old_name:
4247
│ │ @ SymbolNode (location: (5,13)-(5,19))
48+
│ │ ├── flags: ∅
4349
│ │ ├── opening_loc: (5,13)-(5,15) = ":'"
4450
│ │ ├── value_loc: (5,15)-(5,18) = "def"
4551
│ │ ├── closing_loc: (5,18)-(5,19) = "'"
@@ -67,6 +73,7 @@
6773
│ │ └── closing_loc: (7,15)-(7,16) = "\""
6874
│ ├── old_name:
6975
│ │ @ SymbolNode (location: (7,17)-(7,23))
76+
│ │ ├── flags: ∅
7077
│ │ ├── opening_loc: (7,17)-(7,19) = ":'"
7178
│ │ ├── value_loc: (7,19)-(7,22) = "def"
7279
│ │ ├── closing_loc: (7,22)-(7,23) = "'"
@@ -83,12 +90,14 @@
8390
├── @ AliasMethodNode (location: (11,0)-(11,13))
8491
│ ├── new_name:
8592
│ │ @ SymbolNode (location: (11,6)-(11,9))
93+
│ │ ├── flags: ∅
8694
│ │ ├── opening_loc: ∅
8795
│ │ ├── value_loc: (11,6)-(11,9) = "foo"
8896
│ │ ├── closing_loc: ∅
8997
│ │ └── unescaped: "foo"
9098
│ ├── old_name:
9199
│ │ @ SymbolNode (location: (11,10)-(11,13))
100+
│ │ ├── flags: ∅
92101
│ │ ├── opening_loc: ∅
93102
│ │ ├── value_loc: (11,10)-(11,13) = "bar"
94103
│ │ ├── closing_loc: ∅
@@ -105,12 +114,14 @@
105114
├── @ AliasMethodNode (location: (15,0)-(15,12))
106115
│ ├── new_name:
107116
│ │ @ SymbolNode (location: (15,6)-(15,9))
117+
│ │ ├── flags: ∅
108118
│ │ ├── opening_loc: ∅
109119
│ │ ├── value_loc: (15,6)-(15,9) = "foo"
110120
│ │ ├── closing_loc: ∅
111121
│ │ └── unescaped: "foo"
112122
│ ├── old_name:
113123
│ │ @ SymbolNode (location: (15,10)-(15,12))
124+
│ │ ├── flags: ∅
114125
│ │ ├── opening_loc: ∅
115126
│ │ ├── value_loc: (15,10)-(15,12) = "if"
116127
│ │ ├── closing_loc: ∅
@@ -119,12 +130,14 @@
119130
├── @ AliasMethodNode (location: (17,0)-(17,13))
120131
│ ├── new_name:
121132
│ │ @ SymbolNode (location: (17,6)-(17,9))
133+
│ │ ├── flags: ∅
122134
│ │ ├── opening_loc: ∅
123135
│ │ ├── value_loc: (17,6)-(17,9) = "foo"
124136
│ │ ├── closing_loc: ∅
125137
│ │ └── unescaped: "foo"
126138
│ ├── old_name:
127139
│ │ @ SymbolNode (location: (17,10)-(17,13))
140+
│ │ ├── flags: ∅
128141
│ │ ├── opening_loc: ∅
129142
│ │ ├── value_loc: (17,10)-(17,13) = "<=>"
130143
│ │ ├── closing_loc: ∅
@@ -133,12 +146,14 @@
133146
├── @ AliasMethodNode (location: (19,0)-(19,15))
134147
│ ├── new_name:
135148
│ │ @ SymbolNode (location: (19,6)-(19,9))
149+
│ │ ├── flags: ∅
136150
│ │ ├── opening_loc: (19,6)-(19,7) = ":"
137151
│ │ ├── value_loc: (19,7)-(19,9) = "=="
138152
│ │ ├── closing_loc: ∅
139153
│ │ └── unescaped: "=="
140154
│ ├── old_name:
141155
│ │ @ SymbolNode (location: (19,10)-(19,15))
156+
│ │ ├── flags: ∅
142157
│ │ ├── opening_loc: (19,10)-(19,11) = ":"
143158
│ │ ├── value_loc: (19,11)-(19,15) = "eql?"
144159
│ │ ├── closing_loc: ∅
@@ -147,12 +162,14 @@
147162
├── @ AliasMethodNode (location: (21,0)-(21,9))
148163
│ ├── new_name:
149164
│ │ @ SymbolNode (location: (21,6)-(21,7))
165+
│ │ ├── flags: ∅
150166
│ │ ├── opening_loc: ∅
151167
│ │ ├── value_loc: (21,6)-(21,7) = "A"
152168
│ │ ├── closing_loc: ∅
153169
│ │ └── unescaped: "A"
154170
│ ├── old_name:
155171
│ │ @ SymbolNode (location: (21,8)-(21,9))
172+
│ │ ├── flags: ∅
156173
│ │ ├── opening_loc: ∅
157174
│ │ ├── value_loc: (21,8)-(21,9) = "B"
158175
│ │ ├── closing_loc: ∅
@@ -161,12 +178,14 @@
161178
└── @ AliasMethodNode (location: (23,0)-(23,11))
162179
├── new_name:
163180
│ @ SymbolNode (location: (23,6)-(23,8))
181+
│ ├── flags: ∅
164182
│ ├── opening_loc: (23,6)-(23,7) = ":"
165183
│ ├── value_loc: (23,7)-(23,8) = "A"
166184
│ ├── closing_loc: ∅
167185
│ └── unescaped: "A"
168186
├── old_name:
169187
│ @ SymbolNode (location: (23,9)-(23,11))
188+
│ ├── flags: ∅
170189
│ ├── opening_loc: (23,9)-(23,10) = ":"
171190
│ ├── value_loc: (23,10)-(23,11) = "B"
172191
│ ├── closing_loc: ∅

‎test/prism/snapshots/arrays.txt‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@
8383
│ │ └── @ AssocNode (location: (5,1)-(5,12))
8484
│ │ ├── key:
8585
│ │ │ @ SymbolNode (location: (5,1)-(5,3))
86+
│ │ │ ├── flags: ∅
8687
│ │ │ ├── opening_loc: ∅
8788
│ │ │ ├── value_loc: (5,1)-(5,2) = "a"
8889
│ │ │ ├── closing_loc: (5,2)-(5,3) = ":"
@@ -92,11 +93,13 @@
9293
│ │ │ ├── flags: ∅
9394
│ │ │ ├── elements: (length: 2)
9495
│ │ │ │ ├── @ SymbolNode (location: (5,5)-(5,7))
96+
│ │ │ │ │ ├── flags: ∅
9597
│ │ │ │ │ ├── opening_loc: (5,5)-(5,6) = ":"
9698
│ │ │ │ │ ├── value_loc: (5,6)-(5,7) = "b"
9799
│ │ │ │ │ ├── closing_loc: ∅
98100
│ │ │ │ │ └── unescaped: "b"
99101
│ │ │ │ └── @ SymbolNode (location: (5,9)-(5,11))
102+
│ │ │ │ ├── flags: ∅
100103
│ │ │ │ ├── opening_loc: (5,9)-(5,10) = ":"
101104
│ │ │ │ ├── value_loc: (5,10)-(5,11) = "c"
102105
│ │ │ │ ├── closing_loc: ∅
@@ -110,23 +113,27 @@
110113
│ ├── flags: ∅
111114
│ ├── elements: (length: 5)
112115
│ │ ├── @ SymbolNode (location: (9,1)-(9,3))
116+
│ │ │ ├── flags: ∅
113117
│ │ │ ├── opening_loc: (9,1)-(9,2) = ":"
114118
│ │ │ ├── value_loc: (9,2)-(9,3) = "a"
115119
│ │ │ ├── closing_loc: ∅
116120
│ │ │ └── unescaped: "a"
117121
│ │ ├── @ SymbolNode (location: (9,5)-(9,7))
122+
│ │ │ ├── flags: ∅
118123
│ │ │ ├── opening_loc: (9,5)-(9,6) = ":"
119124
│ │ │ ├── value_loc: (9,6)-(9,7) = "b"
120125
│ │ │ ├── closing_loc: ∅
121126
│ │ │ └── unescaped: "b"
122127
│ │ ├── @ SymbolNode (location: (10,0)-(10,2))
128+
│ │ │ ├── flags: ∅
123129
│ │ │ ├── opening_loc: (10,0)-(10,1) = ":"
124130
│ │ │ ├── value_loc: (10,1)-(10,2) = "c"
125131
│ │ │ ├── closing_loc: ∅
126132
│ │ │ └── unescaped: "c"
127133
│ │ ├── @ IntegerNode (location: (10,3)-(10,4))
128134
│ │ │ └── flags: decimal
129135
│ │ └── @ SymbolNode (location: (14,0)-(14,2))
136+
│ │ ├── flags: ∅
130137
│ │ ├── opening_loc: (14,0)-(14,1) = ":"
131138
│ │ ├── value_loc: (14,1)-(14,2) = "d"
132139
│ │ ├── closing_loc: ∅
@@ -137,23 +144,27 @@
137144
│ ├── flags: ∅
138145
│ ├── elements: (length: 5)
139146
│ │ ├── @ SymbolNode (location: (18,1)-(18,3))
147+
│ │ │ ├── flags: ∅
140148
│ │ │ ├── opening_loc: (18,1)-(18,2) = ":"
141149
│ │ │ ├── value_loc: (18,2)-(18,3) = "a"
142150
│ │ │ ├── closing_loc: ∅
143151
│ │ │ └── unescaped: "a"
144152
│ │ ├── @ SymbolNode (location: (18,5)-(18,7))
153+
│ │ │ ├── flags: ∅
145154
│ │ │ ├── opening_loc: (18,5)-(18,6) = ":"
146155
│ │ │ ├── value_loc: (18,6)-(18,7) = "b"
147156
│ │ │ ├── closing_loc: ∅
148157
│ │ │ └── unescaped: "b"
149158
│ │ ├── @ SymbolNode (location: (19,0)-(19,2))
159+
│ │ │ ├── flags: ∅
150160
│ │ │ ├── opening_loc: (19,0)-(19,1) = ":"
151161
│ │ │ ├── value_loc: (19,1)-(19,2) = "c"
152162
│ │ │ ├── closing_loc: ∅
153163
│ │ │ └── unescaped: "c"
154164
│ │ ├── @ IntegerNode (location: (19,3)-(19,4))
155165
│ │ │ └── flags: decimal
156166
│ │ └── @ SymbolNode (location: (23,0)-(23,2))
167+
│ │ ├── flags: ∅
157168
│ │ ├── opening_loc: (23,0)-(23,1) = ":"
158169
│ │ ├── value_loc: (23,1)-(23,2) = "d"
159170
│ │ ├── closing_loc: ∅
@@ -756,16 +767,19 @@
756767
│ ├── flags: ∅
757768
│ ├── elements: (length: 3)
758769
│ │ ├── @ SymbolNode (location: (62,3)-(62,6))
770+
│ │ │ ├── flags: ∅
759771
│ │ │ ├── opening_loc: ∅
760772
│ │ │ ├── value_loc: (62,3)-(62,6) = "one"
761773
│ │ │ ├── closing_loc: ∅
762774
│ │ │ └── unescaped: "one"
763775
│ │ ├── @ SymbolNode (location: (62,7)-(62,10))
776+
│ │ │ ├── flags: ∅
764777
│ │ │ ├── opening_loc: ∅
765778
│ │ │ ├── value_loc: (62,7)-(62,10) = "two"
766779
│ │ │ ├── closing_loc: ∅
767780
│ │ │ └── unescaped: "two"
768781
│ │ └── @ SymbolNode (location: (62,11)-(62,16))
782+
│ │ ├── flags: ∅
769783
│ │ ├── opening_loc: ∅
770784
│ │ ├── value_loc: (62,11)-(62,16) = "three"
771785
│ │ ├── closing_loc: ∅
@@ -805,16 +819,19 @@
805819
│ ├── flags: ∅
806820
│ ├── elements: (length: 3)
807821
│ │ ├── @ SymbolNode (location: (69,3)-(69,6))
822+
│ │ │ ├── flags: ∅
808823
│ │ │ ├── opening_loc: ∅
809824
│ │ │ ├── value_loc: (69,3)-(69,6) = "one"
810825
│ │ │ ├── closing_loc: ∅
811826
│ │ │ └── unescaped: "one"
812827
│ │ ├── @ SymbolNode (location: (69,7)-(69,10))
828+
│ │ │ ├── flags: ∅
813829
│ │ │ ├── opening_loc: ∅
814830
│ │ │ ├── value_loc: (69,7)-(69,10) = "two"
815831
│ │ │ ├── closing_loc: ∅
816832
│ │ │ └── unescaped: "two"
817833
│ │ └── @ SymbolNode (location: (69,11)-(69,16))
834+
│ │ ├── flags: ∅
818835
│ │ ├── opening_loc: ∅
819836
│ │ ├── value_loc: (69,11)-(69,16) = "three"
820837
│ │ ├── closing_loc: ∅
@@ -854,16 +871,19 @@
854871
│ ├── flags: ∅
855872
│ ├── elements: (length: 3)
856873
│ │ ├── @ SymbolNode (location: (76,3)-(76,6))
874+
│ │ │ ├── flags: ∅
857875
│ │ │ ├── opening_loc: ∅
858876
│ │ │ ├── value_loc: (76,3)-(76,6) = "one"
859877
│ │ │ ├── closing_loc: ∅
860878
│ │ │ └── unescaped: "one"
861879
│ │ ├── @ SymbolNode (location: (76,7)-(76,10))
880+
│ │ │ ├── flags: ∅
862881
│ │ │ ├── opening_loc: ∅
863882
│ │ │ ├── value_loc: (76,7)-(76,10) = "two"
864883
│ │ │ ├── closing_loc: ∅
865884
│ │ │ └── unescaped: "two"
866885
│ │ └── @ SymbolNode (location: (76,11)-(76,16))
886+
│ │ ├── flags: ∅
867887
│ │ ├── opening_loc: ∅
868888
│ │ ├── value_loc: (76,11)-(76,16) = "three"
869889
│ │ ├── closing_loc: ∅

0 commit comments

Comments
 (0)