`StringLiteral` #4

imteekay · 2023-04-23T18:43:16Z

Add token
Parse String literals
Add tests

Resources

sandersn · 2023-04-24T22:13:09Z

src/lex.ts

@@ -40,6 +40,10 @@ export function lex(s: string): Lexer {
        text in keywords
          ? keywords[text as keyof typeof keywords]
          : Token.Identifier;
+    } else if (['"', "'"].includes(s.charAt(pos))) {
+      scanForward((c) => /[_a-zA-Z0-9'"]/.test(c));


strings should be able to contain any character except the matching " or '. And you'll need to allow for escaping like with \' as well.

sandersn · 2023-04-27T23:36:52Z

src/lex.ts

+      case CharCodes.doubleQuote:
+        return '"';
+      default:
+        return '';


I can't remember what invalid escapes are supposed to do, but I think it's returning char itself, or maybe slash+char.

Exactly, it returns the char itself using the String.fromCharCode() fn (ts sourcecode)
Just made the changes here imteekay/mini-typescript@bc76e30

sandersn · 2023-04-27T23:38:45Z

src/lex.ts

@@ -66,6 +65,62 @@ export function lex(s: string): Lexer {
  function scanForward(pred: (x: string) => boolean) {
    while (pos < s.length && pred(s.charAt(pos))) pos++;
  }
+
+  function scanString() {


feels like this code needs a lot more tests

Just made some adjustments to how I scan, parse, and emit escape characters. And separated the StringLiteral node from the Literal (I will rename the Literal to NumericLiteral and refactor the code in the future) (imteekay/mini-typescript@c5e8220).

Also added more tests as you recommended (imteekay/mini-typescript@ead71e2 and imteekay/mini-typescript@b04fc8a).

baselines/reference/stringLIteral.tree.baseline

baselines/reference/singleTypedVar.errors.baseline

baselines/reference/singleTypedVar.js.baseline

baselines/reference/singleTypedVar.tree.baseline

imteekay · 2023-04-28T17:48:36Z

baselines/reference/stringLIteral.js.baseline

@@ -0,0 +1 @@
+"var singleQuote = 'singleQuote';\nvar doubleQuote = \"doubleQuote\";\nvar escapedSingleQuote = 'escapedSingle\\'Quote';\nvar escapedDoubleQuote = \"escapedDouble\\\"Quote\";\n(missing)"


Should remove the (missing) after merging the EmptyStatement PR.

imteekay · 2023-04-28T17:49:04Z

baselines/reference/stringLIteral.tree.baseline

+    {
+      "kind": "ExpressionStatement",
+      "expr": {
+        "kind": "Identifier",
+        "text": "(missing)"
+      }
+    }


Should replace this node with the EmptyStatement.

the reason that you have empty statements is that ; is parsed as a separator, not a terminator, so a complete program looks like X;Y;Z not X;Y;Z;.

(separators have become less and less popular over the history of programming languages)

imteekay · 2023-04-28T17:50:05Z

src/lex.ts

+      if (pos >= s.length) {
+        // report unterminated string literal error
+      }


Still not sure how to report this kind of error in the lexer scope 🤔

you'll probably have to make the lexer able to report errors the same way the other phases do

I decided to add a new exercise to add the support for the lexer to report errors (imteekay/mini-typescript@65a3270). I also saw an interesting tweet by Maria about reporting errors for string literals.

sandersn

Looks good. As you noted, you just need a way to report errors, and some refactoring of the literal types.

sandersn · 2023-05-01T15:36:07Z

src/types.ts

+export type StringLiteral = Location & {
+  kind: Node.StringLiteral;
+  value: string;
+  isSingleQuote: boolean;


this is a decent choice; you could also have the emitter always produce double quotes or single quotes (and, in theory, make that configurable).

sandersn · 2023-05-01T15:37:10Z

baselines/reference/stringLIteral.tree.baseline

+    {
+      "kind": "ExpressionStatement",
+      "expr": {
+        "kind": "Identifier",
+        "text": "(missing)"
+      }
+    }


the reason that you have empty statements is that ; is parsed as a separator, not a terminator, so a complete program looks like X;Y;Z not X;Y;Z;.

(separators have become less and less popular over the history of programming languages)

sandersn · 2023-05-01T15:46:43Z

src/lex.ts

+      if (pos >= s.length) {
+        // report unterminated string literal error
+      }


you'll probably have to make the lexer able to report errors the same way the other phases do

imteekay · 2023-05-02T01:34:52Z

and some refactoring of the literal types.

@sandersn just decided to add a new exercise to refactor/rename from Literal to NumericLiteral later on

edit: just opened a new PR to handle that imteekay/mini-typescript#6

imteekay added 6 commits April 23, 2023 12:07

Add String literals

b6af384

Merge branch 'main' into string-literals

6d8e517

Scan single quote

f56c6dd

Support double quote

e158033

Add baselines

d2d3efd

Update baselines

53ae450

imteekay self-assigned this Apr 24, 2023

merge with master

0b5d70f

imteekay mentioned this pull request Apr 24, 2023

Clarification on String literals sandersn/mini-typescript#15

Closed

imteekay added 3 commits April 24, 2023 11:19

typecheck string literals

a1df2c7

Merge branch 'main' into string-literals

4e4d434

check "Add string literals" exercise

3d70d3b

imteekay mentioned this pull request Apr 24, 2023

fix when literal is string sandersn/mini-typescript#2

Open

sandersn reviewed Apr 24, 2023

View reviewed changes

imteekay added 2 commits April 24, 2023 19:30

Merge branch 'main' into string-literals

ba4c745

Improve string literal

60ceb41

sandersn reviewed Apr 27, 2023

View reviewed changes

imteekay added 3 commits April 28, 2023 10:08

Return char for invalid escapes

bc76e30

Fix escape sequence

c5e8220

Add more tests

ead71e2

imteekay force-pushed the string-literals branch from 2a76e7e to ead71e2 Compare April 28, 2023 17:29

imteekay added 2 commits April 28, 2023 14:38

Fix some baselines references

b04fc8a

Merge branch 'main' into string-literals

191b18c

imteekay commented Apr 28, 2023

View reviewed changes

baselines/reference/singleTypedVar.errors.baseline Outdated Show resolved Hide resolved

imteekay commented Apr 28, 2023

View reviewed changes

baselines/reference/singleTypedVar.js.baseline Outdated Show resolved Hide resolved

imteekay commented Apr 28, 2023

View reviewed changes

baselines/reference/singleTypedVar.tree.baseline Outdated Show resolved Hide resolved

imteekay commented Apr 28, 2023

View reviewed changes

imteekay requested a review from sandersn April 28, 2023 18:10

sandersn reviewed May 1, 2023

View reviewed changes

imteekay changed the title ~~String literal~~ StringLiteral May 1, 2023

imteekay added 6 commits May 1, 2023 15:05

Merge branch 'main' into string-literals

0958d11

Add exercise to add support for the lexer to report errors

65a3270

Fix all baselines after mergin with master

282af71

Remove build

d8cae0e

Rename files

de468a7

Add exercise to rename Literal to NumericLiteral

459e037

imteekay requested a review from sandersn May 2, 2023 01:48

imteekay merged commit f547ee3 into main May 2, 2023
1 check passed

imteekay deleted the string-literals branch May 2, 2023 21:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`StringLiteral` #4

`StringLiteral` #4

imteekay commented Apr 23, 2023 •

edited

sandersn Apr 24, 2023

sandersn Apr 27, 2023

imteekay Apr 28, 2023

sandersn Apr 27, 2023

imteekay Apr 28, 2023 •

edited

imteekay Apr 28, 2023

imteekay Apr 28, 2023

sandersn May 1, 2023

imteekay Apr 28, 2023

sandersn May 1, 2023

imteekay May 1, 2023

sandersn left a comment

sandersn May 1, 2023

sandersn May 1, 2023

sandersn May 1, 2023

imteekay commented May 2, 2023 •

edited

		@@ -0,0 +1 @@
		"var singleQuote = 'singleQuote';\nvar doubleQuote = \"doubleQuote\";\nvar escapedSingleQuote = 'escapedSingle\\'Quote';\nvar escapedDoubleQuote = \"escapedDouble\\\"Quote\";\n(missing)"

StringLiteral #4

StringLiteral #4

Conversation

imteekay commented Apr 23, 2023 • edited

Resources

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imteekay Apr 28, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandersn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imteekay commented May 2, 2023 • edited

`StringLiteral` #4

`StringLiteral` #4

imteekay commented Apr 23, 2023 •

edited

imteekay Apr 28, 2023 •

edited

imteekay commented May 2, 2023 •

edited