Add `parse` function, emit AST #274

natemoo-re · 2022-01-31T15:38:00Z

Changes

Adds a function named parse which can emit an AST
Adds @astrojs/compiler/utils with a walk function and is type guards.
Adds fully typed AST Nodes
Example

import { parse } from '@astrojs/compiler';
import { walk, is } from '@astrojs/compiler/utils';

const result = await parse(source, {
  position: false, // defaults to `true`
});

walk(result.ast, (node) => {
  // `tag` nodes are `element` | `custom-element` | `component`
  if (is.tag(node)) {
    console.log(node.name);
  }
})

Testing

Minimal Go test scaffolded out
Simple JS test added

Docs

README has been updated. Docs should probably remain internal for a while until this is stabilized.

changeset-bot · 2022-01-31T15:38:02Z

🦋 Changeset detected

Latest commit: ecdc90e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@astrojs/compiler	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

lib/compiler/README.md

jonathantneal

I can’t possibly fathom it all without a walk thru, but this looks phenomenal.

Co-authored-by: Jonathan Neal <jonathantneal@hotmail.com>

natemoo-re

Leaving some comments for reference

natemoo-re · 2022-01-31T16:12:58Z

cmd/astro-wasm/astro-wasm.go

+			doc.AppendChild(n)
+		}
+
+		result := printer.PrintToJSON(source, doc, parseOptions)


This is very similar to the Transform function other than the way we print. Instead of generating a JS module from our AST, we have a new printer that generates stringified JSON.

natemoo-re · 2022-01-31T16:15:24Z

internal/printer/print-to-json.go

+func (n ASTNode) String() string {
+	str := fmt.Sprintf(`{"type":"%s"`, n.Type)
+	if n.Kind != "" {
+		str += fmt.Sprintf(`,"kind":"%s"`, n.Kind)
+	}
+	if n.Name != "" {
+		str += fmt.Sprintf(`,"name":"%s"`, escapeForJSON(n.Name))
+	}
+	if n.Value != "" || n.Type == "attribute" {
+		str += fmt.Sprintf(`,"value":"%s"`, escapeForJSON(n.Value))
+	}
+	if len(n.Attributes) > 0 {
+		str += `,"attributes":[`
+		for i, attr := range n.Attributes {
+			str += attr.String()
+			if i < len(n.Attributes)-1 {
+				str += ","
+			}
+		}
+		str += `]`
+	}
+	if len(n.Directives) > 0 {
+		str += `,"directives":[`
+		for i, attr := range n.Directives {
+			str += attr.String()
+			if i < len(n.Directives)-1 {
+				str += ","
+			}
+		}
+		str += `]`
+	}
+	if len(n.Children) > 0 {
+		str += `,"children":[`
+		for i, node := range n.Children {
+			str += node.String()
+			if i < len(n.Children)-1 {
+				str += ","
+			}
+		}
+		str += `]`
+	}
+	if n.Position.Start.Line != 0 {
+		str += `,"position":{`
+		str += fmt.Sprintf(`"start":{"line":%d,"column":%d,"offset":%d}`, n.Position.Start.Line, n.Position.Start.Column, n.Position.Start.Offset)
+		if n.Position.End.Line != 0 {
+			str += fmt.Sprintf(`,"end":{"line":%d,"column":%d,"offset":%d}`, n.Position.End.Line, n.Position.End.Column, n.Position.End.Offset)
+		}
+		str += "}"
+	}
+	str += "}"
+	return str
+}


Just want to note that our String() function is implemented manually, but recurses through the AST just fine.

I had originally used json.Marshal but it relies heavily on Reflection which is not well supported in tinygo. This is more performant, anyway.

natemoo-re · 2022-01-31T16:16:38Z

internal/printer/print-to-json.go

+	}
+	var node ASTNode
+
+	node.Position = positionAt(p, n, opts)


Position data is not super robust at the moment and might be wrong in some places. This should work but we'll need a cleanup pass to normalize this data in the parser.

natemoo-re · 2022-01-31T16:20:59Z

lib/compiler/shared/ast.ts

+export interface BaseNode {
+  type: string;
+  position?: Position;
+}
+
+export interface ParentLikeNode extends BaseNode {
+  type: 'element' | 'component' | 'custom-element' | 'expression' | 'root';
+  children: Node[];
+}
+
+export interface LiteralNode extends BaseNode {
+  type: 'text' | 'doctype' | 'comment' | 'frontmatter';
+  value: string;
+}


This is a pretty good approximation of our AST! Intentionally very close to rehype which is built on unist.

natemoo-re · 2022-01-31T16:22:05Z

lib/compiler/shared/ast.ts

+export interface AttributeNode extends BaseNode {
+  type: 'attribute';
+  kind: 'quoted' | 'empty' | 'expression' | 'spread' | 'shorthand' | 'template-literal';
+  name: string;
+  value: string;
+}
+
+export interface DirectiveNode extends Omit<AttributeNode, 'type'> {
+  type: 'directive';
+}


One choice worth calling out—I've split attributes and directives into two separate nodes. This should make it easy to find and validate official directive without needing to loop through all the existing attributes. The compiler filters them out for you.

matthewp

lgtm

* feat: implement Astro.self (#270) * feat: add parser * chore: lint * Update lib/compiler/README.md Co-authored-by: Jonathan Neal <jonathantneal@hotmail.com> Co-authored-by: Jonathan Neal <jonathantneal@hotmail.com>

* chore: enter pre * Treat all documents as fragments which generate no implicit tags (#264) * feat: remove "as", treat all documents as fragments with no implicit tags * test: fix failing tests * lint: remove unused utils * [DO NOT MERGE] add `renderHead` util to output (#265) * feat: add renderHead util to output * chore: update ci to run on `next` * test: update tests * chore: add changeset * fix transform output * Add `parse` function, emit AST (#274) * feat: implement Astro.self (#270) * feat: add parser * chore: lint * Update lib/compiler/README.md Co-authored-by: Jonathan Neal <jonathantneal@hotmail.com> Co-authored-by: Jonathan Neal <jonathantneal@hotmail.com> * chore: add changeset * test: update tests for no implicit tags * test: update tests for no implicit tags * chore: update tests for no implicit tags * chore: update build script * chore: update node bindings * chore: update wasm file * feat: expose AST types * chore: update tests for no implicit tags * [ci] release (next) (#288) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * chore: update exposed types * [ci] release (next) (#304) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Warn when using an expression on hoist `src` attr (#303) * Warn when using an expression on hoist `src` attr * Adds a changeset * Update the warning based on feedback * [ci] release (next) (#305) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix: export `./types` * [ci] release (next) (#306) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix: export all shared types * fix: export named types * [ci] release (next) (#307) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Client only metadata (#308) * Include metadata on client:only components * Adds a changeset * [ci] release (next) (#309) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Fix next `head` behavior (#310) * feat: update WASM tests * fix: prefer astro.Parse to astro.ParseFragment * fix: prefer `Parse` to `ParseFragment` * fix: update parse test * chore: add changeset * test: add parse-ii test * [ci] release (next) (#311) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Add `fragment` to AST (#312) * feat: add fragment node to AST output * fix: edge case with Fragment parsing in <head> * chore: add changeset * chore(lint): fix lint * chore(lint): fix lint * [ci] release (next) (#313) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Improve `renderHead` behavior (#315) * feat: improve render-head behavior * chore: add changeset * chore: fix wasm test * [ci] release (next) (#317) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Refactor WASM tests (#316) * refactor: switch tests to uvu * chore(ci): update test script * chore(lint): fix * test: enable head placeholder test * test: fix head placeholder test * refactor: remove escapeHTML usage, fix slot in head behavior * [ci] release (next) (#320) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix: failing test * feat: expose fragment to AST (#314) Co-authored-by: Jonathan Neal <jonathantneal@hotmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Matthew Phillips <matthew@skypack.dev>

natemoo-re and others added 2 commits January 30, 2022 19:34

feat: implement Astro.self (#270)

7d907c3

feat: add parser

bf3a37f

chore: lint

aba9d91

jonathantneal reviewed Jan 31, 2022

View reviewed changes

lib/compiler/README.md Outdated Show resolved Hide resolved

jonathantneal approved these changes Jan 31, 2022

View reviewed changes

Update lib/compiler/README.md

ecdc90e

Co-authored-by: Jonathan Neal <jonathantneal@hotmail.com>

natemoo-re commented Jan 31, 2022

View reviewed changes

matthewp approved these changes Jan 31, 2022

View reviewed changes

natemoo-re merged commit 0ce2f67 into next Jan 31, 2022

natemoo-re deleted the feat/parse branch January 31, 2022 19:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `parse` function, emit AST #274

Add `parse` function, emit AST #274

Uh oh!

natemoo-re commented Jan 31, 2022 •

edited

Loading

Uh oh!

changeset-bot bot commented Jan 31, 2022 •

edited

Loading

Uh oh!

Uh oh!

jonathantneal left a comment

Uh oh!

natemoo-re left a comment

Uh oh!

natemoo-re Jan 31, 2022

Uh oh!

natemoo-re Jan 31, 2022

Uh oh!

natemoo-re Jan 31, 2022

Uh oh!

natemoo-re Jan 31, 2022

Uh oh!

natemoo-re Jan 31, 2022

Uh oh!

matthewp left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add parse function, emit AST #274

Add parse function, emit AST #274

Uh oh!

Conversation

natemoo-re commented Jan 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Testing

Docs

Uh oh!

changeset-bot bot commented Jan 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Uh oh!

jonathantneal left a comment

Choose a reason for hiding this comment

Uh oh!

natemoo-re left a comment

Choose a reason for hiding this comment

Uh oh!

natemoo-re Jan 31, 2022

Choose a reason for hiding this comment

Uh oh!

natemoo-re Jan 31, 2022

Choose a reason for hiding this comment

Uh oh!

natemoo-re Jan 31, 2022

Choose a reason for hiding this comment

Uh oh!

natemoo-re Jan 31, 2022

Choose a reason for hiding this comment

Uh oh!

natemoo-re Jan 31, 2022

Choose a reason for hiding this comment

Uh oh!

matthewp left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add `parse` function, emit AST #274

Add `parse` function, emit AST #274

natemoo-re commented Jan 31, 2022 •

edited

Loading

changeset-bot bot commented Jan 31, 2022 •

edited

Loading