Porting the compilation pipeline #419

wodeni · 2020-12-03T20:39:25Z

Description

⚠️ DNR: Do Not Rebase ⚠️

Related issues: #336 #428

We will port the Domain, Substance Style parsers/checkers and compiler into TypeScript.

Implementation strategy and design decisions

AST types are converted to TS. See penrose-web/src/compiler/Style.ts for a possible place to put the Style compiler code. The top-level function should be compileStyle, and its signature looks like something like this:

export const compileStyle = (
  env: VarEnv,
  subAST: SubProg,
  styAST: StyProg
) /* :State */ => {
  // TODO: fill in the style compiler here
};

Examples with steps to reproduce them

TODO

Checklist

TODO: update checklist for TS

I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass locally using npm test
I ran npm run docs and there were no errors when generating the HTML site
My code follows the style guidelines of this project (e.g.: no ESLint warnings)

Open questions

Questions that require more discussion or to be addressed in future development:

[sty] add Style AST types and create compiler module [sty] simplify substance types and export them as TS types fix: bypass ts checking by temporarily disabling return type

wodeni · 2020-12-03T21:11:01Z

Summarizing the plan for this branch:

@hypotext and I will separately branch off from style-compiler, working on the compiler and the parser. We may clean up our own feature branches before merging them onto this branch when we get working versions of both components. From then on, we collaboratively work on style-compiler together and will not alter the history of this branch again. The final merge onto master will be a merge commit instead of squash & merge.

* [sty-parse] nearley setup * [sty-parse] selector grammar; sepBy combinator * [sty-parser] complete selector grammar * [sty-parser] parse multiple blocks & whitespace/range cleanup * [sty-parser] start jest test suite * [sty-parser] compile grammar before running test; more test cases * [sty-arse] resolve all ambiguities on selector grammar * [sty-parse] stmt sequencing grammar; better testing for comments * [sty-parse] start working on expression grammar * [sty-parse] parse path assign fix ws issue at the beginning of programs bump ts to 3.9.5 for fromEntries * [sty-parse] layering expression * [sty-parse] computations * [sty-parse] gpi decl, constr/obj fns * [sty-parse] floating point number * [sty-parse] tag but not reject style types replace all raw string types with identifier in ASTNode rework some node props to be descriptive add test for keywords * [sty-parse] first working arith expr parser * [sty-parse] list-like exprs; unary minus * [sty-parse] access path and test cases * [sty-parse] multiline comments; remaining stmt types * [sty-parse] systematically parse all example programs in the wiki * [sty-parser] integrate the parser with build pipeline add test to main test module * [sty-parse] remove obsolete optimizer and evaluator tests * fix: increase resource size to avoid bundling errors

wodeni · 2020-12-16T23:28:31Z

Just merged the new parser. I expect some experimentation when porting the Style compiler early on. I'd suggest staying on your branch until it's stable (e.g. compiles 1+ programs in our wiki) @hypotext

kai-qu · 2020-12-16T23:33:48Z

Ok, thanks, will do! @wodeni

wodeni · 2021-01-06T16:39:05Z

Addressed both #436 and #437 on this branch. Feel free to pull them from your branch @hypotext

wodeni · 2021-01-12T22:49:21Z

Ported the domain parser (#441). I'll merge this branch soon and start porting the semantic checker. Once I get there, we'll have a (potentially revised) VarEnv type, and @hypotext can review it and see if the Style compiler can work with it.

* [domain] start empty domain parser; abstract out parser util functions; improve build script * [domain] domain AST * [domain] parse type and predicate decls * [domain] parse functions * [domain] simplify domain grammar by removing dependent types; add constr and prelude * [domain] parse notations; add epsilon rules for arg and type param lists * [domain] only allow typevars as type params; clean up prop type * [domain] scaffloding for domain checker; working example of Either pattern and error types * [domain] new foldM function; start checking types; fix Var type * [domain] check types and constructors; util all combinator; fix output type in AST * [domain] first pass of domain checker; subtypes/notations not handled; lightly tested * [domain] separate out error module; reexport either monad * [domain] add subtype graph building and cycle detection * [domain] add isSubType function using dijkstra \(!\)

* [sub-parse] substance parser setup; move common tokens to ParserUtil * [sub-parse] parse decls and label stmts * [sub-parse] parse decls and label stmts * [sub-parse] resolve cons, func, and pred ambiguity via a generic type; parse preds * [sub-parse] parse equiv. exprs and preds * [sub-parse] parse existing examples; fix newline at EOF * [domain] fix newline at EOF * [sub-check] setup Substance checker * [domain] add real domain programs to the test suite * [sub-parse] only parse vars or preds in pred arg * [sub-check] check decls; generalize misc error util functions * [sub-check] check type list via unification; lightly tested with known problems in checkTypeconstructor * chore: add command for reporting test coverage * [sub-check] add more sophisticated type eq check; fix the logic of substitution error checking * [sub-check] rename subtype checker; return bottom for unbounded vars in expressions; add subtype tests * [sub-check] check functions * [sub-check] check deconstructors * [sub-check] check predicates * [sub-check] check labeling stmts and type eq stmts * [parser] add node information such as children and node type to all parsed nodes; include tree integrity tests in all parser tests * [sub-check] add label postprocessing; mark errors with error types after typechecking * [sub-check] add real programs to Substance checker test suite

wodeni · 2021-01-21T04:16:02Z

Substance parser and typechecker are both ported and tested with 102 test cases (#447) 🎉 . Next step: finish Style compiler and the port should be done!

* [cli-style] add cli * [cli-style] parse sub/dsl * [cli-style] add roger install script * [cli-style] remove old cli * [cli-style] gives roger more personality (readme, emojis) Co-authored-by: wodeni <wn2155@columbia.edu>

* [scp] add style program JSON conversion in backend, and add example JSON file (for orthogonal vectors) in frontend tests. also add orthogonal vectors to registry * [scp] add input types + implementation stub for style compiler * [scp] fix indentation * [scp] fix indentation, add some work on selector checking * [scp] finish selector environment-building code (with selector in env for debugging); all error-checking omitted * [scp] implement some of the substitution finding code * [parser] fix off-by-one-error in selector parsing * [scp] get (ported part of) style compiler compatible with ASTs, no crash but haven't checked correctness of SelEnvs yet * [scm] confirm + improve SelEnv schema * [scm] add some misc sel checking functions * [scp] add test compiler data for linear-algebra-simple.sty, as well as tests for substitution-finding helper functions * [scp] add some function stubs for style matching * [scp] add rel-substitution code (unrun/untested) and documentation of compiler so far * [scp] add rel pretty-printing code and informally verify that all substitution code works (with a minor parser/AST incompatibility in RelPred name). TODO write tests for substitution code * [scp] update compiler and parsed ASTs to deal with minor changes in parser/grammar fixing #436 #437 * [scp] test substitutions (and regenerate style test data WRT parser change) * [scp] add code for testing that a substituted relational line matches a substance program line + code for converting style exprs/predicates to substance ones. add (passing) test that the compiler finds right substitutions on the LA style program * [scp] add code for replacing anon statements with local vars, as well as (passing) test * [scp] add some boilerplate for writing style program translation code * [scp] add some boilerplate for style translation and substitution-applying code * [scp] finish substitution-applying code (not fully tested, but runs); add hacks for issues #443 #444 * [scp] add some translation code (currently not working on GPIs and w/ a hack for local vars); update for #443 #444 * [scp] integrate LocalVar with Path grammar (since parser parses into that form, not IInternalLocalVar) and remove hacks related to it * [scp] add GPI insertion; compiler generates first translation before genOptProblem (untested) * [scp] add boilerplate genOptProblem + initial state * [scp] add some genOptProblem code for finding various kinds of paths in the translation * [scp] add most of the rest of genOptProblem code for generating translation (except initShape and initProperty) * [scp] fix circular import of canvasSize * [scp] fix bug in finding varying val in vector * [scp] add shape initialization code (untested) * [scp] fix varying property finding bug; add `insertNames` * [scp] fix `findExpr` bug on vectors; seems like there might be a VarAD (vector) conversion error? runs and generates translation, and crashes on (old) evaluator. [untested] * [scp] convert `penrose-web` to use new AST types throughout; remove `ts-nocheck`; system runs (and crashes at runtime) * [scp] implement `insertLabels` and AccessPath lookup; still crashes on runtime * [scp] get first figure on screen (though with NaNs, number/object errors, etc). fix boolean literal in style program, label Substance objects by default, skip shape layering for now, and add hack for accesspath index parsing (#448) * [scp] fix compiler bug with initializing shape properties; fix Canvas bug with making translation differentiable again (only needed for state w/ old compiler and decodeJSON, now removed); currently crashes on labels * [scp] fix bugs with pending properties and vector sampling; don't use sorted shapes w/ layering; fix bug with updated shape dimensions in translation but not in shapes (not sure why this broke?); initial output looks reasonable, but optimization and resampling are not tested * [scp] render strokeOpacity in Square shape * [scp] comment out old console logs; replace Optimizer logs with console.log for now; fix opt init param + fn name bugs * [scp] first working compiler with optimization and resample (initialize opt weight in compiler; fix shape find bug) * [scp] gather partial layerings in compiler; improve path printing * [scp] load dsl/sub/sty programs using the new toolchain; compiler still needs to be tested * [scp] implement substance equality check and address #448, now LA image appears onscreen (with funky labels) * [scp] implement some selector checking using domain/substance checkers [untested] * [scp] fix bug with Env (`clone` doesn't work; using Map's immutability is better) * [scp] implement more selector checking using Sub/Domain function (for decls) and valcons subtype checking * [scp] implement path aliasing for GPI properties in deleteProperty and insertExpr (for properties only) * [scp] add more test programs for whole thing; works with the set tree example; set circles mostly works except for a selector match bug possibly w/ nested predicates (?) * [scp] hook up new style compiler to CLI and program toolchain, and remove `processData`! working example: `roger watch linear-algebra-domain/twoVectorsPerp_unsugared.sub linear-algebra-domain/linear-algebra-paper-simple.sty linear-algebra-domain/linear-algebra.dsl` * [scp] preprocess Substance program to disambiguate `Func` #453; now the big set circle example works [the functional version of substance preprocessing is just for testing, not final, needs to mutate ast] * [scp] fix Sub prog Func disambiguation to be imperative #453; should now be more general * [scp] get all old examples working (adding unsugared Substance programs, temp string hacks #458, and Style SEFuncOrValCons disambiguation based on the Substance ones). found some label bugs #459 #460 * [scp] revert hacks for #458 * [scp] some cleanup of logs + putting comments back in * [style-compiler-port] hotfix to allow reconnecting to the socket via button * [scp] return errors in `checkSels` and `insertExpr` (if in `compiling` mode) instead of throwing them; clean up some other internal errors * [scp] add some ad-hoc tests for selector errors and translation errors * [scp] fix JS heap issue after 2501f73 #450 * [scp] implement layering sort function * [scp] port old Style compiler tests to work on loaded LA programs Co-authored-by: katherineye <kye@princeton.edu> Co-authored-by: maxkrieger <themaxaxis@gmail.com> Co-authored-by: wodeni <wn2155@columbia.edu>

* [scp] add style program JSON conversion in backend, and add example JSON file (for orthogonal vectors) in frontend tests. also add orthogonal vectors to registry * [scp] add input types + implementation stub for style compiler * [scp] fix indentation * [scp] fix indentation, add some work on selector checking * [scp] finish selector environment-building code (with selector in env for debugging); all error-checking omitted * [scp] implement some of the substitution finding code * [parser] fix off-by-one-error in selector parsing * [scp] get (ported part of) style compiler compatible with ASTs, no crash but haven't checked correctness of SelEnvs yet * [scm] confirm + improve SelEnv schema * [scm] add some misc sel checking functions * [scp] add test compiler data for linear-algebra-simple.sty, as well as tests for substitution-finding helper functions * [scp] add some function stubs for style matching * [scp] add rel-substitution code (unrun/untested) and documentation of compiler so far * [scp] add rel pretty-printing code and informally verify that all substitution code works (with a minor parser/AST incompatibility in RelPred name). TODO write tests for substitution code * [scp] update compiler and parsed ASTs to deal with minor changes in parser/grammar fixing #436 #437 * [scp] test substitutions (and regenerate style test data WRT parser change) * [scp] add code for testing that a substituted relational line matches a substance program line + code for converting style exprs/predicates to substance ones. add (passing) test that the compiler finds right substitutions on the LA style program * [scp] add code for replacing anon statements with local vars, as well as (passing) test * [scp] add some boilerplate for writing style program translation code * [scp] add some boilerplate for style translation and substitution-applying code * [scp] finish substitution-applying code (not fully tested, but runs); add hacks for issues #443 #444 * [scp] add some translation code (currently not working on GPIs and w/ a hack for local vars); update for #443 #444 * [scp] integrate LocalVar with Path grammar (since parser parses into that form, not IInternalLocalVar) and remove hacks related to it * [scp] add GPI insertion; compiler generates first translation before genOptProblem (untested) * [scp] add boilerplate genOptProblem + initial state * [scp] add some genOptProblem code for finding various kinds of paths in the translation * [scp] add most of the rest of genOptProblem code for generating translation (except initShape and initProperty) * [scp] fix circular import of canvasSize * [scp] fix bug in finding varying val in vector * [scp] add shape initialization code (untested) * [scp] fix varying property finding bug; add `insertNames` * [scp] fix `findExpr` bug on vectors; seems like there might be a VarAD (vector) conversion error? runs and generates translation, and crashes on (old) evaluator. [untested] * [scp] convert `penrose-web` to use new AST types throughout; remove `ts-nocheck`; system runs (and crashes at runtime) * [scp] implement `insertLabels` and AccessPath lookup; still crashes on runtime * [scp] get first figure on screen (though with NaNs, number/object errors, etc). fix boolean literal in style program, label Substance objects by default, skip shape layering for now, and add hack for accesspath index parsing (#448) * [scp] fix compiler bug with initializing shape properties; fix Canvas bug with making translation differentiable again (only needed for state w/ old compiler and decodeJSON, now removed); currently crashes on labels * [scp] fix bugs with pending properties and vector sampling; don't use sorted shapes w/ layering; fix bug with updated shape dimensions in translation but not in shapes (not sure why this broke?); initial output looks reasonable, but optimization and resampling are not tested * [scp] render strokeOpacity in Square shape * [scp] comment out old console logs; replace Optimizer logs with console.log for now; fix opt init param + fn name bugs * [scp] first working compiler with optimization and resample (initialize opt weight in compiler; fix shape find bug) * [scp] gather partial layerings in compiler; improve path printing * [scp] load dsl/sub/sty programs using the new toolchain; compiler still needs to be tested * [lerna] initialize lerna * [lerna] start deprecating all Haskell related code * [lerna] remove haskell precommmit hook * [lerna] add test script for all packages * [lerna] delete top-level haskell configs * [lerna] install husky in monorepo * [lerna] test husky * [lerna] test husky * fix: install script for roger * v1.0.1 * chore: add changelog * [scp] implement substance equality check and address #448, now LA image appears onscreen (with funky labels) * [scp] implement some selector checking using domain/substance checkers [untested] * [scp] fix bug with Env (`clone` doesn't work; using Map's immutability is better) * [lerna] remove haskell lint config and update ci workflow * [lerna] fix build_other_branches workflow * [lerna] updated CI config * [lerna] install lerna in ci * [lerna] sudo lerna install * [lerna] store test results in CI * [lerna] add junit * Updated config.yml * [lerna] full paths for artifacts * [lerna] collect test report in dedicated directory * [scp] implement more selector checking using Sub/Domain function (for decls) and valcons subtype checking * [scp] implement path aliasing for GPI properties in deleteProperty and insertExpr (for properties only) * [scp] add more test programs for whole thing; works with the set tree example; set circles mostly works except for a selector match bug possibly w/ nested predicates (?) * [lerna] add bootstrap command and test with automator * [lerna] add codecov detector * [scp] hook up new style compiler to CLI and program toolchain, and remove `processData`! working example: `roger watch linear-algebra-domain/twoVectorsPerp_unsugared.sub linear-algebra-domain/linear-algebra-paper-simple.sty linear-algebra-domain/linear-algebra.dsl` * [scp] preprocess Substance program to disambiguate `Func` #453; now the big set circle example works [the functional version of substance preprocessing is just for testing, not final, needs to mutate ast] * [scp] fix Sub prog Func disambiguation to be imperative #453; should now be more general * [scp] get all old examples working (adding unsugared Substance programs, temp string hacks #458, and Style SEFuncOrValCons disambiguation based on the Substance ones). found some label bugs #459 #460 * [scp] revert hacks for #458 * [scp] some cleanup of logs + putting comments back in * [style-compiler-port] hotfix to allow reconnecting to the socket via button * [lerna-merge] fix JS heap issue * [lerna-merge] fix CI JSON issue * [scp] return errors in `checkSels` and `insertExpr` (if in `compiling` mode) instead of throwing them; clean up some other internal errors * [scp] add some ad-hoc tests for selector errors and translation errors * [scp] fix JS heap issue after 2501f73 #450 * [scp] implement layering sort function * [scp] port old Style compiler tests to work on loaded LA programs * [lerna] fix style test case path * [lerna] fix import issue in Style Co-authored-by: katherineye <katherine.ye@gmail.com> Co-authored-by: katherineye <kye@princeton.edu> Co-authored-by: maxkrieger <themaxaxis@gmail.com>

codecov-io · 2021-02-04T00:46:35Z

Codecov Report

❗ No coverage uploaded for pull request base (master@f56d55f). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #419   +/-   ##
=========================================
  Coverage          ?   25.68%           
=========================================
  Files             ?       74           
  Lines             ?     5607           
  Branches          ?     1127           
=========================================
  Hits              ?     1440           
  Misses            ?     4167           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f56d55f...2dda7b6. Read the comment docs.

[sty] convert Substance and Style types to TS

f193fbc

[sty] add Style AST types and create compiler module [sty] simplify substance types and export them as TS types fix: bypass ts checking by temporarily disabling return type

wodeni mentioned this pull request Dec 3, 2020

Porting the Style compiler to the web #413

Closed

5 tasks

wodeni assigned wodeni and kai-qu Dec 3, 2020

wodeni mentioned this pull request Dec 11, 2020

Porting Style parser #428

Merged

5 tasks

katherineye and others added 3 commits December 17, 2020 13:15

[parser] fix off-by-one-error in selector parsing

c3b5f83

[style-compiler] fix: #436

e9b10b5

[style-compiler] fix: #437

c07cf18

wodeni added system:port system:style labels Jan 6, 2021

[style-compiler] add SEBind back to SelExpr

f583726

wodeni changed the title ~~Porting the Style compiler~~ Porting the compilation pipeline Jan 12, 2021

wodeni mentioned this pull request Jan 12, 2021

Porting the Domain parser and semantic checker #441

Merged

5 tasks

wodeni added 2 commits January 12, 2021 22:35

fix: #442

7b39f52

fix: #443 #444; add type annotations to all postprocessors

c952679

kai-qu mentioned this pull request Jan 14, 2021

Port Style compiler to typescript #446

Merged

30 tasks

wodeni mentioned this pull request Jan 17, 2021

Porting the Substance parser and semantic checker #447

Merged

5 tasks

wodeni added system:domain system:style system:substance and removed system:style labels Jan 21, 2021

fix: #448

940f118

wodeni mentioned this pull request Jan 23, 2021

Allow floating-point constants in Style to start with decimal #449

Closed

wodeni modified the milestone: Raytracing domain done Jan 24, 2021

chore: add compiler/parser functions to the API

9907599

wodeni force-pushed the style-compiler branch from 5de118a to 9907599 Compare January 27, 2021 22:50

wodeni mentioned this pull request Jan 28, 2021

JavaScript heap error on npm start #450

Closed

Add roger CLI (#451)

e6c81d6

* [cli-style] add cli * [cli-style] parse sub/dsl * [cli-style] add roger install script * [cli-style] remove old cli * [cli-style] gives roger more personality (readme, emojis) Co-authored-by: wodeni <wn2155@columbia.edu>

wodeni mentioned this pull request Feb 1, 2021

Restructure Penrose packages + better tooling 🍱 #455

Merged

17 tasks

wodeni and others added 6 commits February 1, 2021 22:09

fix: add string transform to string_literal token (#458)

9fb2169

fix: add decl_bind back to Substance parser (#457)

c6760e0

Merge branch 'master' into style-compiler

f73539a

fix: add missing contrib functions

2dda7b6

v1.0.0

6bb1c13

wodeni merged commit db210de into master Feb 4, 2021

kai-qu deleted the style-compiler branch February 26, 2021 00:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Porting the compilation pipeline #419

Porting the compilation pipeline #419

wodeni commented Dec 3, 2020 •

edited

wodeni commented Dec 3, 2020

wodeni commented Dec 16, 2020

kai-qu commented Dec 16, 2020

wodeni commented Jan 6, 2021

wodeni commented Jan 12, 2021 •

edited

wodeni commented Jan 21, 2021 •

edited

codecov-io commented Feb 4, 2021

Porting the compilation pipeline #419

Porting the compilation pipeline #419

Conversation

wodeni commented Dec 3, 2020 • edited

Description

Implementation strategy and design decisions

Examples with steps to reproduce them

Checklist

Open questions

wodeni commented Dec 3, 2020

wodeni commented Dec 16, 2020

kai-qu commented Dec 16, 2020

wodeni commented Jan 6, 2021

wodeni commented Jan 12, 2021 • edited

wodeni commented Jan 21, 2021 • edited

codecov-io commented Feb 4, 2021

Codecov Report

wodeni commented Dec 3, 2020 •

edited

wodeni commented Jan 12, 2021 •

edited

wodeni commented Jan 21, 2021 •

edited