v1.0.0 - Production release

• Added ability to read from stdin in ictcc and generated binaries by specifying a filename of "-".
• Inherited attributes in an SDTS are now considered unsupported and trying to use them will cause an error. Inherited attribute use can be enabled by passing --exp inherited-attributes to ictcc, but it is unlikely to function properly.
• SDTS hook implementation functions can now return an error instead of just panicking when something goes wrong.
• Disconnected dependency graphs in simulated SDTSs are downgraded to warning, and only become error if there are multiple graphs that lead to the IR.
• Token classes can now be retrieved by class ID by using ByID() in the generated frontend's token package.
• Generated frontends can now detect when they are being fed empty input and will return a different error than just "unexpected end-of-text token".
• Codegen for binaries now clears old code before generating, preventing issues from arising when new generation doesn't completely replace every file.
• Added integration testing framework in tests and example testing framework in examples, both via shell.
• Significantly improved code style; replaced magic literals with proper constants, removed many unused functions, and limited the exported functions to only those necessary, among other things.
• References to loading a parser or other frontend components from 'cache' have been completely removed.
• The environment variable ICTIOBUS_SOURCE may now be used to give an alternate location for local Ictiobus source code when calling ictcc with the --dev flag.
• Fixed lack of support for built-in IR types as args to --ir.
• Fixed error message for impossible attribute reference pre-check giving the name of the attribute instead of the symbol it is on.
• Fixed referring to attribute that doesn't exist at translation time causing a panic. Now it's a normal error.
• Fixed epsilon productions completely breaking parser generation. This fixes the issue of SLR parsers being broken as it was the root cause.
• Fixed some generated token class variable names being generated as just "TC" if relying on rune names.
• Fixed generated SDTSs not checking the error returned by Bind() while initializing.
• Fixed SDTSs being applied in non-deterministic way. Now they will prefer translating the left-most nodes of parse trees, preventing some nasty possible use-before-set errors.
• Fixed (or removed) broken unit tests that were previously commented out.
• Fixed syntax error messages for token classes that start with newline sequences. Before, they were pointing to the apparently blank end of line where the token started, which while technically true, is next to useless for a human attempting to read the error.
• Updated CLI flags for ictcc:
  • Added -S/--suppress and -F/--fatal flags to control suppression and promotion of warnings.
  • Added --debug-sdts to debug the FISHI SDTS.
  • Added -P/--preproc to output a preprocessed FISHI spec before it is sent to the frontend.
  • Added -C/--command to provide FISHI code via CLI args.
  • Added -D/--dfa to output the generated parser's deterministic finite automaton.
  • Added -T/--parse-table to output the generated parser's parsing table.
  • Added --exp to enable experimental features.
  • Removed -p/--parser, --no-cache, and --no-cache-out flags.
• Updated CLI flags for generated binaries:
  • Added -S/--suppress and -F/--fatal flags to control suppression and promotion of warnings.
  • Added -s/--debug-sdts for debugging the SDTS.
  • Added -C/--command to provide input source code via CLI args.
  • Added -P/--preproc to output preprocessed source intput before it is sent to the frontend. This is only included if the generated binary performs a formatting/preprocessing step.
  • Changed -t to --tree and made -t be the shorthand version.
• Improved documentation:
  • Updated README.md to actually have example uses
  • Added new ictcc.md file to serve as the complete ictcc command reference.
  • Added new fishi-usage.md file to give full tutorial on FISHI usage.
  • Significatly improved code docs.
  • Added a version history file RELEASES.md

Self-hosting complete

This release has completed replaced the hand-written bootstrap frontend with one generated by ictiobus its own spec from fishi.md.

Diagnostics binary

• Added the ability to generate a diagnostics binary from a spec. This binary was created by taking the one previously generated solely for input simulation and adding additional features. This binary contains a fully-functional generated frontend and is capable of analyzing source written in the language defined by the spec. It also supports optional preformatting/preprocessing steps. Optionally, this binary can instead be instructed to perform input simulation of the frontend instead of analyzing input files.
• All flags parsing now uses the spf13/pflag library for GNU-style options
• added new flags to ictcc:
  • -d/--diag for generating a diagnostics binary for a language
  • -f/--diag-format-pkg and -c/--diag-format-call for handling preprocessing of analyzed source in the diagnostics binary
  • -q/--quiet to enable quiet mode to suppress progress output and other supplementary output.
  • --preserve-bin-source to keep source code output when it is generated as part of creating a binary.
  • --prefix to set a prefix for all generated source files. Combines well with --preserve-bin-source for debugging.
  • --tmpl-main to allow specification of the template used for the main.go file in generated binaries.
• Updated flags for ictcc:
  • added -v as shortcut for --lang-ver
  • changed --val-sdts-off, --val-sdts-trees, --val-sdts-graphs, --val-sdts-first, and --val-sdts-skip options to --sim-off, --sim-trees, --sim-graphs, --sim-first-err, and --sim-skip-errs respectively to match convention in generated diagnostics/simulation binary.
  • changed --pre-format to --debug-templates to betta demonstr8 its purpose.
• The current frontend for FISHI is now implicitly trusted as valid and will not have input simulation run on it; --sim-X options now apply to the language being analyzed.
• Fixed test of non-deterministic output of grammar.CreateFewestNonTermsAlternationsTable by adding additional possible success conditions for flaky cases.
• Added additional checks of SDTS sections of input specs during spec interpretation (%%actions section). Some of these cases were being caught during simulation, but checking during interpretation improves error reporting and causes failure cases to fail sooner in the process.
  • Specs can no longer define new attributes starting with $.
  • Attribute refs must point to an existing symbol in their action's associated grammar rule.
  • Attribute refs to terminal symbols can only refer to the built-in attributes $text and $ft.
  • Attribute refs to non-terminal symbols cannot refer to $text.
• Improved error reporting during checks of SDTS sections of specs.
• Fixed boostrap lexer error where attribute refs to terminal symbols were lexed as TCTerminal instead of TCAttrRef.

Self-hostable language

This is the first version that can process a fishi.md file and correctly pull out and test the entire parser. In theory, from this point, we can generate the first compiler and move from bootstrap stage to first stage by reading the fishi spec for fishi.

Add additional support for codegen

• Added output codegen with executing (altho currently crashing) SDTS simulator program
• Added new ictcc flags:
  • -tmpl-tokens, -tmpl-lexer, -tmpl-parser, -tmpl-sdts, and -tmpl-frontend to manually specify templates for codegen
  • -no-ambig to disallow ambiguous grammars
  • -ll, -slr, -clr, and -lalr to select specific parser type, if none given defaults to trying LL, SLR, LALR, then CLR in that order.
  • -hooks to specify path to hooks package for use with generated simulator.
  • -hooks-table to specify way to get hooks table for use with generated simulator.
  • -ir to specify explicit type of intermediate representation output of frontend to aid in codegen.
• Update the SDTS error detection to be able to see panics and properly report them.

-pre-format flag

• Adds a -pre-format flag to the CLI, which dumps out generated code prior to running it through gofmt.
• Also adds in-prog codegen using template files. template files are embedded so updating requires rebuilding. Cannot render past parser.go.tmpl at this time.

CLI can now create and output `fishi.Spec` from AST

Full validation on SDTS not yet available, working on it. It still defaults to running it on the bootstrap SDTS itself rather than the language spec.

Updated CLI language accepted in process of complete first fishi spec

v0.4.1

Updated CLI langauge in process of completing initial fishi spec

CLI that's worth using

CLI tool ictcc can now parse FISHI markdown files to ASTs with -a/-ast option and/or parse trees with -t/-tree option. There are also a variety of additional CLI flags that make testing the bootstrap frontend possible.

Right now, a lot of the frontend options for testing the grammar being read and testing the bootstrap frontend itself are interleaved. They may separate in the future, with fishi module able to build them for other frontends but choosing to keep them private for its own frontend.

The CLI needs to be built before it can be used, which can be done in Go 1.19 or later. Just execute ./build.sh on any system with a Go compiler present.

Run with -h to to see the current options, or see the doc comment for the main module in cmd/ictcc/main.go.