Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overhaul test suite #189

Closed
4 tasks
jgm opened this issue Feb 24, 2022 · 2 comments
Closed
4 tasks

Overhaul test suite #189

jgm opened this issue Feb 24, 2022 · 2 comments

Comments

@jgm
Copy link
Owner

jgm commented Feb 24, 2022

The test suite is a complicated mess, and it's hard to add new tests.

  • Reorganize it for simplicity
  • Consider using tasty-golden
  • Add documentation on adding tests
  • Consider adding a benchmark suite with tasty-bench
@jgm
Copy link
Owner Author

jgm commented Mar 20, 2022

Notes on the current test suite:

Directory structure:

  • tests
    • writers (contains .tex .omml .mml .eqn) (used as golden files for writer tests)
    • readers (used as golden files for reader tests, and as sources for writer tests and round-trip tests)
      • mml (contains .native)
      • tex (contains .native)
      • omml (contains .native)
    • src (contains .mml .tex) (used as input for reader tests)

I think one of the things that makes the test suite hard to modify is that the same files are used as golden files for reader tests and as sources for writer tests... (tests/readers)

Standard tests:

  • runReader on all readers (tex, mml, omml)

    • compares result of converting input src/%.EXT to the result in readers/EXT/%.native where EXT depends on reader
    • if --regenerate-tests, then regenerate this output file
  • runWriter on all writers (tex, mml, omml, eqn) - currently pandoc writer is not tested

    • for sources, uses both readers/mml/%.native and readers/tex/%.native
    • converts these and compares to results in writers/%.EXT where EXT depends on the writer
    • if --regenerate-tests, then regenerate the golden file

Optional round-trip (with --round-trip), for tex, mml, omml:

  • convert readers/EXT/%.native using the specified writer, then use the reader to convert back to native, and compare to the original

@jgm
Copy link
Owner Author

jgm commented Mar 20, 2022

fmt=omml ; mkdir -p test/reader/$fmt;  for x in tests/src/*.$fmt; do out=test/reader/$fmt/${$(basename $x)%.$fmt} ; echo "Creating $out"; echo "<<< $fmt" >$out ; cat $x >>$out; echo >>$out ;  echo ">>> native" >>$out ; ppsh < tests/readers/$fmt/${$(basename $x)%.$fmt}.native >>$out ; done
fmt=tex ; mkdir -p test/writer/$fmt;  for x in tests/readers/mml/*.native tests/readers/tex/*.native ; do out=test/writer/$fmt/${$(basename $x)%.$fmt} ; echo "Creating $out"; echo "<<< native" >$out ; ppsh < $x >>$out;  echo ">>> $fmt" >>$out ; cat tests/writers/${$(basename x)%.native}.$fmt >>$out ; done

@jgm jgm closed this as completed in 18ed6c4 Mar 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant