Testing formatters

[astrale-sharp]

Looks really nice!

Maybe it could be use to test formatters as well!

Given a formatter and a test file

• Tests that the compilation gives the same result before and after formatting
• Else pinpoints where it broke the code file
  Apply diffs with dichotomy and test the output, it's a bit of a headache to implement because the diff types I used were hard to understand I think

It could happily live next to typstfmt when I get up to it but I think it might be better suited in typst-test

[tingerrr]

Apply diffs with dichotomy and test the output, it's a bit of a headache to implement because the diff types I used were hard to understand I think

Can you elaborate on this part? I'd be happy to make this more formatter friendly if I know what features are needed for it. I know that the currently generated diff images are a little hard to use, but I've already got plans on how to improve this.

Currently, you can already use it in non typst packages using the --root option, so testing the formatter should already be possible! Although, I'm currently working on a different test structure and configuration on where to put them.

[astrale-sharp]

Elaborating :

You'd use something like
typst-test testfmt typstfmt source.typ

let output = compile(source.typ)
let fmt-output = compile(fmt(source.typ))
// either we have a new error or a different compilation result
if fmt-output != output {
    let diffs = get_diffs(source.typ, fmt(source.typ))
    let range = all(source.typ)
    let mut was_reduced = true
    while was_reduced apply_half_the_diff(range);
    print the problematic diffs
}

This naive approach would already give pretty good result and help notice when formatting a document broke something, I tried my hand at that but got too busy a while ago

[tingerrr]

I'm not sure if I want to tie testing so close to formatting. But I see how comparing two different versions of the same document which are both created on demand would also be helpful.

I'm trying to think of a nice way to fit this into the current system without trying it too closely to formatting. Perhaps by allowing the user to create the references on demand, using the custom actions which are run before/after tests.

[astrale-sharp]

I'm trying to think of a nice way to fit this into the current system without trying it too closely to formatting. Perhaps by allowing the user to create the references on demand, using the custom actions which are run before/after tests.

• references to what?
• what kind of actions are we talking about?

[tingerrr]

• references to what?
• what kind of actions are we talking about?

I mean the reference images, creating them on demand instead of assuming they exist, would be some kind of action the suer could specify for a test, and how to do that exactly.

[tingerrr]

@astrale-sharp hey it's been a while, I've implemented ephemeral tests on 7e9ef0f, I would kindly ask you to try it out.

They come with some rough edges, as I don't have any custom user actions to actually do any action on the test file before creating the references for now, so you'd need a task runner which does this for you.

You can do the following to test it out:

1. typst-test --root . init
2. typst-test --root . add your-test --ephemeral
3. edit tests/your-test/ref.typ
4. typstfmt tests/your-test/ref.typ -o tests/your-test/test.typ
5. typst-test --root . run

I've run this locally with Nushell and a Justfile like so:

fmt := './typstfmt'
tt := 'typst-test --root .'

test:
	#! /usr/bin/env nu
	let tests = {{ tt }} list --format=plain | lines | split column ' ' name type
	for test in $tests {
		let ref = ('tests' | path join $test.name | path join 'ref.typ')
		let test = ('tests' | path join $test.name | path join 'test.typ')
		{{ fmt }} $ref -o $test
	}

	{{ tt }} run

Essentially, typst-test list can be used to automate the process of formatting ref.typ to test.typ.

I'm planning to make this whole ordeal easier to use, but for now I'm busy with my master’s thesis, so I can't work on the rewrite all that often :/.

[tingerrr]

I'll close this as completed, typstfmt is currently archived and this will only get simpler with the rewrite and #11 in conjunction with ephemeral tests.