tests: trycmd for integration testing

[gierens]

I read through the discussion regarding VHS for testing and googled around a little and found this: https://crates.io/crates/trycmd

One can define the binary, arguments and environment variables in .toml files and stdin in separate files. A simple .rs file does the setup and integration with cargo test. After defining those one can record the stdout and stderr with TRYCMD=dump cargo test --test cli_tests which dumps them to dump/. Once moved to the chosen test folder, here tests/cmd, one can actually perform tests with normal cargo test, getting something like:

And if a test fails you get quite extensive output:

You can also easily overwrite outdated outputs with TRYCMD=overwrite.

This might have some caveats like that there are no colors, and maybe things like terminal size can't be limited like with VHS, but I feel this is a very strong and solid framework with good integration, automation and very detailed information on test failure. Figures, since it's made by the guy that maintains clap which apparently also at least partly uses this for tests.

This assert-rs community seems to have a couple of interesting crates for testing like https://crates.io/crates/assert_cmd for very fine grained cli tests. They describe it in the docs like assert_cmd is for treating tests like pets and trycmd for treating them like cattle, so easily managing hundreds of tests at once.

Any thoughts, @cafkafk and @PThorpe92

[PThorpe92]

This sounds almost exactly like what @cafkafk was talking about writing... I wonder if this is the answer to testing #197

Seems like a good proof of concept for both. The interface is literally what I would have to heavily modify VHS to do the same thing

EDIT: Approved for concept anyway, obviously i don't assume that is our new pipeline XD

[gierens]

Yeah I read your discussion and thought, wait, we can't be the first that are trying to test cli apps like that ... and I had to search a bit and first found assert_cmd which is a bit older and more popular and then luckily this

Although I did plan to spend the evening on a different project I just had to try this, this looks like exactly what we were looking for. And it's made by a clap guy and they know what they are doing.

Once @cafkafk gives her ok, we can start building a ton of tests, merge this, and then use it to verify the clap PR.

[MartinFillon]

This will be super useful indeed for the parser, I also thought about python based tests but staying in rust is also good.

[cafkafk]

In general, this looks like just what we've been searching for! Very cool!

Figures, since it's made by the guy that maintains clap which apparently also at least partly uses this for tests.

This is also great, it would suck to adopt something too obscure.

Once @cafkafk gives her ok, we can start building a ton of tests, merge this, and then use it to verify the clap PR.

I'm gonna try this out a bit and get a feel for it. Probably an okay from me, but I just wanna see if there are any obvious problems. Will get back to you on that shortly.

[cafkafk]

Two things I think we need to keep in mind:

• We'll likely still need sandboxing
• We might need some platform specific tests, e.g. for Windows (see: https://github.com/eza-community/eza/actions/runs/6150741633/job/16692367062?pr=26)
• Sandboxing on windows will likely be harder, and might simply be out of scope for now.

[cafkafk]

We need some logic to run these tests in a platform dependent way so windows CI isn't broken, but else, I'm very much approving this! It's looks really nice.

Ideally, that's all the changes we do in this PR, I'd prefer this merging as soon as it's not breaking anything so we can get to work on using this. E.g. I'd like to cleanup some of the "itest" stuff I did.

[gierens]

We need some logic to run these tests in a platform dependent way so windows CI isn't broken, but else, I'm very much approving this! It's looks really nice.

Yep, I already thought about that yesterday and I think this should be fairly doable within the integration as it's in Rust and we should be able to leverage #[cfg(target_os=....

Ideally, that's all the changes we do in this PR, I'd prefer this merging as soon as it's not breaking anything so we can get to work on using this. E.g. I'd like to cleanup some of the "itest" stuff I did.

Alright, makes sense I'll cleanup a bit, make the reference folder for testing a little more elaborate with more than just plain and empty files, and try to get the windows stuff either running or skipped.

[cafkafk]

Alright, makes sense I'll cleanup a bit, make the reference folder for testing a little more elaborate with more than just plain and empty files, and try to get the windows stuff either running or skipped.

Perhaps most of this could be done in a follow up PR as well? if we just get the windows stuff skipped or working I can start working on sandboxing today when that is merged.

[gierens]

Alright I'll try to make it quick ... waiting for the CI is probably what takes most of the time anyway

[gierens]

Alright, I revised cli_tests.rs and the files in tests/cmd to split tests into three categories unix, windows, and all for those that generally run on both. This way we don't have any redundancies and only separate tests when there really is a difference between the platforms.

This already ran through the CI also for windows as I also revised it's expected stdout for the long option where it failed before. Just rebased this again on main. Once the CI is done again this can be merged.

[cafkafk]

This already ran through the CI also for windows as I also revised it's expected stdout for the long option where it failed before. Just rebased this again on main. Once the CI is done again this can be merged.

Alright! LGTM 👍

Changed commit summaries from feature -> feat so this conformed to convco.

[gierens]

Changed commit summaries from feature -> feat so this conformed to convco.

Ahhh, I really gotta remember that one haha

[cafkafk]

Changed commit summaries from feature -> feat so this conformed to convco.

Ahhh, I really gotta remember that one haha

I'll look into adding a CI check, I also sometimes make mistakes with this, specially doc instead of docs >_<

[gierens]

Just one thing I have to say about trycmd in the end: "This is the way!" 😆