[Feature] Pass multiple json files to replica

[teto]

I've see a behavior I can't explain when using a glob: the shell expands correctly tests/*.json to tests/hello.json and replica sees the correct filename but it considers it as an option ?

(ins)❯ replica run tests/*.json
Unknown option: tests/hello.json

Usage: replica COMMAND [COMMAND_OPTIONS]

Integration testing for command line interfaces

Commands:
  run   Run tests from a Replica JSON file
  info  Display information about test suites
  set   Set a global configuration for replica commands

Run 'replica help COMMAND' for more information on a command.

if i type the filename directly it works ok

(ins)❯ replica run tests/hello.json 
--------------------------------------------------------------------------------
Running tests...
  ❌  hello: [Unexpected content for output]
      Expected:
      hello, world!
      
      Given:
      Hello, world!
      
--------------------------------------------------------------------------------
Summary:
  ❌  (Failure): 1 / 1

Finally I tried by adjusting the working directory

(ins)❯ replica run -w tests hello.json 
Can't access file hello.json

This is with rev e6abb84

[berewt]

I try to replicate it this morning and I didn't encounter the issue. So it may be OS-specific (I'm on macOS).
As soon as I can I'll test it in a docker container to try to understand what is going on.

The unknown option is a generic error message when replica doesn't succeed to parse the args, but I don't get why it fails here.

Finally, -w just change the directory where the tests are run, the path is resolved prior to that (I'll probably have to clarify it in the help message).

Thanks for the feedback!

[berewt]

It seems to be ok on ubuntu-20.04 too (at least if I can trust my docker image, that I will update in a next PR to ease the tests).
So, either it's OS related but the issue is more OS-version specific, or my next clue is a change in Idris that impact the resolution...
Otherwise I'm out of clue at the moment).

[teto]

I am using replica through the nix flake, revision e6abb84. I can reproduce on another computer. One of the advantages of nix is reproducibility so in case you have it installed, you can reproduce using nix develop
https://github.com/teto/quantum/commit/49d2a737f563576d85e834ea33c5431c3a0eb31f
Indeed might be worth adding some precision about -w
Also would be nice to have a replica version.
I believe the issue is that replica accepts only one json test file ? If I try replica run tcp.json hello.json it fails.

[teto]

e.g. what's the best way to run a bunch of .json tests ?

[berewt]

The way I deal with it is by creating a huge json out of smaller dhall files with dhall import facilities. Dealing with several files need a bit of design thinking, but I may start it the easy way by considering that the set of files are parts of one big unique test suite.
I don't have nix on my machine but I will try nix on docker and see how it goes. It would be nice if I could finally replicate it.

replica version wasn't a huge priority so far but I agree it would help for bug reports.

[teto]

I might be able to do that, I have already split my tests into several .dhall files. any public example ? dhall looks simple enough but it can speed up bootstrap to see an example.

NB: flakes is a feature of nixUnstable so less straightforward to test. I think the problems is just because I tried to run several .json files so no need to bother you further :)

[berewt]

Oh... if it works with a single test file in the directory, then it's probably the multi input: the parser consider all the arguments as options, only the last one is parsed as a file.
For Dhall, you can check how replica test suite is built: tests.dhall. Lines 4 and 5 are import and they are concatenated with local tests on line 225.
The tests suite of the project could also gives you an idea on how tests are structured in Dhall.
There is also a documentation page about the tests options in the project, if it can help. It has to be improved but at least it exists.

If you want to stick to json, the best solution I can think of at the moment is a shell loop, not sure if "best" is the right word here.

[berewt]

Yes, I was able to replicate the issue with several json files:

# mkdir tests
# tee tests/hello.json <<EOF
> { "hello": {"command": "echo \"hello, world!\""} }
> EOF
{ "hello": {"command": "echo \"hello, world!\""} }
# tee tests/hell.json <<EOF
> { "hell": {"command": "true"}}
> EOF
{ "hell": {"command": "true"}}
# replica run tests/*.json
Unknown option: tests/hell.json

[teto]

Unrelated but puzzled by the following:

replica help set                                                                                                                                                                                                                              

Usage: replica set [OPTIONS] KEY=VALUE

Set a global configuration for replica commands

Options:
  --global, -g  Set a global config value (in `$HOME/.replica.json`)
  --local, -l   Set a local config value (in `./.replica.json`) (default)

Available keys, and description:
  replicaDir (or replica-dir, rDir) where replica stores internal information (default `./.replica`)
  globalDir  (or global-dir, gDir) where replica stores golden values (default `./.replica/tests`)

I think I understand the set syntax yet the following call fails

$  replica set -l globalDir=tests/golden                                                                                                                                                                                                         
Unknown option: globalDir=tests/golden

Usage: replica COMMAND [COMMAND_OPTIONS]

Integration testing for command line interfaces|

[berewt]

The syntax is correct, I think the integration of replica set wasn't fully performed on your version... I was able to replicate it with yours and it works well with v0.3.

I talked to fast, this one may be os related. It's definitely my fault: I've tested if the result of a set command works, but not the set command itself.
I'll have a look at it during lunch or this evening, I have to understand why the parsing is different depending on the platform

[berewt]

Stupid me: it's goldenDir not globalDir the only issue is the command documentation.

[teto]

While I would like replica run tests/test1.json tests/test2.json I can work around it so I'll let you decide on when to close the ticket as we've mentioned several issues here. Thanks for the help.

[berewt]

let's keep it open and I'll try to include it to the roadmap.