Stopped working for benchee / only runs one command

[PragTob]

Hello there my old and trusty tool author! And thanks for ex_guard 💚

I been getting back a bit into updating benchee but I noticed ex_guard doesn't work for me any more. I'm on all the most recent versions. Setup etc. over here: https://github.com/bencheeorg/benchee/blob/main/.exguard.exs

essentially this is my .exguard:

use ExGuard.Config

project_files = ~r{\.(erl|ex|exs|eex|xrl|yrl)\z}i
deps = ~r{deps}

guard("compile and warn", run_on_start: true)
|> command("MIX_ENV=test mix compile --warnings-as-errors")
|> watch(project_files)
|> ignore(deps)
|> notification(:auto)

guard("credo", run_on_start: true)
|> command("mix credo")
|> watch(project_files)
|> ignore(deps)
|> notification(:auto)

guard("mix format", run_on_start: true)
|> command("mix format --check-formatted")
|> watch(project_files)
|> ignore(deps)
|> notification(:auto)

guard("dialyzer", run_on_start: true)
|> command("mix dialyzer --halt-exit-status")
|> watch(project_files)
|> ignore(deps)
|> notification(:auto)

guard("test", run_on_start: true)
|> command("mix test --color")
|> watch(project_files)
|> ignore(deps)
|> notification(:auto)

it always ever only runs the tests, and not all the other tasks that I set it up for:

tobi@qiqi:~/github/benchee(minor-fixes)$ mix guard
ex_guard is executing mix test --color
Excluding tags: [needs_fast_function_repetition: true]

.....................................................................................................................................................................................................................................................................................................................................................................................................Failed to write log message to stdout, trying stderr

12:32:55.790 [info] Failed to write to standard out (:epipe)
......................*.............
Finished in 7.7 seconds (6.1s async, 1.5s sync)
128 doctests, 301 tests, 0 failures, 4 excluded, 1 skipped

Randomized with seed 908964
ex_guard is executing mix test --color
Compiling 7 files (.ex)
Compiling 6 files (.ex)
Excluding tags: [needs_fast_function_repetition: true]

...................................................................................................................................................................................................................................................................................................................................................................................Failed to write log message to stdout, trying stderr

12:34:40.114 [info] Failed to write to standard out (:epipe)
.......................*..............................
Finished in 8.0 seconds (6.4s async, 1.6s sync)
128 doctests, 301 tests, 0 failures, 4 excluded, 1 skipped

Randomized with seed 114835

Cheers,
Tobi

[slashmili]

You have a nice setup! I might steal it for myself 😆

well it works for me 🙈

mix guard
ex_guard is executing mix test --color
warning: function my_function/0 is unused
  lib/foo_bar.ex:19

.

  1) doctest FooBar.hello/0 (1) (FooBarTest)
     test/foo_bar_test.exs:3
     Doctest failed
     doctest:
       iex> FooBar.hello()
       :space
     code:  FooBar.hello() === :space
     left:  :world
     right: :space
     stacktrace:
       lib/foo_bar.ex:11: FooBar (module)


Finished in 0.02 seconds (0.00s async, 0.02s sync)
1 doctest, 1 test, 1 failure

Randomized with seed 635921
ex_guard is executing mix dialyzer --halt-exit-status
warning: function my_function/0 is unused
  lib/foo_bar.ex:19

Finding suitable PLTs
Checking PLT...
[:bunt, :compiler, :credo, :crypto, :eex, :elixir, :ex_guard, :ex_unit, :file_system, :foo_bar, :fs, :inets, :jason, :kernel, :logger, :stdlib]
PLT is up to date!
No :ignore_warnings opt specified in mix.exs and default does not exist.

halt_exit_status is no longer a valid CLI argument.
Starting Dialyzer
[
  check_plt: false,
  init_plt: ~c"/private/tmp/foo_bar/_build/dev/dialyxir_erlang-26.0.2_elixir-1.15.4_deps-dev.plt",
  files: [~c"/private/tmp/foo_bar/_build/dev/lib/foo_bar/ebin/Elixir.FooBar.beam"],
  warnings: [:unknown]
]
Total errors: 0, Skipped: 0, Unnecessary Skips: 0
done in 0m2.18s
done (passed successfully)
ex_guard is executing mix format --check-formatted
ex_guard is executing mix credo
Checking 3 source files ...

Please report incorrect results: https://github.com/rrrene/credo/issues

Analysis took 0.1 seconds (0.06s to load, 0.08s running 55 checks on 3 files)
4 mods/funs, found no issues.

Showing priority issues: ↑ ↗ →  (use `mix credo explain` to explain issues, `mix credo --help` for options).
ex_guard is executing MIX_ENV=test mix compile --warnings-as-errors
warning: function my_function/0 is unused
  lib/foo_bar.ex:19

Compilation failed due to warnings while using the --warnings-as-errors option

I'm testing on Mac and this is Elixir/Erlang versions

 elixir --version
Erlang/OTP 26 [erts-14.0.2] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit:ns]

Elixir 1.15.4 (compiled with Erlang/OTP 26)

[PragTob]

Let me run it on the new framework and let me downgrade elixir... might be some Linux stuff maybe?

Does it work for you when running my setup from the benchee repo?

[slashmili]

I didn't try it on benchee but a simple app. I'll try it out.

I'll also try to run it on Linux.

[PragTob]

so... same on the new framework, however that made me remember this warning:

17:22:05.894 [error] backend port not found: :inotifywait

which may or may not be part of it, I'll see to fix that 😁

[PragTob]

after installing inotify-tools and recompiling exguard (just removed all deps, fetched again and recompiled) still same behavior sadly :(

[slashmili]

I ran two tests

1. Ran it on benchee and I also saw the same behaviour with error:

elixir --version
Erlang/OTP 26 [erts-14.1.1] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit:ns]

Elixir 1.16.0-rc.0 (792d4cc) (compiled with Erlang/OTP 26)
mix guard
.....
t, trying stderr

17:31:47.234 [error]   crasher:
    initial call: 'Elixir.IEx.Evaluator':init/5
    pid: <0.186.0>
    registered_name: []
    exception error: terminated
      in function  io:put_chars/2
         called as io:put_chars(standard_io,
                                [[<<"** (ArgumentError) errors were found at the given arguments:\n\n  * 1st argument: broken pipe (epipe)\n">>],
                                 10,
                                 <<"    (stdlib 5.1.1) io.erl:103: :io.put_chars(:standard_io, [\"Operating System: macOS\\nCPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz\\nNumber of Available Cores: 16\\nAvailable memory: 32 GB\\nElixir 1.16.0-rc.0\\nErlang 26.1.2\\nJIT enabled: true\\n\", 10])\n    (benchee 1.2.0) lib/benchee/output/benchmark_printer.ex:40: Benchee.Output.BenchmarkPrinter.configuration_information/1\n    (benchee 1.2.0) lib/benchee/benchmark.ex:120: Benchee.Benchmark.collect/3\n    (benchee 1.2.0) lib/benchee.ex:48: Benchee.run/2\n    iex:1: (file)\n">>])
         *** argument 1: the device has terminated
      in call from 'Elixir.IEx.Evaluator':parse_eval_inspect/4 (lib/iex/evaluator.ex, line 297)
      in call from 'Elixir.IEx.Evaluator':loop/1 (lib/iex/evaluator.ex, line 187)
      in call from 'Elixir.IEx.Evaluator':init/5 (lib/iex/evaluator.ex, line 32)
    ancestors: [<0.178.0>]
    message_queue_len: 1
    messages: [{done,<0.178.0>,false}]
    links: []
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 10958
    stack_size: 28
    reductions: 276167
  neighbours: []

---

2. Ran the command on my sample apps using the same Elixir/Erlang

elixir --version
Erlang/OTP 26 [erts-14.1.1] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit:ns]

Elixir 1.16.0-rc.0 (792d4cc) (compiled with Erlang/OTP 26)
mix guard
ex_guard is executing mix test --color
    warning: function my_function/0 is unused
    │
 19 │   defp my_function do
    │        ~
    │
    └─ lib/foo_bar.ex:19:8



  1) doctest FooBar.hello/0 (1) (FooBarTest)
     test/foo_bar_test.exs:3
     Doctest failed
     doctest:
       iex> FooBar.hello()
       :space
     code:  FooBar.hello() === :space
     left:  :world
     right: :space
     stacktrace:
       lib/foo_bar.ex:11: FooBar (module)

.
Finished in 0.02 seconds (0.00s async, 0.02s sync)
1 doctest, 1 test, 1 failure

Randomized with seed 738023
ex_guard is executing mix dialyzer --halt-exit-status
    warning: function my_function/0 is unused
    │
 19 │   defp my_function do
    │        ~
    │
    └─ lib/foo_bar.ex:19:8

Finding suitable PLTs
Checking PLT...
[:bunt, :compiler, :credo, :crypto, :eex, :elixir, :ex_guard, :ex_unit, :file_system, :foo_bar, :fs, :inets, :jason, :kernel, :logger, :stdlib]
Looking up modules in dialyxir_erlang-26.1.2_elixir-1.16.0-rc.0_deps-dev.plt
Finding applications for dialyxir_erlang-26.1.2_elixir-1.16.0-rc.0_deps-dev.plt
Finding modules for dialyx
....

And it works even when the files are changed.

[PragTob]

@slashmili the error is some weird random failure in benchee's test execution... I don't think it is related although it may if some console interactivity breaks. 😅 But I'd still expect exguard to continue invoking the next steps? 🤔

[PragTob]

(sadly benchee's test are a level of flakey as it involves a lot of integration style testing up to building an escript etc. the tests pass consistently these days but there's a lot of retrying going on)

Maybe I should try the same exguard file on a simpler project though 🤔

[PragTob]

Huh. Seems it is related to benchees test... when I comment that out from the file it works, running just a simple test file also works... but.. why? The tests are passing 👀

exit status is also definitely 0 so unless that completely borks my terminal I don't see why exguard should stop working 😅

[PragTob]

Seems like Mix.Shell.IO.cmd never returns:

  def execute({guard_config, files}) do
    umbrella_app? = Keyword.get(guard_config.options, :umbrella_app, false)

    files =
      if umbrella_app? do
        Enum.map(files, fn path ->
          String.replace(path, ~r{^apps/[a-zA-z_]+/}, "")
        end)
      else
        files
      end

    arg = Enum.join(files, " ")
    cmd = String.trim("#{guard_config.cmd} #{arg}")
    IO.puts("ex_guard is executing #{cmd}")

    IO.puts("Tobi was here")

    case Mix.Shell.IO.cmd(cmd) |> IO.inspect(label: "Mix Shell return") do
      0 -> {:ok, 0, "", guard_config}
      status -> {:error, status, "", guard_config}
    end |> IO.inspect(label: "Finished execution of: #{cmd}")
  end

produces:

tobi@qiqi:~/github/benchee(main)$ mix guard
ex_guard is executing mix test --color;echo $?;
Tobi was here
Excluding tags: [needs_fast_function_repetition: true]

......................................................................................................................................................................................................................................................................................................................................................................................................Failed to write log message to stdout, trying stderr

20:45:32.725 [error]   crasher:
    initial call: 'Elixir.IEx.Evaluator':init/5
    pid: <0.183.0>
    registered_name: []
    exception error: terminated
      in function  io:put_chars/2
         called as io:put_chars(standard_io,
                                [[<<"** (ArgumentError) errors were found at the given arguments:\n\n  * 1st argument: broken pipe (epipe)\n">>],
                                 10,
                                 <<"    (stdlib 5.1.1) io.erl:103: :io.put_chars(:standard_io, [\"Operating System: Linux\\nCPU Information: AMD Ryzen 9 5900X 12-Core Processor\\nNumber of Available Cores: 24\\nAvailable memory: 31.25 GB\\nElixir 1.16.0-rc.0\\nErlang 26.1.2\\nJIT enabled: true\\n\", 10])\n    (benchee 1.2.0) lib/benchee/output/benchmark_printer.ex:40: Benchee.Output.BenchmarkPrinter.configuration_information/1\n    (benchee 1.2.0) lib/benchee/benchmark.ex:120: Benchee.Benchmark.collect/3\n    (benchee 1.2.0) lib/benchee.ex:48: Benchee.run/2\n    iex:1: (file)\n">>])
         *** argument 1: the device has terminated
      in call from 'Elixir.IEx.Evaluator':parse_eval_inspect/4 (lib/iex/evaluator.ex, line 297)
      in call from 'Elixir.IEx.Evaluator':loop/1 (lib/iex/evaluator.ex, line 187)
      in call from 'Elixir.IEx.Evaluator':init/5 (lib/iex/evaluator.ex, line 32)
    ancestors: [<0.175.0>]
    message_queue_len: 1
    messages: [{done,<0.175.0>,false}]
    links: []
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 10958
    stack_size: 28
    reductions: 271278
  neighbours: []
......*..............................
Finished in 8.1 seconds (6.4s async, 1.6s sync)
128 doctests, 303 tests, 0 failures, 4 excluded, 1 skipped

Randomized with seed 146797
0

[PragTob]

Verified it's definitel Mix.Shell that hangs and well that's not your responsibility, I identified a test that's likely responsible on my end. Thanks for ex_guard!

[slashmili]

Thanks a lot for debugging the issue.

That's still odd... I'll play around with that, maybe we are sitting on bug and can be reported to elixir project.

[PragTob]

@slashmili it may be but honestly once I figured out the test (by commenting out/selective running) it seems very much like a "me" problem 😂 (open a port to an iex shell, never close the port or shut down iex, just end the test) and it produces the error consistently. Like, ideally it'd still work but I may just be fing the shell somehow 😅

[PragTob]

(at first I was sure it was related to all the io capture I do in parallel as a new elixir version at some point was buggy aroudn that, doesn't seem to be the case though)

[PragTob]

Trying to find some support in elixirforum 😅

https://elixirforum.com/t/starting-shutting-down-iex-with-a-port-gracefully/60388