cargo test --nocapture no longer works after docopts landing

[lightsofapollo]

The help command sorta explains this if you think about it (and know) the meaning of -- as you can do this and it will work just fine cargo test -- --nocapture but this is less then obvious. Not sure what a sane way to bridge test runner CLI options would be but it would be nice to have them listed in the cargo help test command so it's more obvious how to pass these flags.

[alexcrichton]

cc @BurntSushi, would you know if this is a bug in docopt or does the specification for cargo test need tweaking?

[BurntSushi]

Hmm. Trying to understand here. Should the behavior of cargo test --nocapture be the same as cargo test -- --nocapture?

[alexcrichton]

Ah yes, sorry for the lack of context! Here's the docopt for cargo test

Execute all unit and integration tests of a local package

Usage:
    cargo-test [options] [--] [<args>...]

Options:
    -h, --help              Print this message
    -j N, --jobs N          The number of jobs to run in parallel
    -u, --update-remotes    Update all remote packages before compiling
    --manifest-path PATH    Path to the manifest to compile
    -v, --verbose           Use verbose output

All of the trailing arguments are passed to the test binaries generated for
filtering tests and generally providing options configuring how they run.

Namely when you execute cargo test --nocapture the intention is to realize that --nocapture isn't a flag to cargo-test but rather to the test binary itself, so I was hoping it would get lumped with with <args> and then cargo-test would pass it through.

So to answer your question, yes, I'm wondering if it's possible to have cargo test --nocapture act the same as cargo test -- --nocapture

[BurntSushi]

So to answer your question, yes, I'm wondering if it's possible to have cargo test --nocapture act the same as cargo test -- --nocapture

Unfortunately, I don't think so. Fundamentally, the only thing that can match an <arg> is a positional argument. In fact, the only reason why -- even works here is that options_first is enabled. This is a bug. See docopt/docopt.rs#8.

Even when this bug is fixed, cargo test --nocapture and cargo test -- --nocapture will produce different parses. I think this is fixed in the design of Docopt. (I don't think it's necessarily a bad thing either. -- is a pretty strong convention.)

[BurntSushi]

One idea to make the usage clearer would be to couple the -- with the <arg>...:

Usage:
    cargo-test [options] [(-- <args>...)]

This would mean that the following would be invalid invocations, but it would be specified more clearly by the usage string (i.e., you can't have an <args> without a preceding --).

cargo test --nocapture
cargo test --

[alexcrichton]

Hm that is a bit unfortunate. cc @wycats, any opinions on this?

[keleshev]

I probably miss a lot of context here. I don't even know Rust, but here you go.

Docopt's option options_first was introduced in order to solve the following problems:

• Strict posix compatibility requires options to precede any positional arguments.
• In order to be able to delegate options to another script.

I assume that the second problem is the one you are trying to solve?

Here's how it is normally solved:

Let's say you want to implement python command line interface. You want to be able to say python script.py --option and delegate the option to the script.py. In this case you pass options_first=True to docopt, with the following usage:

Usage: 
  python [options] <script> [<args>...]

Options:
  -h, --help

In this case all the things after <script> will be interpreted as <args>, even if they start with --. This is because <script> is a positional argument, that means that with options_first=True all arguments after it are interpreted as positional arguments.

The problem is that your usage says cargo-test and not cargo test. In case of cargo test and options_first=True, test would be interpreted as the first positional argument, thus allowing the rest to be interpreted as positional arguments. But, if you write cargo test, you loose the ability to have [options] as in cargo test [options], because all those options will be interpreted as arguments. Instead you could have the following:

Usage:
    cargo [options] test [<args>...]

Options:
    -h, --help              Print this message

Now if you run cargo test --option you get:

{'--help': False,
 '<args>': ['--option'],
 'test': True}

(Assuming options_first=True). And as far as I understand that's what you want to achieve?

However, I'm not sure what you expect in case of cargo test -h. Do you expect -h to be passed to the delegated CLI, or to be handled by cargo?

Also, note, that to do cargo test instead of cargo-test in your "usage", the cargo script should not slice off cargo from argv before delegating to the script that handles cargo test.

P.S. "I'm sorry I wrote you such a long letter; I didn't have time to write a short one."

[alexcrichton]

@halst, thanks for the response! In case you're interested, here's a bit of background about cargo. Cargo is structured like git in that there's one main cargo command and then there's a bunch of sub-commands as cargo-foo binaries.

The main cargo command's help page looks like:

Usage:
    cargo <command> [<args>...]

The cargo command also parses its options with options_first as true (for the reasons you stated). We then delegate to the cargo-test binary with <args> as all the arguments.

You've noticed that cargo test -h shows cargo-test as the binary name instead of cargo test. I've currently done this because the command test doesn't actually appear in the command line for the cargo-test command line (it's in the binary name). I can and probably should fix this (especially for the documentation at the very least).

Now to explain what cargo-test is doing. The command builds the unit tests into a binary, and then runs the binary itself (to run the tests). The <args>... for the cargo-test command are the arguments that are passed through to the test binary (things like filtering tests, various flags, etc).

So all-in-all, this is in theory what I was expecting:

1. cargo test --nocapture, this parses to {'<args>': ['test', '--nocapture']} for the cargo command.
2. cargo invokes the cargo-test command with all of <args>
3. cargo-test --nocapture, this parses to {'<args>': ['--nocapture']} for the cargo-test command.
4. cargo-test invokes the binary with all of <args>
5. The test binary is run with the --nocapture flag.

I think the concept I have in my head is that flags to cargo test are prioritized as flags to cargo-test, but any unknown flags are passed through as <args> to the test binary itself. So to answer your question, I would expect cargo test -h to show the help for the cargo-test command, whereas cargo test --foo would run the test binary with the --foo flag.

That may have been a bit rambly, but does that make sense? Thanks again for responding!

[keleshev]

That totally makes sense. But the notion of "any unknown flags are passed as <args>" is challenging for docopt. Except for that, you can achieve what you want with:

1. cargo test --nocaputre parses to '<args>': ['test', '--nocapture'] for cargo
2. cargo prepends 'cargo' to <args> and dispatches ['cargo', 'test', '--nocapture'] to cargo-test, which usage is cargo test (with space).
3. cargo-test dispatches its <args> to the test binary.

So the only problem is handling options to cargo-test which should not be passed to the binary. This could be done with:

Manually inspecting the args produced by cargo-test to see if it contains any "special" options or not, then depending on that, either setting options_first to true or false. True if you want to pass them to test binary, false if you want to treat them as options to cargo-test.

This seems quite possible. I don't see any apparent flaws.

Anyway, you should be careful. You might end up with "collisions" of cargo-test options with the test binary options. Even worse: imagine you introduced a new option to cargo-test, and that suddenly broke your users build, because they happen to use option with the same name for their test binary.

[alexcrichton]

That also makes sense, thanks @halst! I'll noodle on this and see what I can come up with.

[wycats]

@alexcrichton do we have a good solution for this? If not, I am sad 😦

[alexcrichton]

Sadly I don't think I do know of a great solution to this without jumping through many hoops, but the fear being almost unable to add new options in the future does seem like a real worry to me, and with docopt it's at least consistent. That being said, it would still be nice to have a solution to this...

[alexcrichton]

For now I'm going to close this as working as intended. Currently we have a very consistent interface for all of the cargo-related tools, and in the future we may want to apply some of the tricks previously mentioned in this issue. I am quite worried by how solving this would severely limit us in adding future options to cargo test, so I think that today's situation may not be so bad after all.