Ability to generate help output golden test file?

[SUPERCILEX]

Please complete the following tasks

• I have searched the discussions
• I have searched the open and rejected issues

Clap Version

3.x

Describe your use case

It's extremely helpful for reviewers to be able to see the changes to your CLI's interface from the user's point of view. It's also helpful to have what's essentially a rendered man page autogenerated for users to browse.

Describe the solution you'd like

Offer some way to generate one string with every commands help page concatenated together. Maybe also offer easy golden testing though honestly that should just be left to the user with an example using the goldenfile crate.

Alternatives, if applicable

Maybe trycmd, but I don't know how you automatically get all the help pages.

Additional Context

No response

[SUPERCILEX]

Extending this idea a little further, you could potentially output the help as markdown with header sections for each command which could then be used as an autogenerated docs website.

I guess that means the interface should actually return a map with the command path to its help output (so you can be flexible and output one page per command for example) and some way to specify what that output should look like. Then I can concatenate it myself and pass it through a golden test.

[epage]

Created a quick example of what this could potentially look like

#!/usr/bin/env -S rust-script --debug

//! ```cargo
//! [dependencies]
//! clap = { version = "3.2.8", features = ["env", "derive"] }
//! ```

use std::ffi::OsString;
use std::path::PathBuf;

use clap::{Args, Parser, Subcommand};

/// A fictional versioning CLI
#[derive(Debug, Parser)]
#[clap(name = "git")]
#[clap(about = "A fictional versioning CLI", long_about = None)]
struct Cli {
    #[clap(subcommand)]
    command: Commands,
}

#[derive(Debug, Subcommand)]
enum Commands {
    /// Clones repos
    #[clap(arg_required_else_help = true)]
    Clone {
        /// The remote to clone
        #[clap(value_parser)]
        remote: String,
    },
    /// pushes things
    #[clap(arg_required_else_help = true)]
    Push {
        /// The remote to target
        #[clap(value_parser)]
        remote: String,
    },
    /// adds things
    #[clap(arg_required_else_help = true)]
    Add {
        /// Stuff to add
        #[clap(required = true, value_parser)]
        path: Vec<PathBuf>,
    },
    Stash(Stash),
    #[clap(external_subcommand)]
    External(Vec<OsString>),
}

#[derive(Debug, Args)]
#[clap(args_conflicts_with_subcommands = true)]
struct Stash {
    #[clap(subcommand)]
    command: Option<StashCommands>,

    #[clap(flatten)]
    push: StashPush,
}

#[derive(Debug, Subcommand)]
enum StashCommands {
    Push(StashPush),
    Pop {
        #[clap(value_parser)]
        stash: Option<String>,
    },
    Apply {
        #[clap(value_parser)]
        stash: Option<String>,
    },
}

#[derive(Debug, Args)]
struct StashPush {
    #[clap(short, long, value_parser)]
    message: Option<String>,
}

fn main() {
    use clap::CommandFactory;
    let mut command = Cli::command();

    let mut buffer: Vec<u8> = Default::default();
    command.build();
    write_help(&mut buffer, &mut command, 0);
    let buffer = String::from_utf8(buffer).unwrap();
    println!("{}", buffer);
}

fn write_help(buffer: &mut impl std::io::Write, cmd: &mut clap::Command<'_>, depth: usize) {
    let header = (0..=depth).map(|_| '#').collect::<String>();
    let _ = writeln!(buffer, "{} {}", header, cmd.get_name());
    let _ = writeln!(buffer);
    let _ = cmd.write_long_help(buffer);

    for sub in cmd.get_subcommands_mut() {
        let _ = writeln!(buffer);
        write_help(buffer, sub, depth + 1);
    }
}

EDIT: Updated to capture it in-memory

Note: you could use snapbox to do snapshot testing. It is the core of trycmd, so you'd save on compile times.

[epage]

While I can understand the importance of this and the value of a happy path to it to encourage it, I feel like there is too much policy involved in this to have clap involved atm.

I could see us adding an example of this though.

[SUPERCILEX]

This is really great, thank you! Implemented it here for the curious: SUPERCILEX/ftzz@734d0e3

[SUPERCILEX]

Some notes: using the wrap_help feature breaks things — this could probably be fixed by disabling wrapping when some env var is detected, but eh that'll be finicky.

A gitattribute needs to be added to the golden file that checks it out with LN endings on windows. (Or use a golden library that ignores different line endings.)

[epage]

Re-opening to track adding an example

[SUPERCILEX]

Here's the latest that I've settled on if you want to copypasta. I'm quite happy with it:

    #[test]
    #[cfg_attr(miri, ignore)] // wrap_help breaks miri
    fn help_for_review() {
        let mut command = Ftzz::command();

        command.build();

        let mut long = String::new();
        let mut short = String::new();

        write_help(&mut long, &mut command, LongOrShortHelp::Long);
        write_help(&mut short, &mut command, LongOrShortHelp::Short);

        expect_file!["../command-reference.golden"].assert_eq(&long);
        expect_file!["../command-reference-short.golden"].assert_eq(&short);
    }

    #[derive(Copy, Clone)]
    enum LongOrShortHelp {
        Long,
        Short,
    }

    fn write_help(buffer: &mut impl Write, cmd: &mut Command, long_or_short_help: LongOrShortHelp) {
        write!(
            buffer,
            "{}",
            match long_or_short_help {
                LongOrShortHelp::Long => cmd.render_long_help(),
                LongOrShortHelp::Short => cmd.render_help(),
            }
        )
        .unwrap();

        for sub in cmd.get_subcommands_mut() {
            writeln!(buffer).unwrap();
            writeln!(buffer, "---").unwrap();
            writeln!(buffer).unwrap();

            write_help(buffer, sub, long_or_short_help);
        }
    }