[feature] allow hooks to extend conan command line interface

[SSE4]

it would be nice to allow conan hooks to extend conan command line interface to provide user-specified commands. this would bring more level to conan extensibility.

for instance, user may want to write custom command to display a package_id for the given recipe and settings. he may write his own hook, like:

def pre_cmd(output, conanfile, conanfile_path, reference, **kwargs):
    parser = argparse.ArgumentParser(description=self.inspect.__doc__,
                                         prog="conan print_package_id",
                                         formatter_class=SmartFormatter)
        parser.add_argument("path_or_reference", help="Path to a folder containing a recipe"
                            " (conanfile.py) or to a recipe file. e.g., "
                            "./my_project/conanfile.py. It could also be a reference")
    return {"print_package_id": parser}

and then use it like:

conan print_package_id conanfile.py

as a reference, right now Git offers similar thing - it will automatically pick up git-xxx executable and use it as git xxx sub-command (source).

few examples of using such extensibility:

• git flow

• git annex

• git ftw

• I've read the CONTRIBUTING guide.

[memsharded]

I wouldn't mind to provide this functionality, and we should at least design the Conan 2.0 api and command line (cc/ @czoido) to consider this possibility. Probably it should be called "hooks", but "porcelain" or "command alias" or something like that, I think this is a different concept.

[SSE4]

alias is also nice thing, but I suppose it's a little bit different. it's for providing shortcuts for long, but frequently used commands. e.g. git co might be a common alias to git checkout, or git unstage could be an alias to git reset HEAD --.

similarly, we may allow such thing for conan as well, e.g. alias conan rm to conan remove -f.

but here the suggestion is to provide brand new commands, which aren't represented but the currently available conan commands. e.g. allow hook author to write his own logic to print package_id, or print whatever (e.g. run conan lint and display linter checks against the conanfile).

[SSE4]

just in case, breakdown to the 3 smaller things here to avoid misunderstanding:

• alias or shortcut, provide shorter alternative for already available command (conan remove -f -> conan rm)
• git way: pick up existing tool/script named conan-xxx from the PATH and redirect arguments to it (conan-lint.exe -> conan lint)
• hooks way: provide pre_cmd conan hook to allow to register user-specified command line handlers, this also allows to directly access conan API and objects provided by conan (such as config, graph, etc.)

[jgsogo]

Just wanted to link here some valuable comments when we started to talk about plugins (not hooks, but there are things to keep in mind): #3778

[solvingj]

Here's how docker implemented the same concept... to be used as prior art: docker/cli#1534

[solvingj]

I've just started using an initial implementation of this and have several pieces of feedback i'll be posting shortly.

[solvingj]

We'll definitely need some distinguishing prefix for all user commands to avoid any possibility of conflict with future commands Conan decides to add. Possible mechanism include

• auto-prefixing all user commands as subcommands to user command: conan custom mycommand
• auto-prefixing characters on all user commands: conan custom-mycommand

[solvingj]

Some of the most generic features I'm looking forward to as advantages over just writing my own scripts which invoke conan_api mostly surround avoiding boilerplate python module setup:

• Unified logging with the same control mechanisms and locations as conan logging/tracing
• Sensible console output handling/coloring/etc
• Sensible wrappers for subprocess calls
• Sensible wrappers for http calls based on whatever http library conan is using at the time
• Pattern-based copy like self.copy() in conanfile

[SSE4]

linking #7170 as it's a bit related

[solvingj]

Additional feedback, it will be common for commands to be combinations or "customizations" of other commands. For example, in my very first experimental command, I wanted to replace a script which had been written before, with a new custom command.
The old script called the commands:

conan install .
conan info . --paths -n package_folder

To get the paths of the package folder in the local conan.

From inside a custom command, I can do this to easily invoke the conan install command properly:

    api_v1, _, _ = Conan.factory()
    command = Command(api_v1)
    command.install(args)

However, that only works because I don't care about the output of that command. For the next command: conan info... I need the output back. In theory I should not have to resort to parsing JSON or stdout because I have the conan_api at my fingertips.

Unfortunately, this common situation lands me in an unfortunate place between just calling another command and calling the API function for that command. If I call the command the same way I did with conan.install(args), I have to parse JSON or stdout. If I want to call the API equivalent function, I have to completely re-implement (copy/paste) the implementation of the 30 lines of argument parsing from the def inspect(self, *args): function in commands.py to my own custom command.

Ideally, for Conan 2.0, I suggest we implement a change to these two lines at the end of the inspect() command method.

        result = self._conan.inspect(args.path_or_reference, attributes, args.remote, quiet=quiet)
        Printer(self._out).print_inspect(result, raw=args.raw)
        ....

In theory, we could just change all these command-level functions to always return the result so that they could all be called programmatically the way I want to do for this case. Then each could have a wrapper function which does the post-processing of the result for the normal case.

[jgsogo]

After #7170 which should introduce a new CLI interface, we need to refactor all the conan_api in order to document it and, later, we will document how to add custom commands. This will take time, let's go step by step.

[solvingj]

I'll continue experimenting on the future user experience side and capturing my feedback in this ticket since it's the most appropriate place at the moment. I'll continue to provide detailed user stories like this one from those experiments. They can be reviewed during the implementation process whenever that reaches a suitable point.

[solvingj]

Just like native conan commands, one of the first things I need is a proper configuration file strategy to define defaults for all my custom commands for my organization. Obviously, we don't want to involve conan.conf for custom commands, and we don't want each command to have to define and parse it's own config file manually, nor require users to pass a path to a config on the CLI for it to be used. Thus, the only reasonably elegant approach is to provide an automatic convention-based strategy for looking for and loading config files for each command, which probably should all follow the "ini" config convention of conan.conf.

So, for custom command my_command , automatically look for my_command.conf, load it, and pass the parsed config object into the command as an additional parameter. It could be in the commands directory, or perhaps a new configs subdirectory. That detail can remain open for discussion.

This one-config-file-per-command strategy has a nice quality of modularity and consistency, in that each config file and command are nice and isolated. It's easy to understand and use for the first time. However, it has a scalability quality which is not-so-nice. If an organization has 20 custom commands, managing 20 config files feels real bad. Thus, we should offer an alternative strategy for this case.

In addition to searching for command_name.conf, we should also search for custom_commands.conf which will be an ini-format with a section for each command. Thus, organizations can have a single config file for managing all their custom command configs. And then, if both files exist, the command_name.conf values should probably override custom_commands.conf.

[memsharded]

Just like native conan commands, one of the first things I need is a proper configuration file strategy to define defaults for all my custom commands for my organization. Obviously, we don't want to involve conan.conf for custom commands, and we don't want each command to have to define and parse it's own config file manually, nor require users to pass a path to a config on the CLI for it to be used.

We like to think in terms of pains to solve, issues to address, instead of features to implement. It would be good to illustrate it with some real world examples. The truth is that so far in the conan.conf there are practically no command-specific configuration. There are things like cpu_count or revisions_enabled, that apply to all or many commands. Otherwise, they are command default arguments, not configuration.

So I would say that this is too early to be considered for implementation, and not necessary to include this in the first iterations of the "user commands" feature.

[solvingj]

Another item which came up immediately was how to handle additional third-party python package dependencies users may wish to use in their custom commands. Initially, it seems to be just like with conanfile.py. If users want to use additional libraries, they'll have to ensure they install those separately with pip. Thus, using custom PIP packages in a custom command is a substantial barrier to sharing command publicly, but it's conceivable that some enterprise organizations may choose to accept the tradeoff. However, it's likely people will find reason to use them, and I anticipate a non-zero number of issues reported where users get an error from their custom commands because they're trying to run them on a new machine/container which doesn't have some implicit prerequisite pip package installed and they think it's Conan's problem. Same as with conanfile.py i guess.

So, in summary, this comment is a no-op, but will be good here for future readers to know that it was considered.

[solvingj]

I have just published the first POC here based on a recent commit of @czoido 's conan fork (CLIV2 branch).
CONAN_V2_CLI=1 environment variable must be set in that branch to test the new feature.
https://github.com/solvingj/custom_conan_commands/tree/with_docker
Based off this commit in @czoido fork:
https://github.com/czoido/conan/tree/ada4cf9e893a4944bfc84b8e6b97b3ebeedaaa74
Note: CONAN_V2_CLI=1 environment variable must be set to test the new feature.

WRT the purpose of the command:

It's a very straightforward and common workflow to want to run Conan inside a docker container. In particular, developers often do development work inside their local environment, and then just need to do a sanity test in at least one docker container before pushing to let CI do a more comprehensive test, Conan package tools provides similar wrapping logic and can theoretically achieve something similar, however that project completely different architecture, priorities, and many different implications which make it less desirable for this case.

WRT some of my previous comments about the experience of creating this command:

I implemented a fairly robust configuration file management strategy, which took about 60 lines. In this case, about 50 of those are boilerplate and 10 actually declare the configuration items and their types/defaults/etc. If argparse had first-class support for env vars and configuration files, none of that would have been necessary.

I implemented generic subprocess wrappers because the Conan "runner" wasn't exactly what I needed, so that added another 25 lines of boilerplate.

About 75 lines were spent orchestrating the cross-platform docker logic and command composition which is more or less the core purpose of the command.

About 30 lines are the command-line arguments parsing.

[memsharded]

A new mechanism is already in place "modular user custom commands" for Conan 2.0, already merged to develop2, so this feature can be considered done.

[SzBosch]

Hello together,

we are trying at the moment to find the best way to get the RREV from a conanfile in a machine-readable form (e.g. on stdout or json file).

It seems conan does not provide it via commands like export or info.
export is very close, but we do not want to export it to the cache, just calculate the RREV and the stdout is no nice and stable interface.

Could we use (conan 1.x) the above mentioned CONAN_V2_CLI=1 to create the mentioned "modular user custom commands" to have a nice interface to get the RREV?
Is there any documentation?

[memsharded]

Hi @SzBosch

we are trying at the moment to find the best way to get the RREV from a conanfile in a machine-readable form (e.g. on stdout or json file).

A conanfile doesn't have a rrev recipe-revision per se. It depends on the exported files, if any of the exported files changes, a new rrev is get. So it is basically impossible to compute the recipe revision without executing an actual export to the cache.

Using an actual export, you can get the rrev in machine readable format with conan export . --format=json (Conan 2.0, in Conan 1.X I guess you need to capture and parse the output)

modular user custom commands

I am afraid that these only exist in Conan 2.0, it depends on 2.0 new architecture, it is also impossible to backport them to Conan 1.X