Feature/extension scripts #224

timabrmsn · 2021-02-04T13:25:40Z

This PR adds some helpers for writing code42cli powered extensions with a minimal amount of boilerplate, giving the extension-writer cli profiles, exception handling, and sdk initialization out of the box, so all they need to do is their custom script logic. Extensions can be standalone scripts that require no installation, or can be installed as a plugin so the commands become part of the main CLI usage (using the click_plugins package that helpfully keeps everything else working if an extension breaks).

Standalone scripts

To write a cli-powered standalone script, you just need to import the script custom click.Group and sdk_options, then add your click commands. This example is a script that takes a guid from an option and pretty-prints the resulting settings dict:

import click
from code42cli.extensions import script, sdk_options
from pprint import pprint

@click.option("--guid", required=True)
@sdk_options
def get_device_settings(state, guid):
    settings_response = state.sdk.devices.get_settings(guid)
    pprint(settings_response.data)


if __name__ == "__main__":
    script.add_command(get_device_settings)
    script()

If there's only a single defined command, the custom group makes it the default, so it can be invoked just from the base script itself:

# python script.py -h
Usage: script.py  [OPTIONS]

Options:
  --guid TEXT     [required]
  -d, --debug     Turn on debug logging.
  --profile TEXT  The name of the Code42 CLI profile to use when executing this command.
  -h, --help      Show this message and exit.

If multiple commands are defined, then script operates as a normal group and enumerates them, requiring the subcommand be called:

import click
from code42cli.extensions import script, sdk_options
from pprint import pprint


@click.option("--guid", required=True)
@sdk_options
def get_device_settings(state, guid):
    settings_response = state.sdk.devices.get_settings(guid)
    pprint(settings_response.data)


@click.argument("guid")
@sdk_options
def get_device_user(state, guid):
    settings_response = state.sdk.devices.get_settings(guid)
    print(settings_response.user_id)


if __name__ == "__main__":
    script.add_command(get_device_settings)
    script.add_command(get_device_user)
    script()

# python script.py -h
Usage: script.py [OPTIONS] COMMAND [ARGS]...

Options:
  -h, --help  Show this message and exit.

Commands:
  get-device-settings
  get-device-user

To help when a user might have multiple python installations in their path and python script.py isn't finding the code42cli package (but the regular code42 -h command works), a --python option has been added to print out the path to the interpreter that code42 is installed in. So the user can then just call $(code42 --python) script.py and it should just work without having to fiddle with their paths.

Installable extensions

To make the above script installable as a permanent extension to the CLI, you can put it in a package structure with a setup.py where the defined entrypoints are code42cli.plugins, the CLI will then be able to find them.

I've created an example repo with two installable packages here: https://github.com/timabrmsn/cli_plugin_test

To install the settings example command from above you can run

$(code42 --python) -m pip install "git+https://github.com/timabrmsn/cli_plugin_test#egg=code42cli_settings&subdirectory=settings"

And to install the users one:

$(code42 --python) -m pip install "git+https://github.com/timabrmsn/cli_plugin_test#egg=code42cli_users&subdirectory=users"

This allows support/PS to have a single repo of their extension scripts, and they can either link directly to the standalone one, or instruct the user to install it, depending on the need/use-case.

… to InterpreterGroup

github-actions · 2021-02-04T13:25:54Z

CLA Assistant Lite bot All contributors have signed the CLA ✍️

alanag13 · 2021-02-04T15:06:21Z

A couple thoughts here:

It may be valuable to look into a way to see if it would be possible to dynamically discover/import these scripts. For example, could we have a designated directory in .code42cli where these could be dropped, and then they would all show up as additional options in the main help text? It seems like it would be really nice to be able to keep a collection of these and not have to remember the names/paths of custom scripts as they get developed over time.
We should probably look over our API surface and come up with a better organizational scheme for what is public (okay to call via an extension) and what isn't, and make sure to adjust our coding practices to reflect that the terminal itself is no longer the only interface.
We should demo this to PS, perhaps even only in this state, to give them a feel for what the experience would be like for them.

antazoey · 2021-02-04T15:45:13Z

src/code42cli/click_ext/groups.py

+    def parse_args(self, ctx, args):
+        if len(args) > 1 and args[0] == "--script":
+            args[0] = sys.executable
+            if platform.system() == "Windows":


Can we sanitize the args somehow?
Probably a far-fetched concern, but if the CLI was running on a server, like maybe as part of an integration of some sort, and users were able to call custom scripts remotely, could this could be used as part of a shell-injection attack?

Bandit scan:

Issue: [B605:start_process_with_a_shell] Starting a process with a shell, possible injection detected, security issue. Severity: High Confidence: High Location: src/code42cli/click_ext/groups.py:40 More Info: https://bandit.readthedocs.io/en/latest/plugins/b605_start_process_with_a_shell.html 39 cmd = shlex.join(args) 40 status = os.system(cmd) 41 sys.exit(status)

Oddly, bandit claims it is a HIGH severity but the documentation says it should be LOW/

I don't think we need to worry about that since this function is explicitly for running user-provided code. If someone can hijack the actual execution of the CLI to get it to call --script on something unintended, they likely already have the ability to just run any arbitrary command they want.

I was imagining a scenario where some integration or app intentionally allows user input to the Code42 CLI, which like I said, is maybe a far-fetched scenario. But, in this scenario, they expose the CLI, so it's not hijacked (yet).

but it would be outside of our control if someone did that, albeit a dangerous decision on their part, but sanitization on our end could be an extra layer of security

I did start out playing around with just having code42 --python just return the full path to the interpreter code42 was installed under. So then if a user was having problems with the regular python script.py they could try $(code42 --python) script.py.

I think I'll just revert the exec python logic back to just printing the python path. It keeps the code simple and gives users a relatively easy path to figuring things out if their environment is broken.

antazoey · 2021-02-04T20:19:42Z

A command that executes raw input from the user is kind of a red flag to me. That should be avoided whenever possible.

What if this PR could be made into a private extension for internal Code42 use? They would use this private CLI package (who's upstream is this public repo), and from there, they can add their own scripts?

Just a thought.

antazoey · 2021-02-04T23:39:10Z

What if - we made another python package, like code42cli-lib that contains the parts of the code42cli that we want to share. Then, they can just write normal python scripts.

Edit: Is it important that the be able call it via code42 <cmd> rather than just python <custom-script.py ?

Edit2: Or it can just be in this package -- just a code42cli.lib path where you can import the profile and stuff.

kiran-chaudhary · 2021-02-05T06:59:30Z

I am sure this might have been discussed but for my benefit I would like to raise it, if someone knows python then they can use py42, the only advantage cli can provide here to use script is about incremental results. Also one cannot use the custom script to go ahead and try something that is not supported in Py42!! So do we have real use cases here? the whole purpose of having it open source was others can come and add commands they feel are useful and raise PR against it!!

kiran-chaudhary · 2021-02-05T07:03:05Z

the standard way to support such thing would be to have -i or --interactive option instead of a --script option. As we do python -i and that would load some default modules/objects and one can start doing as explained in the description.

kiran-chaudhary · 2021-02-05T07:09:34Z

Also tested out the same,

code42 --script -h throws error AttributeError: module 'shlex' has no attribute 'join'

timabrmsn · 2021-02-05T13:36:36Z

@kiran-chaudhary the background for this is that while we do want internal teams who write scripts that go to customers (Pro Service, Support, SST, etc) to contribute to the CLI development, the CLI isn't going to be able to accommodate every specific use-case. Sometimes a customer has a very particular need that the general goal of the CLI doesn't accomplish, and so support/PS/SST will need to write something custom for them.

Exposing this "extension" functionality in the CLI relieves the writer of the burden of having to do any boilerplate logic around getting credentials from the user, initializing the sdk, and handling/logging every possible Py42 error.

So remember the use case here is primarily for internal teams to be able to quickly and easily write custom scripts that are intended to be given to and run by customers (who often don't have a lot of programming experience).

With these, our teams can write up a script quickly, give it to the customer and just say "run pip3 install code42cli and create a profile". And this only needs to be done once for all future custom scripts to just work!

And to @unparalleled-js's question about is it important if they call it code42 --script script.py instead of python script.py, it's important to have an easy option to run it if there are path issues that prevent python script.py from working (like if their pip cmd is linked to a different python install than their python cmd). This way we know that if code42 <regular_cmd> works, then our script should just work too.

timabrmsn · 2021-02-05T13:39:52Z

AttributeError: module 'shlex' has no attribute 'join'

Apparently .join() was only added in 3.8. I'll update.

antazoey · 2021-02-05T13:51:18Z

Can we at least get reach out to the security guild or someone and get their advice?

antazoey · 2021-02-05T13:59:31Z

@timabrmsn If this is intended for internal people to use, as you say, shouldn't it not be part of the main/publicly available CLI?

timabrmsn · 2021-02-05T14:13:16Z

The primary writers of these scripts are going to be internal teams, yes. But the primary runners of these scripts will be customers, so they have to work with regular public-facing CLI installs.

antazoey · 2021-02-05T14:18:37Z

The primary writers of these scripts are going to be internal teams, yes. But the primary runners of these scripts will be customers, so they have to work with regular public-facing CLI installs.

Ah ok I did not realize that. I will process this.

alanag13 · 2021-02-05T14:52:13Z

@timabrmsn @unparalleled-js @kiran-chaudhary there's a lot of chatter happening here and comments are getting buried. Lets set up a meeting and get everyone on the same page.

kiran-chaudhary

LGTM from code's point of view. Wondering why we haven't documented this feature, the description given in the PR should be documented for future reference.

kiran-chaudhary · 2021-02-17T04:40:44Z

src/code42cli/extensions.py

@@ -0,0 +1,5 @@
+from code42cli.click_ext.groups import ExtensionGroup
+from code42cli.main import CONTEXT_SETTINGS
+from code42cli.options import sdk_options  # noqa: F401


wondering why are ignoring unused imports or suppressing warnings, noqa... statement? I need to read more on the usage of this statement, but the first thing that comes to my mind is will this suppress warnings only in context to this module or will it suppress all-over code42cli!!?

Also can this be resolved by abstraction/design, breaking up the modules in such a way that we don't have to suppress any warnings or unused imports!

I'm pretty sure the comment only suppresses it for the line being commented, I still got unused import style failures after this commit went in.

I imported sdk_options here just for the convenience of script-writers, so they can just do:

from code42cli.extensions import script, sdk_options

Another way to get around this would be to create a directory named extensions with an __init__.py importing things for convenience; it seems like that is ok to do because we do it other places to make importing things more convenient.

Ah, good call. Making it a module would also allow us to better expand/organize things in the future.

Huh, so the style cop still balks at unused imports even in the module init. I wonder why it doesn't do that in py42? I'm not seeing any config differences...

nevermind. I found where we're ignoring it in the setup.cfg

timabrmsn · 2021-02-17T14:23:29Z

Wondering why we haven't documented this feature

Documentation will be coming, I'm just holding off for now until I can demo the feature to our customer-facing teams to get any feedback before we finalize this implementation. I'm also not sure where the best spot is for the documentation to live. It could be in the README, ReadTheDocs, or the Dev Portal articles...

…ure removal

alanag13 · 2021-02-17T20:14:29Z

LGTM too, but I'll leave this for now in case you get feedback and want to add changes here.

antazoey · 2021-02-26T15:00:13Z

Documentation would be a plus

antazoey

Sorry, I accidentally approved early - everything looks good, just need a changelog and perhaps a readme update

…returning a decorator

timabrmsn · 2021-03-01T16:39:12Z

Built user-guide can be found here: https://clidocs.code42.com/en/feature-extension_scripts/userguides/extensions.html

antazoey · 2021-03-01T18:50:23Z

docs/guides.md

@@ -5,3 +5,4 @@
 * [Ingest file events or alerts into a SIEM](userguides/siemexample.md)
 * [Manage detection list users](userguides/detectionlists.md)
 * [Manage legal hold users](userguides/legalhold.md)
+* [Write custom extension scripts using the code42cli and py42](userguides/extensions.md)


Should the code42cli be the Code42 CLI?

yeah, probably.

docs/userguides/extensions.md

antazoey · 2021-03-01T19:15:38Z

src/code42cli/extensions/__init__.py

+from code42cli.options import profile_option
+
+
+def sdk_options(f):


Maybe you just want to be really explicit, but this could just be

from code42cli.options import sdk_options as sdk_opts def sdk_options(f): """Decorator that adds two `click.option`s (--profile, --debug) to wrapped command, as well as passing the `code42cli.options.CLIState` object using the [click.make_pass_decorator](https://click.palletsprojects.com/en/7.x/api/#click.make_pass_decorator), which automatically instantiates the `py42` sdk using the Code42 profile provided from the `--profile` option. The `py42` sdk can be accessed from the `state.sdk` attribute. Example: @click.command() @sdk_options def get_current_user_command(state): my_user = state.sdk.users.get_current() print(my_user) """ f = sdk_opts()(f) return f

Or even sdk_options = sdk_opts

I mainly wanted to have the object we expose be it's own thing in case we ever want to change our internal sdk_options we're not having to work around the extension backwards compatibility.

antazoey

Very cool!

antazoey · 2021-03-01T19:22:50Z

docs/userguides/extensions.md

+
+## Before you begin
+
+The Code42 CLI is a python application written using the [click framework](https://click.palletsprojects.com/en/7.x/), and the exposed extension objects are custom `click` classes. A basic knowledge of how to define `click` commands, arguments, and options is required.


I am a fan of line breaks in markdown because of horizontal scrolling in text editors. It seems like most markdown renderers can handle converting the newlines to spaces too. Maybe this is the issue with my IDE -- but I have seen markdown rules around line length too, so I know it's not just me

Huh, for some reason I had it in my mind that the newlines could cause odd linebreaks in the rendered sphinx text, but I tested it out and it doesn't seem to matter. Made it much more readable now!

timabrmsn added 8 commits February 3, 2021 07:33

initial work for extension scripts

d97f620

handle cmd construction on Windows

43d6670

use next(iter(dict)) instead of list(dict.keys)[0]

d1c93cf

require --python to be first arg to trigger interpreter

35e3374

adjust arg swap given we know it's arg[0]

8a6f9ea

remove cmd name if there's only one

a77a5fb

rename --python option to --script, document option and add docstring…

718942e

… to InterpreterGroup

alias sdk_options in extensions.py

c4c3976

timabrmsn added 3 commits February 4, 2021 07:34

style

b668c4d

allow single commands to execute when no args/options required

1dde0f2

style

3a2e5e5

antazoey reviewed Feb 4, 2021

View reviewed changes

timabrmsn added 2 commits February 5, 2021 08:43

just print executable path instead of exec-ing directly

1fd7b95

remove unused imports

119ae7a

timabrmsn added 2 commits February 8, 2021 08:39

test installable plugin

71671fb

delete test plugin folder

97f258f

kiran-chaudhary reviewed Feb 17, 2021

View reviewed changes

timabrmsn added 2 commits February 17, 2021 12:23

make extensions a module

908d1f0

add unused import ignore to setup.cfg, document import to prevent fut…

1b7dfaa

…ure removal

Merge branch 'master' into feature/extension_scripts

9e57452

antazoey approved these changes Feb 26, 2021

View reviewed changes

antazoey self-requested a review February 26, 2021 15:00

antazoey suggested changes Feb 26, 2021

View reviewed changes

timabrmsn added 3 commits March 1, 2021 10:26

Add docs and redefine sdk_options to be regular decorator instead of …

abfa3db

…returning a decorator

Merge branch 'master' into feature/extension_scripts

8651fb7

style

66c1e4d

timabrmsn added 4 commits March 1, 2021 10:41

update changelog

3d29f2f

Add note on extensions to README

1c42b6d

style

b6fae55

Merge branch 'master' into feature/extension_scripts

4e5047e

antazoey reviewed Mar 1, 2021

View reviewed changes

timabrmsn added 2 commits March 1, 2021 13:09

code42cli > Code42 CLI

0fb2fb3

remove ignore since we're using all imports now

1b74cff

antazoey reviewed Mar 1, 2021

View reviewed changes

antazoey self-requested a review March 1, 2021 19:18

antazoey approved these changes Mar 1, 2021

View reviewed changes

comma

ae7eb6c

antazoey reviewed Mar 1, 2021

View reviewed changes

make readable w linebreaks

c538691

timabrmsn merged commit db7e49e into master Mar 1, 2021

github-actions bot locked and limited conversation to collaborators Mar 1, 2021

antazoey deleted the feature/extension_scripts branch April 21, 2021 13:56

		from code42cli.options import profile_option


		def sdk_options(f):


		## Before you begin

		The Code42 CLI is a python application written using the [click framework](https://click.palletsprojects.com/en/7.x/), and the exposed extension objects are custom `click` classes. A basic knowledge of how to define `click` commands, arguments, and options is required.

Feature/extension scripts #224

Feature/extension scripts #224

Uh oh!

Conversation

timabrmsn commented Feb 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Standalone scripts

Installable extensions

Uh oh!

github-actions bot commented Feb 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alanag13 commented Feb 4, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antazoey Feb 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antazoey commented Feb 4, 2021

Uh oh!

antazoey commented Feb 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kiran-chaudhary commented Feb 5, 2021

Uh oh!

kiran-chaudhary commented Feb 5, 2021

Uh oh!

kiran-chaudhary commented Feb 5, 2021

Uh oh!

timabrmsn commented Feb 5, 2021

Uh oh!

timabrmsn commented Feb 5, 2021

Uh oh!

antazoey commented Feb 5, 2021

Uh oh!

antazoey commented Feb 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timabrmsn commented Feb 5, 2021

Uh oh!

antazoey commented Feb 5, 2021

Uh oh!

alanag13 commented Feb 5, 2021

Uh oh!

kiran-chaudhary left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timabrmsn commented Feb 17, 2021

Uh oh!

alanag13 commented Feb 17, 2021

Uh oh!

antazoey commented Feb 26, 2021

Uh oh!

antazoey left a comment

Choose a reason for hiding this comment

Uh oh!

timabrmsn commented Feb 4, 2021 •

edited

Loading

github-actions bot commented Feb 4, 2021 •

edited

Loading

antazoey Feb 4, 2021 •

edited

Loading

antazoey commented Feb 4, 2021 •

edited

Loading

antazoey commented Feb 5, 2021 •

edited

Loading

antazoey Mar 1, 2021 •

edited

Loading