Disable/Enable Sensors/Rules/Aliases via blacklisting/whitelisting in pack config

[arm4b]

This was discussed for several times and was an issue more than a few times internally and in #community as well.

Infrastructure as a code is cool if you control the entire code, but StackStorm exchange has packs with Sensors, Rules, Aliases which are enabled by default. This brings a lot of noise after installing a pack.

The worst, - when we change the sensor locally to enabled: false, next pack re-installation or re-provision replaces that change and st2ctl reload brings sensor or whatever back online.
Same problem is when vice-versa content is disabled by default in Exchange Pack and we want to enable it in our system.

---

The proposal is to have a mechanism to optionally blacklist/whitelist Sensors, Rules, Aliases via pack config which will take precedence over the metadata with possibility to include/exclude specific content to be registered or disabled with enable-many, disable-many, enable-one, disable-one, glob logic.

This way we could keep infrastructure as a code via existing /opt/stackstorm/packs/<pack>.yml definition.

This will solve #3016 and help with #3963 as well as other issues related to pack content registration.

[nmaludy]

+1

Some things that i'm thinking about:

• Should there be a global config (maybe: /opt/stackstorm/configs/_global.yaml) with default rule(s), and the exceptions/overrides in the individual pack configs?
• What would the config syntax look like in the /opt/stackstorm/configs/<pack>.yaml and how would this be validated? Some sort of schema additions at registration time?

Brainstorming global config

---
st2:
  rules:
    defaults:
      - name: *
        enabled: false
    exceptions:
      - name: backups.full_backup
        enabled: true
  aliases:
    defaults:
      - name: *
        enabled: false
    exceptions:
      - name: st2.*
        enabled: true

Brainstorming pack config

---
menandmice:
  dev:
    server: menandmice.dev.domain.tld
    username: user
    password: "{{ st2kv.system.menandmice.password }}"


st2:
  rules:
    exceptions:
      - name: menandmice.discovery
        enabled: true
  aliases:
    exceptions:
      - name: menandmimce.*
        enabled: true

[arm4b]

@nmaludy thanks for brainstorming this!

From that ^^ example, I definitely like the first two levels of definition:

---
st2: # as a core-reserved key
  rules:
  aliases:
  sensors:

need to think more about the rest to make it more obvious & clear & easy.

[nmaludy]

Problem

Currently actions, aliases, rules and sensors (called items) installed in a pack
assume the enabled state that is defined in the git repo where the pack
originates. This is fine in most cases, however it is often the case that
deployments will need to selectively enable or disable specific items.

Example:
Organization A is security focused and wants to disable all of the ChatOps
aliases in the packs pack.

Example #2:
Organization B is trying to minimize their StackStorm install and they
want to disable sensors they are not using. In this case they would like
to disable the linux.file_watch_sensor that comes default with StackStorm.

Potential Solutions

In the spirit of StackStorm we would like to implement a solution in the
Infrastructure as Code style. This solution would define a configuration file
/opt/stackstorm/configs/_global.yaml that contains definitions for the desired
enabled/disable state of the pack items. The configuration file will be partitioned
by pack and item, then allow matching of item names via glob patterns abc*.

---
<pack_name>:
  actions:
    ...
  aliases:
    ...
  rules:
    ...
  sensors:
    ...
    
orion:
  aliases:
    ...
  rules:
    ...
    
linux:
  actions:
    ...
  sensors:
    ...

Item defintions will be the same for each item type (actions, aliases, rules and
sensors) and will support both whitelisting and blacklisting. The item definition
will contain a default parameter that defines the default state of all items
within the pack. This will allow blanket enabling/disabling of items:

---
# disable all aliases in the st2 pack
st2:
  aliases:
    default:
      enabled: false

To support both whitelisting and blacklisting we will have a exceptions parameter
that is used to define "exceptions" to the "default" (maybe the terminology can
be improved here). We define "whitelisting" as disabling all item instances by
default then explicitly enabling a select few instances. Conversely "blacklisting"
is when all item instances are enabled by default and a few select instances
are disabled. Both of these scenarios can be accomplished using the default
and exceptions clauses:

Whitelisting Example

Here all aliases in the st2 pack are disabled except st2_actions_list and
st2_rules_list.

---
st2:
  aliases:
    default:
      enabled: false
    exceptions:
      - name: st2_actions_list
        enabled: true
      - name: st2_rules_list
        enabled: true

Blacklisting Example

Here all aliases in the packs pack are enabled except pack_install.

---
packs:
  aliases:
    default:
      enabled: true
    exceptions:
      - name: pack_install
        enabled: false

If this is not enough flexibility we can extend this same concept to pack
config files. We could extend the configuration schema of each pack, allowing
item definitions to be placed in the pack config instead of the new "global"
config.

Global Config

/opt/stackstorm/configs/_global.yaml

Schema

---
type: object
description: >
  Keys are pack names, values contain global config information
parameters:
  actions:
    "$ref": "#/definitions/config_obj"
  aliases:
    "$ref": "#/definitions/config_obj"
  rules:
    "$ref": "#/definitions/config_obj"
  sensors:
    "$ref": "#/definitions/config_obj"

definitions:
  config_obj:
    type: object
    properties:
      default:
        type: object
        description: >
          The default values for the parameters on the action/alias/rule/sensor
        properties:
          enabled:
            type: boolean
            description: "Are item instances enabled by default."
            default: true
      exceptions:
        type: array
        description: >
          List of items that are different from the default
        items:
          type: object
          properties:
            name:
              type: string
              description: "Name of the item (glob patterns accepted)."
            enabled:
              type: boolean
              description: "Is the item instance enabled."

Example

---
# example: whitelist
# disable all aliases in the st2 pack except st2_executions_list
st2:
  aliases:
    default:
      enabled: false
    exceptions:
      - name: st2_executions_list
        enabled: true

# example: blacklist
# enable all aliases in the duo pack except 'auth_auth' and 'auth_demo'
# also disable the auth_wrap_demo action
duo:
  aliases:
    default:
      enabled: true
    exceptions:
      - name: auth_auth
        enabled: false
      - name: auth_demo
        enabled: false
  actions:
    default:
      enabled: true
    exceptions:
      - name: auth_wrap_demo
        enabled: false

# disable all sensors in the linux pack (file_watch_sensor)
linux:
  sensors:
    default: 
      enabled: false

[arm4b]

If we introduce something called global config @nmaludy proposing, I'd suggest:

1. Set behavior to register/disable only "global" content type in configs/global.yaml, like actions, sensors in general (default: register all as before).
2. Set even more granular control over pack content like whitelist/blacklist actions/aliases/sensors/rules by name and in general for pack via configs/<pack>/config.yml (this will take precedence over global config)

It's easier to manage infra-as-code when specific pack config is responsible to enable/disable only content it holds (actions,aliases,etc by name).

Another adjustment is simplifying the schema/defition:

Instead of:

duo:
  aliases:
    default:
      enabled: true1
    exceptions:
      - name: auth_auth
        enabled: false
      - name: auth_demo
        enabled: false

Something along the lines:

duo:
  aliases:
    default: enabled
    disabled:
      - auth_auth
      - auth_demo

and

duo:
  aliases:
    default: disabled
  rules:
    default: disabled
  actions:
    default: enabled

or any other way to simplify the definition if that makes sense to anyone.

[arm4b]

Someone using StackStorm + Infra as code in a real production environment will be aware about this feature. It bug us all the road in our own #ops work running StackStorm in production (it's actual for Docker story too), that's why the feature request is here.

The problem is almost every Exchange pack which comes with a bunch of enabled by default sensors/aliases/rules and so unconfigured sensors or rules you don't want are taking system resources, spamming in logs or weaken chatops security by exposing aliases you can't blacklist.

At the moment there is no mechanism to change this behavior in StackStorm: persistently enable or disable specific pack content in a more granular way (only fork pack or delete rules/sensors by hand which reverts after packs reinstall from Exchange).

@StackStorm/team please provide your opinions in this issue, as @nmaludy is considering to implement this and looking for more feedback to get the design/idea accepted.

[nmaludy]

@armab The reason why i created the schema like:

duo:
  aliases:
    default:
      enabled: true

Was to allow for future extension in case other properties wanted to be overloaded and/or configured.

I do like your idea of the configs/<pack>/config.yaml as well. It could be the same schema without the first level of pack name as a key. Example:

aliases:
  default:
    enabled: false

sensors:
  default:
    enabled: false

[hendec]

Where did this end up? Looking to do something similar

[punkrokk]

I’d like to color in this discussion. Ignoring actions for a moment - for sensors and rules - I think it makes the most sense to think of them like Linux services. And separate enabled out from their config. There is literally never a use case which 100% of users want a sensor or rule enabled by default, and defaulting to disabled / false is the Linux thing to do. There could be a pack install flag that enabled things, but I think this actually would remove a ton of tricky logic vs. a more straightforward implementation. Although there would need to be some state work done, but my guess is that’s already in the DB.

[arm4b]

When orgs/groups are building custom packs for their specific needs, - it's a common case when all sensors, rules, action aliases are enabled and needed from the pack installation straight away.
StackStorm Exchange is a bit different because of its massive use, - only some of the rules, sensors, aliases might be needed depending on users needs.

The point behind pack configs is to allow users to enable/disable specific sensors, rules, action aliases in a granular way and follow the infrastructure as code approach, which StackStorm believes and preaches everywhere in its design. This also conforms with the st2 mechanics as pack validation/db sync/configuration happens on pack content registration stage. This way pack configuration could be also changed on st2ctl reload register stage, not only on initial installation.

[winem]

The discussion in this thread is really interesting. I think about this for 2 days and am still not sure if it's relevant and an actual use-case or not so I just add my 2ct here:

Can't there be a 3rd case beside enabled and disabled? Something like package-default?
I just don't have a good example in mind where users might want to rely on the package default in addition to their black- or whitelist but I also think this depends on the users workflows and mindset. It'll probably rather be used by those who set the default to enabled and explicitly put the unwanted features onto the blacklist than vice-versa.

At the same time I see arguments for and against the package-default option and that it adds an additional layer of complexity. Now I'm curious what you think about it.

[punkrokk]

I’m dropping this thought here while I’m thinking of it: it would be interesting to consider having pack modes: say - production and community. Where the community mode allows overwriting and production does not.

[amanda11]

Coming back to this one again...

I'm not too sure on the global idea applying to all packs. Presumably with the global the precedence would have been:

1. Enabled/disabled from the pack definitions are taken first
2. Then apply any overrides from configs/global.yaml
3. Then apply any pack specific overrides

But if I'm using community packs and my own, then if I want to disable all pack sensors apart from the ones I've wrote, then I would end up having to specify the override in configs/global to disable them all, and then put them back in on the packs I want under packs/configs/pack.yaml.
I see this as being more complicated and unintuitive, then to just list the packs that I want to disable actions from.

So I'm not sure of the value of having a fully global across all packs, and think it might be easier to use if we always specify at a pack level (whether that be in global.yaml with the pack name) or under configs/pack/config.yaml with the syntax where pack name is not needed.

I'd therefore be up for going for one of these options:

1. Put it all in a configs/global.yaml - but definitions per pack e.g. Disable/Enable Sensors/Rules/Aliases via blacklisting/whitelisting in pack config #3973 (comment)
2. Have individual configs//config.yaml with the overrides for each pack.

Thoughts?

I'd be quite interested in taking a look at this if I get a chance...

[nzlosh]

A single global.yaml seems like it'd get messy to maintain as it grew. The more packs StackStorm has installed the more encumbered editing becomes. A more modular approach would be to mirror the pack directory structure in the config directory structure and parse both for pack values and configuration overrides.

The order of evaluation (last value set wins) would be the following:
1st. default state from st2 core
2nd. pack
3rd. config

Taking the email pack as an example. When the pack is installed it comes with the following structure.

/opt/stackstorm/packs# tree email
email
├── actions
│   ├── send_email.py
│   └── send_email.yaml
├── CHANGES.md
├── config.schema.yaml
├── email.yaml.example
├── icon.png
├── LICENSE
├── pack.yaml
├── README.md
├── requirements.txt
├── rules
│   └── sample.emailsample.yaml
└── sensors
    ├── imap_sensor.py
    ├── imap_sensor.yaml
    ├── smtpd_green.py
    ├── smtp_sensor.py
    └── smtp_sensor.yaml

4 directories, 19 files

/opt/stackstorm# cat packs/email/sensors/imap_sensor.yaml 
---
class_name: "IMAPSensor"
entry_point: "imap_sensor.py"
description: "Sensor that emits triggers when e-mail message is received via IMAP"
trigger_types:
  -
    name: imap.message
    description: "An e-mail receieved via the IMAP sensor"
    payload_info:
      - uid
      - from
      - to
      - headers
      - subject
      - date
      - message_id
      - body
      - attachments
      - mailbox_metadata

The smtp_sensor.yaml file does not define enabled explicitly. I don't know what the defined behaviour would be in this case. If the st2 core default values are used in the case the database doesn't have a value for enabled set yet, or if there is a value set for enabled in the database, will it be used instead of the st2 core default value. Either way, there will be uses cases when the resulting behaviour is undesirable.

When talking about the pack being mirrored in the configuration directory, I'm suggesting that the structure and content be managed outside of StackStorm. The only thing StackStorm would do is continue to write pack configuration files at the root level and parse any directories/files inside the configs/pack directory. This structure would allow manual editing (dev environment), scripted editing (docker builds) and configuration management (ansible/salt/puppet/whatever) to have total control over the contents.

/opt/stackstorm# tree configs/
configs/
├── consul.yaml
├── email.yaml
├── mongodb.yaml
├── packs
│   └── email
│       └── sensors
│           ├── imap_sensor.yaml
│           └── smtp_sensor.yaml
├── powerdns.yaml
└── proxmox.yaml

3 directories, 7 files

The override files are linked by pack.class_name just as in a standard pack. Without it there's not way of correctly associating the override with the correct action/rule/sensor/alias.

# cat configs/packs/email/sensors/imap_sensor.yaml 
---
class_name: "IMAPSensor"
enabled: false

This example would only override the enabled value, but any value could be set in it.

[amanda11]

Interesting point above.
I think I agree on preference to have the overrides as pack files rather than global, but I prefer the structure that was originally suggested by @nmaludy

@nzlosh The drawback I see with mirroring the whole structure makes it a bit more difficult to do wildcard overrides. e.g. I quite often want to install a pack but disable all its sensors.

If I have to mirror the structure I have to add a disable on each sensor for example, and each time something new is added to the pack and I install a new version, I have to check what has been added - so it makes it harder to do wildcard overrides.

So I personally would prefer the configs//config.yml

with contents similar to:

sensors:
  default:
     enabled: false
   exceptions:
      - name: sensor_A
        enabled: true

But the advantage of the mirroring is that it makes it easier to do config overrides, e.g force parameters to be mandatory for example, as that would be on a per action etc. And I can see the use for that, for example if a pack supports multiple ways of doing an action, you could use the overrides to disable parameter combinations you don't want to use.

So perhaps there is a need for both the mirror but also to support wildcards (perhaps at the appropriate directory level, e.g. sensors/rules etc).

So perhaps have the mirror that @nzlosh proposed but also the ability to do wildcard at the sensor level, e.g.
configs//sensors/config.yml

with parmaeters that apply to all sensors in that pack, e.g.

enabled: false

So then priority would be:

1st. default state from st2 core
2nd. pack defintion
3rd. overrides from configs///config.yml
4th. overrides from configs///.yml

But I'm also concerned that this makes it overly complicated. It helps when I want to wildcard all sensors, but complicates the logic. So not sure if its necessary or not, and instead if should just cope with having to explicity override each sensor to disable them all?

[nzlosh]

I see what you mean about having global overrides being more convenient. I had a simple shell script in mind to do bulk changes which gives an approximation to global behaviour: e.g.

cd /opt/stackstorm/
for sensor in pack/*/sensors/*.yml
do
mkdir -p config/$(dirname $sensor)
echo "enabled: false" > config/$sensor
done

Perhaps a small Python script could be developed as a helper for managing the override settings? Or even integrated into the st2client.

st2 config override --pack=email --sensor=* key1=value1 key2=value2

This would create the config override in the right place for all sensors registered in the email pack with provided key/value pairs.

[jamesdreid]

This one is near and dear to my heart as we have been bit by the issue of sensors being enabled/disabled in different environments by a pack install. We started working on a solution to this that used a workflow to enable/disable sensors after packs were loaded and I think it may solve all these issues in a simpler, more accessible manner than processing it in the code with the global configs with pack level overrides.
I am sure there is more to work out and likely some limitations in the system/APIs that exist today, but the base pack install process could be updated to NOT register/process sensors at all by default (in line with @punkrokk suggestion to mimic the default Linux standard) but include a rule that will run a workflow whenever the "pack.install" process complete successfully. The default rule installed by the "Pack" pack would point to a simple workflow that processes any/all sensors in the pack that was just installed, reads the config for their status (enabled/disabled) and then installs and registers them using their supplied values. This simple change would mirror the existing functionality and not change any existing setups, but allow for a very complex replacement workflow to be dropped into the system with all the enabling/disabling logic coded into the workflow using Orquesta. To leverage the new, more complex workflow, the default rule used to trigger the sensor processing logic could be updated to point to whichever new workflow that is desired and it should be fairly safe (famous last words...) from modification as the default "Pack" pack is normally only updated during upgrades.

Some simple enhancements to the base workflow could be to have it read the current sensor status from the DB (check to see if an operator clicked the disable button in the GUI) and then re-install the sensor from the pack with the same state. This simple change alone would prevent a number of issues in how the system currently works and would likely be simple to implement in the workflow.

Maybe add some global variables to the "st.conf" file associated with the system in use that were included in the workflow context so those global values could be used during the processing of the sensor workflow. This would allow the SysAdmin the ability to "tag" the system as DEV/QA/PROD and using that value in the workflow to branch as needed. These values could likely be used as the basis for global overrides if the workflow was implemented correctly.

Along with the global variables inserted into the workflow context, maybe enhance the pack variables inserted into the workflow context as well so tags could be applied to Exchange packs or internally developed packs and could be processed differently in the workflow as needed (or as deemed fit by security). These values could be used as the basis for pack level overrides.

Finally, a pack of workflows could be developed and shared on the Exchange where various sensor enabling/disabling workflows could be shared across the community and provide examples/enhancements to the base workflow that would make the process more accessible to new users.

Most of what I suggest is specific to sensors but I would imagine would apply to rules/aliases as well with a similar manner assuming the APIs for each supported what is needed to process them in a workflow.

[arm4b]

I get the idea is to manage the state of pack content in the workflow level behind the pack install.
That's an interesting workaround for a single isolated use (though st2-register-content may override content from the previous st2 pack install workflow), but it's not sufficient as a serious core st2 implementation where Infrastructure as a Code is the key design approach behind the StackStorm as a tool.

With that, I'm 👍 to what @nmaludy proposed and @amanda11 raised once again.

[amanda11]

@armab What's your opinion on @nzlosh proposal of mimicing the structure as a override.

This has the benefit of making it easier to update config as well as enabled/disabled, but has the disadvantage of harder to do global overrides.

I'm quite tempted by @nzlosh's proposal @ #3973 (comment)

[arm4b]

@amanda11 I think there's too much nesting and messing with the dirs/files if we go that way with mimicking the full pack structure. It looks too verbose to me, and may also have some leftover and untracked configs in the case when pack content is changed/removed/renamed/moved. Management is more encouraged to get messy.

Another point, this duplication might lead to confusion which config to edit really for content overrides. The first obvious idea is looking at email.yaml, while it's really overrides at a deeper level:

/opt/stackstorm# tree configs/
configs/
├── email.yaml
├── packs
│   └── email
│       └── sensors
│           ├── imap_sensor.yaml
│           └── smtp_sensor.yaml

It doesn't address the case when the user wants to blacklist all sensors in the system, but whitelist sensors only for some specific packs.

---

And so instead of that, I believe configuration like

/opt/stackstorm# tree configs/
configs/
├── _global.yaml
├── email.yaml

to define the enabled/disabled pack content globally + per pack looks balanced enough. You really want to configure things in a fewer places, instead of jumping between the dirs.

[amanda11]

I'll see if I can start working on this in the background, going to start at the pack level first, also thinking perhaps would be more clear to put the overrides into a configs/overrides sub-directory so its clear the point of them.