Generate multiple rules from multiple files

[proffalken]

As part of our CI/CD chain, we want to be able to run sloth against a number of files.

This works for validation, but not for the generation of alerts/rules:

[mmw@localhost adtech-argus]$ sloth validate -i src/sloth/specs/
INFO[0000] SLI plugins loaded                            plugins=0 svc=storage.FileSLIPlugin version=v0.9.0 window=30d
INFO[0000] SLO period windows loaded                     svc=alert.WindowsRepo version=v0.9.0 window=30d windows=2
INFO[0000] Validation succeeded                          slo-specs=2 version=v0.9.0 window=30d


[mmw@localhost adtech-argus]$ sloth generate -i src/sloth/specs/
error: "generate" command failed: could not read SLOs spec file data: read src/sloth/specs/: is a directory

If we can't generate the files in batches then we'll need to iterate over the files, however as our plan is to use the docker container for this it would add significant overhead to our pipeline

[slok]

Hi @proffalken!

Indeed! This was intended. As you saw, the validation has the discovery of manifests on directories, it only needs to validate and doesn't have an output, so it's safe to do it.

On the other part, the generator needs an output and we can't ask the user where to place the rules, so not allowing to do this was a way of avoiding this complexity (on the code and on the user).

Thinking about it, maybe we could manage something easy for the user with some validations, we could allow the user using a directory as output and use the same names as the input specs for the outputs, what do you think?

[proffalken]

@slok that makes sense, thanks.

FWIW, this is exactly what the bash script I've written to wrap around Sloth does.

We start out with a directory structure similar to the following:

.
└── services
    ├── api
    │   └── response_rate.slo.yaml
    ├── database
    │   └── slow_queries.slo.yaml
    └── vector
        └── log_processing.slo.yaml

and end up with the following after running the script:

.
├── prom_alerts
│   └── services
│       ├── api
│       │   └── response_rate.slo.yaml
│       ├── database
│       │   └── slow_queries.slo.yaml
│       └── vector
│           └── log_processing.slo.yaml
└── services
    ├── api
    │   └── response_rate.slo.yaml
    ├── database
    │   └── slow_queries.slo.yaml
    └── vector
        └── log_processing.slo.yaml

where prom_alerts contains the Sloth output. We then use cortextool to upload these prometheus alert rules to Grafana cloud in the next step of our CI pipeline, but we never commit the output from Sloth to Git so we can be confident we're always generating new values.

The ability to pass it an --input-dir and --output-dir when running sloth generate would be awesome, but we have a workaround for now :)

[slok]

Thanks for the explanation! Makes total sense, lets do it then 🚀