Introduce Dynaconf

[0snap]

📔 Description

Switch configuration framework from confuse to dynaconf. The PR updates all of Threat Bus, all plugins, all apps, all integration tests and the MISP test utils.

Docs will follow in a separate PR.

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/threatbus, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

• Craft some config files and Interactively check application startup. Test that all required values are present or the validation should complain otherwise.

[0snap]

@satta ping: we ditch confuse in favor of the superior config framework dynaconf. This might require re-packaging for Debian.

[satta]

@satta ping: we ditch confuse in favor of the superior config framework dynaconf. This might require re-packaging for Debian.

Thanks, good to know. For buster, yes. Bullseye will have python3-dynaconf (2.2.3-2). Can you do with that version or do you really need 3.X?

[0snap]

Hey @satta, I just checked the docs, and we definitely require dynaconf >= 3.1.X for two reasons:

1. According to their changelog there are breaking changes between v2 and v3, which would affect us. They have deprecated a the usage of a global structure and introduced an object-local settings management. Since we use that (in accordance to the v3 docs), the code is not compatible to dynaconf 2.X.
2. We use dynaconf Validators to check the user configurations for correctness. Validators are a new feature as of v3.1

[satta]

Got it. This will mean updating and backporting version 3 as well.
This is pretty similar to the other issue we are facing with pika. Looks like I will need to revisit the question of how to handle multiple versions of the same Python module in Debian.

[lava]

The code changes itself look good to me, but I failed to pass validation when I tried to start threatbus:

$ head -7 config.yaml.example 
logging:
  console: true
  console_verbosity: DEBUG
  file: false
  file_verbosity: DEBUG
  filename: threatbus.log

$ threatbus -c config.yaml.example 
Invalid config: combined validators failed logging.console is required in env main or logging.file is required in env main

$ THREATBUS_LOGGING_CONSOLE=true threatbus -c config.yaml.example 
Invalid config: combined validators failed logging.console is required in env main or logging.file is required in env main

It's not completely clear to me what the issue is.

[0snap]

Both things you tried are invalid. You first need to copy the file to something ending in .yaml or .yml, otherwise dynaconf does not pick it up. So when you follow the readme and cp config.yaml.example config.yaml it should work.

Second, as stated in the readme, THREATBUS_LOGGING_CONSOLE=true is invalid and should instead have two underscores.
Here's a working example that would make Threat Bus additionally log to a file:

THREATBUS_LOGGING__FILE=true threatbus -c config.yaml

[lava]

You first need to copy the file to something ending in .yaml or .yml, otherwise dynaconf does not pick it up.

Shouldn't we then add a validator that checks that the -c parameter value ends in .yaml or .yml?

[0snap]

Shouldn't we then add a validator that checks that the -c parameter value ends in .yaml or .yml?

While we lose the ability to accept toml or otherwise neat config formats, I agree that we have not yet documented the alternatives a user can pick and hence added a strict check for all application to exit early for any other file extension than yaml or yml.

[lava]

The code looks good; I didn't get to do too much local testing but imho it's fine to merge this now and fix any remaining issues as they come up, since it's a month anyways to the next release.