diff --git a/.github/workflows/build-docs.yml b/.github/workflows/build-docs.yml index d70a1f1d4..e0cc45f54 100644 --- a/.github/workflows/build-docs.yml +++ b/.github/workflows/build-docs.yml @@ -45,6 +45,10 @@ jobs: git config user.name intelmq-bot git config user.email intelmq-bot + - name: "Fix images path" + run: | + sed -i 's|docs/static/images/|static/images/|' docs/index.md + - name: "Build docs without publishing" if: github.event_name == 'pull_request' run: | diff --git a/CHANGELOG.md b/CHANGELOG.md index e3049b58e..70e6095f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,7 @@ - `intelmq.lib.upgrades`: Replace deprecated instances of `url2fqdn` experts by the new `url` expert in runtime configuration (PR#2432 by Sebastian Wagner). - `intelmq.lib.bot`: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski). - AMQP Pipeline: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski). +- Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR#2444 by DigitalTrustCenter). ### Development - Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner). @@ -63,11 +64,15 @@ `ssl_ca_cert`, `ssl_cl_cert`, `ssl_cl_cert_key`. - Minor fixes/improvements and some refactoring (see also above: *Core*...). - `intelmq.bots.collectors.amqp`: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski). +- `intelmq.bots.collectors.shadowserver.collector_reports_api`: + - The 'json' option is no longer supported as the 'csv' option provides better performance (PR#2372 by elsif2). #### Parsers - - `intelmq.bots.parsers.netlab_360.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný) - - `intelmq.bots.parsers.webinspektor.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný) - - `intelmq.bots.parsers.sucuri.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný) +- `intelmq.bots.parsers.netlab_360.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný) +- `intelmq.bots.parsers.webinspektor.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný) +- `intelmq.bots.parsers.sucuri.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný) +- `intelmq.bots.parsers.shadowserver._config`: + - Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR#2372 by elsif2). #### Experts - `intelmq.bots.experts.jinja` (PR#2417 by Mikk Margus Möll): @@ -103,6 +108,8 @@ - Update/fix/improve the stuff related to the STOMP bots and integration with the *n6*'s Stream API (PR#2408 and PR#2414 by Jan Kaliszewski). - Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný). +- Adds warning banner if not browsing the latest version of the docs (PR#2445 by Filip Pokorný). +- Fix logo path in index.md when building the docs (PR#2445 by Filip Pokorný). ### Packaging - Add `pendulum` to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner). @@ -162,8 +169,6 @@ - added support for `Subject NOT LIKE` queries, - added support for multiple values in ticket subject queries. - `intelmq.bots.collectors.rsync`: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante). -- `intelmq.bots.collectors.shadowserver.collector_reports_api`: - - The 'json' option is no longer supported as the 'csv' option provides better performance. #### Parsers - `intelmq.bots.parsers.shadowserver._config`: diff --git a/docs/_overrides/main.html b/docs/_overrides/main.html new file mode 100644 index 000000000..1d8f68edf --- /dev/null +++ b/docs/_overrides/main.html @@ -0,0 +1,11 @@ +{#SPDX-FileCopyrightText: 2023 Filip Pokorný#} +{#SPDX-License-Identifier: AGPL-3.0-or-later#} + +{% extends "base.html" %} + +{% block outdated %} + You're not viewing the latest version. + + Click here to go to latest. + +{% endblock %} diff --git a/docs/index.md b/docs/index.md index 79c310174..cf097897c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -7,7 +7,7 @@ [![CII Badge](https://bestpractices.coreinfrastructure.org/projects/4186/badge)](https://bestpractices.coreinfrastructure.org/projects/4186/) --> -![IntelMQ](/docs/static/images/Logo_Intel_MQ.svg) +![IntelMQ](docs/static/images/Logo_Intel_MQ.svg) # Introduction diff --git a/docs/user/bots.md b/docs/user/bots.md index 8c5e49171..f83b97afb 100644 --- a/docs/user/bots.md +++ b/docs/user/bots.md @@ -2128,23 +2128,23 @@ correct mapping of the columns: 1. **Automatic report type detection** - Since IntelMQ version 2.1 the parser can detect the feed based on metadata provided by the collector. - - When processing a report, this bot takes `extra.file_name` from the report and looks in `config.py` how the report - should be parsed. If this lookup is not possible, and the `feedname` is not given as parameter, the feed cannot be - parsed. - - The field `extra.file_name` has the following structure: `%Y-%m-%d-${report_name}[-suffix].csv` where the optional - suffix can be something like `country-geo`. For example, some possible filenames - are `2019-01-01-scan_http-country-geo.csv` or `2019-01-01-scan_tftp.csv`. The important part is the `report_name`, - between the date and the suffix. Since version 2.1.2 the date in the filename is optional, so filenames - like `scan_tftp.csv` are also detected. + Since IntelMQ version 2.1 the parser can detect the feed based on metadata provided by the collector. + + When processing a report, this bot takes `extra.file_name` from the report and looks in `config.py` how the report + should be parsed. If this lookup is not possible, and the `feedname` is not given as parameter, the feed cannot be + parsed. + + The field `extra.file_name` has the following structure: `%Y-%m-%d-${report_name}[-suffix].csv` where the optional + suffix can be something like `country-geo`. For example, some possible filenames + are `2019-01-01-scan_http-country-geo.csv` or `2019-01-01-scan_tftp.csv`. The important part is the `report_name`, + between the date and the suffix. Since version 2.1.2 the date in the filename is optional, so filenames + like `scan_tftp.csv` are also detected. 2. **Fixed report type** - If the method above is not possible and for upgraded instances, the report type can be set with the `feedname` - parameter. Report type is derived from the subject of Shadowserver e-mails. A list of possible values of - the `feedname` parameter can be found in the table below in the column "Report Type". + If the method above is not possible and for upgraded instances, the report type can be set with the `feedname` + parameter. Report type is derived from the subject of Shadowserver e-mails. A list of possible values of + the `feedname` parameter can be found in the table below in the column "Report Type". **Module:** @@ -2200,6 +2200,7 @@ The schema revision history is maintained at https://github.com/The-Shadowserver auto_update: true run_mode: continuous ``` + --- ### Shodan
diff --git a/intelmq/bin/intelmqctl.py b/intelmq/bin/intelmqctl.py index 51301b1d8..2add4b82e 100644 --- a/intelmq/bin/intelmqctl.py +++ b/intelmq/bin/intelmqctl.py @@ -87,6 +87,11 @@ def __init__(self, interactive: bool = False, returntype: ReturnType = ReturnTyp self._parameters.logging_handler = 'file' self._parameters.logging_path = DEFAULT_LOGGING_PATH + try: + self._runtime_configuration = utils.load_configuration(RUNTIME_CONF_FILE) + except ValueError as exc: # pragma: no cover + self.abort(f'Error loading {RUNTIME_CONF_FILE!r}: {exc}') + # Try to get logging_level from defaults configuration, else use default (defined above) defaults_loading_exc = None try: @@ -203,11 +208,6 @@ def __init__(self, interactive: bool = False, returntype: ReturnType = ReturnTyp intelmqctl debug --get-environment-variables ''' - try: - self._runtime_configuration = utils.load_configuration(RUNTIME_CONF_FILE) - except ValueError as exc: # pragma: no cover - self.abort(f'Error loading {RUNTIME_CONF_FILE!r}: {exc}') - self._processmanagertype = getattr(self._parameters, 'process_manager', 'intelmq') if self._processmanagertype not in process_managers(): self.abort('Invalid process manager given: %r, should be one of %r.' '' % (self._processmanagertype, list(process_managers().keys()))) @@ -384,7 +384,8 @@ def __init__(self, interactive: bool = False, returntype: ReturnType = ReturnTyp ) def load_defaults_configuration(self, silent=False): - for option, value in utils.get_global_settings().items(): + global_settings = self._runtime_configuration.get('global', {}) + for option, value in global_settings.items(): setattr(self._parameters, option, value) # copied from intelmq.lib.bot, should be refactored to e.g. intelmq.lib.config diff --git a/mkdocs.yml b/mkdocs.yml index c74b2b247..80492559d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -19,6 +19,7 @@ site_dir: docs_build theme: name: material logo: static/images/Logo_Intel_MQ.svg + custom_dir: docs/_overrides palette: # Palette toggle for light mode - scheme: default