Version 4.0.0 (final)

.n6-version

@@ -1 +1 @@
-3.12.0b2
+4.0.0

CHANGELOG.md

@@ -1,63 +1,77 @@
 # Changelog
 
-Starting with the 3.0.0 release, all notable changes to the
+Starting with the 4.0.0 release, all notable changes to the
 [code of _n6_](https://github.com/CERT-Polska/n6) are continuously
 documented here.
 
-The format of this document is based, to much extent, on
+Some features of this document's format are based on
 [Keep a Changelog](https://keepachangelog.com/).
 
-## [3.12.0-beta2] - 2022-08-30
 
-[TBD...]
+## [4.0.0] - 2023-06-03
 
-## [3.0.1] - 2021-12-03
+**This release is a big milestone.**
 
-### Changes and Fixes
+Among others:
 
-- [docs] A bunch of fixes and improvements regarding the documentation,
-  including major changes to its structure, layout and styling.
+- the *n6 Portal* gained support for OpenID-Connect-based *single
+  sign-on* (SSO) authentication;
+
+- the *n6 Stream API* (STOMP-based) now supports authentication based on
+  API keys (the same ones that have already been accepted by the *n6 REST
+  API*); the new mechanism replaces the previously used one (based on
+  X.509 client certificates);
+
+- added a bunch of new components which obtain and process security
+  data from external sources: 26 *collectors* and 86 *parsers*; now,
+  in total, we have in `N6DataSources` 35 *collectors* and 91 *parsers*;
+
+- got rid of the legacy, *Python-2-only*, stuff (most of which were
+  Python 2 versions *collectors* and *parsers*) that used to reside in
+  `N6Core` and `N6CoreLib` (the *Python-2-only* variants of `N6Lib` and
+  `N6SDK` have also been removed); note that the components related to
+  active data sources has been migrated to Python 3 (8 *collectors* and
+  7 *parsers* -- now they reside in `N6DataSources`); therefore, now *n6*
+  is *Python-3-only* (finally!);
+
+- significant optimizations have been accomplished: certain kinds of data
+  queries (via the *n6 REST API* or *n6 Portal*) are much faster and
+  `n6aggretator`'s memory consumption has been considerably reduced;
+
+- also, many minor improvements, a bunch of fixes, some refactorization
+  and various cleanups have been made...
+
+Note that some of the changes are *not* backwards-compatible.
 
-- [setup] `do_setup.py`: regarding the default value of the option
-  `--additional-packages` under Python 3, the version of the `mkdocs`
-  package has been pinned (`1.2.3`), and the `mkdocs-material` package
-  (providing the `material` docs theme) has been added (and its version is
-  also pinned: `8.0.3`); regarding the same under Python 2, the `mkdocs`
-  package has been removed.
 
 ## [3.0.0] - 2021-12-01
 
 **This release is a big milestone.** It includes, among others:
 
 - migration to Python 3
-- in the _n6_ data pipeline infrastructure: optional integration
+
+- in the *n6* data pipeline infrastructure: optional integration
   with [IntelMQ](https://github.com/certtools/intelmq)
-- in the _n6 Portal:_ a new frontend (implemented using
+
+- in the *n6 Portal:* a new frontend (implemented using
   [React](https://reactjs.org/)), two-factor authentication
   (based on [TOTP](https://datatracker.ietf.org/doc/html/rfc6238)),
   user's/organization's own data management (including config update
   and password reset forms, with related e-mail notices), and other
   goodies...
-- in the _n6 REST API:_ API-key-based authentication
+
+- in the *n6 REST API:* API-key-based authentication
+
 - and many, many more improvements, a bunch of fixes, as well as
   some refactorization, removals and cleanups...
 
-Beware that many of the changes are _not_ backwards-compatible.
+Note that many of the changes are *not* backwards-compatible.
 
-Note that most of the main elements of _n6_ -- namely:
+Also, note that most of the main elements of *n6* -- namely:
 `N6DataPipeline`, `N6DataSources`, `N6Portal`, `N6RestApi`,
 `N6AdminPanel`, `N6BrokerAuthApi`, `N6Lib` and `N6SDK` -- are now
-_Python-3-only_ (more precisely: are compatible with CPython 3.9).
-
-The legacy, _Python-2-only_ stuff -- most of which are _collectors_ and
-_parsers_ (external-data-sources-related components) -- reside in
-`N6Core` and `N6CoreLib`; the collectors and parsers placed in `N6Core`,
-if related to non-obsolete external data sources, will be gradually
-migrated to _Python-3-only_ `N6DataSources` (so that, finally, we will
-be able to rid of `N6Core` and `N6CoreLib`). There are also
-_Python-2-only_ variants of `N6Lib` and `N6SDK`: `N6Lib-py2` and
-`N6SDK-py2` (needed only as dependencies of `N6Core`/`N6CoreLib`).
-
-[3.12.0-beta2]: https://github.com/CERT-Polska/n6/compare/v3.0.1...v3.12.0b2
-[3.0.1]: https://github.com/CERT-Polska/n6/compare/v3.0.0...v3.0.1
+*Python-3-only* (more precisely: are compatible with CPython 3.9).
+
+
+[4.0.0]: https://github.com/CERT-Polska/n6/compare/v3.0.0...v4.0.0
 [3.0.0]: https://github.com/CERT-Polska/n6/compare/v2.0.6a2-dev1...v3.0.0

N6AdminPanel/n6adminpanel/app.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2018-2022 NASK. All rights reserved.
+# Copyright (c) 2018-2023 NASK. All rights reserved.
 
 import ast
 import json
@@ -138,7 +138,10 @@
     is_seq,
 )
 from n6lib.common_helpers import ThreadLocalNamespace
-from n6lib.config import ConfigMixin
+from n6lib.config import (
+    Config,
+    ConfigMixin,
+)
 from n6lib.const import (
     WSGI_SSL_ORG_ID_FIELD,
     WSGI_SSL_USER_ID_FIELD,
@@ -1350,7 +1353,16 @@ class AdminPanel(ConfigMixin):
         baddomains_auth_token_cache_dir =
     '''
 
-    config_filename_regex = re.compile(
+    @classmethod
+    def ensure_admin_panel_specific_config_filename_regex_will_always_be_used(cls):
+        cm = Config.overriden_init_defaults(
+            config_filename_regex=cls._admin_panel_specific_config_filename_regex)
+        type(cm).__enter__(cm)
+        # Let's keep `cm` alive (if it was garbage-collected,
+        # its `__exit__()` would be executed immediately):
+        cls.__config_overriden_init_defaults_cm = cm
+
+    _admin_panel_specific_config_filename_regex = re.compile(
         # Explaining it roughly: the filename must include the "admin"
         # and "panel" words, and must end with ".conf".
 
@@ -1372,8 +1384,6 @@ class AdminPanel(ConfigMixin):
         r'.*'
         r'\.conf\Z')    # the filename must end with ".conf" (lowercase).
 
-    engine_config_prefix = ''
-
     table_views = [
         (Org, OrgView),
         (OrgConfigUpdateRequest, OrgConfigUpdateRequestView),
@@ -1560,6 +1570,7 @@ def get_app():
         A flask.app.Flask instance.
     """
     with logging_configured():
+        AdminPanel.ensure_admin_panel_specific_config_filename_regex_will_always_be_used()
         monkey_patch_flask_admin()
         engine = SQLAuthDBConfigMixin().engine
         admin_panel = AdminPanel(engine)

N6AdminPanel/n6adminpanel/tests/test_config_filename_regex.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2022 NASK. All rights reserved.
+# Copyright (c) 2022-2023 NASK. All rights reserved.
 
 import unittest
 
@@ -27,7 +27,7 @@ class TestAdminPanel_config_filename_regex(unittest.TestCase):
         '1Admin2paneL3.conf',
     )
     def test_config_filename_regex_matches(self, filename):
-        assert AdminPanel.config_filename_regex.search(filename)
+        assert AdminPanel._admin_panel_specific_config_filename_regex.search(filename)
 
     @foreach(
         'panel_admin.conf',
@@ -48,4 +48,4 @@ def test_config_filename_regex_matches(self, filename):
         'admin_panel.spam',
     )
     def test_config_filename_regex_does_not_match(self, filename):
-        assert not AdminPanel.config_filename_regex.search(filename)
+        assert not AdminPanel._admin_panel_specific_config_filename_regex.search(filename)

N6BrokerAuthApi/development_stream_api.ini

@@ -41,15 +41,26 @@ auth_db.ssl_key = /some/path/to/private/ClientCertificateKeyFile.pem
 
 broker_auth_api.auth_manager_maker_class = n6brokerauthapi.auth_stream_api.StreamApiBrokerAuthManagerMaker
 
-## `n6brokerauthapi.auth_stream_api.StreamApiBrokerAuthManagerMaker`-specific configuration
-## (to be uncommented and adjusted if defaults are not sufficient)
-#stream_api_broker_auth.push_exchange_name = _push
-#stream_api_broker_auth.privileged_component_logins = rabbit-inner  ; possibly many comma-separated
-#stream_api_broker_auth.autogenerated_queue_prefix = stomp
+# Important: the value of the following option has to be identical
+# to the value of the `api_key_based_auth.server_secret` option in
+# the *n6* Portal API's configuration. If not, users will be unable to
+# authenticate to *n6* Stream API with any API keys obtained/generated
+# via *n6* Portal).
+#
+# Important: the value below is only an example, and -- for security
+# purposes -- you are strongly advised not to use it in a production
+# environment.
+stream_api_broker_auth.server_secret = INSECURE EXAMPLE VALUE THAT MUST BE REPLACED
+
+# The rest of the
+# `n6brokerauthapi.auth_stream_api.StreamApiBrokerAuthManagerMaker`-specific
+# configuration (to be uncommented and adjusted if defaults are not sufficient):
+;stream_api_broker_auth.push_exchange_name = _push
+;stream_api_broker_auth.autogenerated_queue_prefix = stomp
 
 
 ###
-# server configuration
+# dev/test server configuration
 ###
 
 [server:main]