Merge tag '2.2.0'

2.2.0 Feature release This IntelMQ Manager version requires IntelMQ >= 2.2.0. ### Backend - `config`: Get file paths from `intelmctl debug --get-paths` if possible and fall back to hard-coded paths otherwise. Thereby environment variables influencing the paths are respected (certtools#193). ### Pages #### About - Show output of `intelmqctl debug`. ### Documentation - Update release from intelmq's release documentation. - Update Installation documentation: Fix & update dependencies and supported operating systems. ### Packaging - Update default `positions.conf` to the default runtime/pipeline configuration of intelmq >= 2.1.1. ### Known issues * Missing CSRF protection (certtools#111). * Graph jumps around on "Add edge" (certtools#148). * wrong error message for new bots with existing ID (certtools#152). * `ALLOWED_PATH=` violates CSP (certtools#183). * Monitor page: Automatic log refresh reset log page to first one (certtools#190).
CZ-NIC · Jun 24, 2020 · 6c467c4 · 6c467c4
2 parents e5bc7ed + 776d091
commit 6c467c4
Show file tree

Hide file tree

Showing 13 changed files with 198 additions and 84 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,32 @@ CHANGELOG
 =========
 
 
+2.2.0 (2020-06-23)
+------------------
+This IntelMQ Manager version requires IntelMQ >= 2.2.0.
+
+### Backend
+- `config`: Get file paths from `intelmctl debug --get-paths` if possible and fall back to hard-coded paths otherwise. Thereby environment variables influencing the paths are respected (#193).
+
+### Pages
+#### About
+- Show output of `intelmqctl debug`.
+
+### Documentation
+- Update release from intelmq's release documentation.
+- Update Installation documentation: Fix & update dependencies and supported operating systems.
+
+### Packaging
+- Update default `positions.conf` to the default runtime/pipeline configuration of intelmq >= 2.1.1.
+
+### Known issues
+* Missing CSRF protection (#111).
+* Graph jumps around on "Add edge" (#148).
+* wrong error message for new bots with existing ID (#152).
+* `ALLOWED_PATH=` violates CSP (#183).
+* Monitor page: Automatic log refresh reset log page to first one (#190).
+
+
 2.1.1 (2020-04-27)
 ------------------
 Bernhard Herzog (Intevation) discovered that the backend incorrectly handled messages given by user-input in the "send" functionality of the Inspect-tool of the Monitor component. An attacker with access to the IntelMQ Manager could possibly use this issue to execute arbitrary code with the privileges of the webserver.

diff --git a/NEWS.md b/NEWS.md
@@ -3,6 +3,16 @@ NEWS
 
 See the changelog for a full list of changes.
 
+
+2.2.0 (2020-06-23)
+------------------
+This IntelMQ Manager version requires IntelMQ >= 2.2.0.
+
+### Paths
+The paths for configuration files are queried from the IntelMQ Core.
+Thus, the environment variables `INTELMQ_ROOT_DIR` and `INTELMQ_PATHS_NO_OPT`/`INTELMQ_PATHS_OPT` are now respected.
+
+
 2.1.1 (2020-04-27)
 ------------------
 

diff --git a/SECURITY.md b/SECURITY.md
@@ -4,7 +4,7 @@ IntelMQ Manager Security Notes
 Running IntelMQ Manager securely
 --------------------------------
 
-Please read the installation documentation at `docs/INSTALL.md`, section "Security considerations".
+Please read the installation documentation at [`docs/INSTALL.md`, section "Security considerations"](docs/INSTALL.md#security-considerations).
 
 Found a security issue?
 -----------------------

diff --git a/debian/changelog b/debian/changelog
@@ -1,3 +1,15 @@
+intelmq-manager (2.2.0-1) stable; urgency=medium
+
+  * Update to 2.2.0.
+
+ -- Sebastian Wagner <wagner@cert.at>  Tue, 23 Jun 2020 10:46:42 +0200
+
+intelmq-manager (2.2.0~alpha1-1) unstable; urgency=medium
+
+  * Update to 2.2.0 alpha 1.
+
+ -- Sebastian Wagner <wagner@cert.at>  Mon, 16 Mar 2020 16:55:47 +0100
+
 intelmq-manager (2.1.1-1) stable; urgency=critical
 
   * Update to version 2.1.1.

diff --git a/debian/control b/debian/control
@@ -10,7 +10,7 @@ Package: intelmq-manager
 Architecture: all
 Depends: ${misc:Depends}, apache2, apache2-utils,
   libapache2-mod-php | libapache2-mod-php5, sudo,
-  adduser, intelmq (>= 1.0.1)
+  adduser, intelmq (>= 2.2.0)
 Description: Graphical interface to manage configurations for the IntelMQ framework.
  A IntelMQ configuration is a set of config files which describe which
  bots and processing steps should be run in which order. It is similar

diff --git a/debian/patches/fix-paths.patch b/debian/patches/fix-paths.patch
@@ -1,12 +1,26 @@
 Description: Fix paths for LSB
 Author: Sebastian Wagner <wagner@cert.at>
-Last-Update: 2020-04-27
+Last-Update: 2020-04-28
 ---
 This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
---- a/intelmq-manager/php/config.php	2020-04-27 17:37:51.310011401 +0200
-+++ b/intelmq-manager/php/config.php	2020-04-27 21:28:30.432810932 +0200
-@@ -1,18 +1,18 @@
- <?php
+--- a/intelmq-manager/php/config.php	2020-04-28 15:17:27.858979961 +0200
++++ b/intelmq-manager/php/config.php	2020-04-28 15:18:38.985712752 +0200
+@@ -2,7 +2,7 @@
+     $backwardscompat = getenv("INTELMQ_MANGER_CONTROLLER_CMD");
+     $controller = getenv("INTELMQ_MANAGER_CONTROLLER_CMD");
+     if(!($c = $controller ? $controller : $backwardscompat)) {
+-        $c = "sudo -u intelmq /usr/local/bin/intelmqctl";
++        $c = "sudo -u intelmq /usr/bin/intelmqctl";
+     }
+
+     // to be displayed so that user can replicate
+@@ -16,17 +16,17 @@
+     $BOT_ID_REJECT_REGEX = '/[^A-Za-z0-9.-]/';
+     $VERSION = "2.2.0";
+
+-    $ALLOWED_PATH = "/opt/intelmq/var/lib/bots/"; // PHP is allowed to fetch the config files from the current location in order to display bot configurations.
++    $ALLOWED_PATH = "/var/lib/intelmq/bots/"; // PHP is allowed to fetch the config files from the current location in order to display bot configurations.
+     $FILESIZE_THRESHOLD = 2000; // config files under this size gets loaded automatically; otherwise a link is generated
 
      $FILES = array(
 -        'bots' 		=> '/opt/intelmq/etc/BOTS',
@@ -24,19 +38,5 @@ This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
 +        'system' 	=> '/etc/intelmq/system.conf',
 +        'positions' => '/etc/intelmq/manager/positions.conf',
      );
-     $backwardscompat = getenv("INTELMQ_MANGER_CONTROLLER_CMD");
-     $controller = getenv("INTELMQ_MANAGER_CONTROLLER_CMD");
-     if(!($c = $controller ? $controller : $backwardscompat)) {
--        $c = "sudo -u intelmq /usr/local/bin/intelmqctl";
-+        $c = "sudo -u intelmq /usr/bin/intelmqctl";
-     }
-
-     // to be displayed so that user can replicate
-@@ -26,6 +26,6 @@
-     $BOT_ID_REJECT_REGEX = '/[^A-Za-z0-9.-]/';
-     $VERSION = "2.1.1";
-
--    $ALLOWED_PATH = "/opt/intelmq/var/lib/bots/"; // PHP is allowed to fetch the config files from the current location in order to display bot configurations.
-+    $ALLOWED_PATH = "/var/lib/intelmq/bots/"; // PHP is allowed to fetch the config files from the current location in order to display bot configurations.
-     $FILESIZE_THRESHOLD = 2000; // config files under this size gets loaded automatically; otherwise a link is generated
- ?>
+     # get paths from intelmqctl directly if it works
+     $proc = proc_open($c . "--type json debug --get-paths", [
diff --git a/debian/positions.conf b/debian/positions.conf
@@ -1,9 +1,9 @@
 {
-    "abusech-domain-parser": {
+    "feodo-tracker-browse-parser": {
         "x": -304,
         "y": 250
     },
-    "abusech-feodo-domains-collector": {
+    "feodo-tracker-browse-collector": {
         "x": -508,
         "y": 282
     },
@@ -58,5 +58,9 @@
     "url2fqdn-expert": {
         "x": 275,
         "y": -116
+    },
+    "settings": {
+        "physics": false,
+        "live": true
     }
-}
+}
diff --git a/docs/INSTALL.md b/docs/INSTALL.md
@@ -29,82 +29,67 @@ The following instructions assume the following requirements:
 * a supported operating system
 
 Supported and recommended operating systems are:
-* Debian 8, 9, 10
-* OpenSUSE Leap 42.2, 15
-* Ubuntu: 14.04, 16.04, 18.04 LTS
+* Debian 9, 10
+* Fedora 30, 31, 32
+* OpenSUSE Leap 15.1, 15.2
+* Ubuntu: 16.04, 18.04, 20.04
 
 Partly supported are:
-* RHEL 7
 * CentOS 7
+* RHEL 7
 See [Notes on CentOS / RHEL](#notes-on-centos--rhel)
 
 # Install Dependencies
 
 If you are using native packages, you can simply skip this section as all dependencies are installed automatically.
 
-## Ubuntu 14.04 / Debian 8
-
-```bash
-apt-get install git apache2 php5 libapache2-mod-php5
-```
-
-## Ubuntu 16.04 / Debian 9
-
-```bash
-apt-get install git apache2 php libapache2-mod-php7.0
-```
 
-## Ubuntu 18.04
+## Debian / Ubuntu
 
 ```bash
-apt-get install git apache2 php libapache2-mod-php7.2
+apt-get install git libapache2-mod-php php-json
 ```
 
-## Debian 10
+## CentOS / RHEL
 
 ```bash
-apt install libapache2-mod-php7.3
+yum install epel-release
+yum install git httpd httpd-tools php
 ```
 
-## CentOS 7
+## Fedora
 
 ```bash
-yum install epel-release
-yum install git httpd httpd-tools php
-```
+dnf install git httpd php php-common php-json
 
-## openSUSE Leap 15.1
+## openSUSE
 
 ```bash
-zypper install git apache2 apache2-utils apache2-mod_php7
+zypper install git apache2 apache2-utils apache2-mod_php php-json
 ```
 
 # Installation
 
 ## Native packages
 
-Get the install instructions for your operating system here:
-https://software.opensuse.org/download.html?project=home%3Asebix%3Aintelmq&package=intelmq-manager
+This is only recommended if you also installed intelmq itself with packages.
+As you already have the repository configured, you can install the package called `intelmq-manager` using your operating system's package manager.
+Complete install instructions for your operating system can be found here:
+https://software.opensuse.org/download.html?project=home:sebix:intelmq&package=intelmq-manager
 
 Currently, these operating systems are supported by the packages:
 * CentOS 7, install `epel-release` first
 * RHEL 7, install `epel-release` first
-* Debian 8 and Debian 9 (install `php-json` too)
-* Fedora 25, 26 and Rawhide
-* openSUSE Leap 42.2 and Leap 42.3
+* Debian 9, 10
+* Fedora 30, 31, 32
+* openSUSE Leap 15.1, 15.2
 * openSUSE Tumbleweed
-* Ubuntu 16.04 and Ubuntu 17.04, install `php-json` too
+* Ubuntu 16.04, 18.04, 19.10, 20.04
 
-The package is always called `intelmq-manager`.
-
-For Debian and Ubuntu you need to make the configuration files writable by the group:
-```bash
-chmod 664 /etc/intelmq/*.conf /etc/intelmq/manager/positions.conf
-```
 
 ## Manually
 
-Clone the repository and copy the files in the subfolder `intelmq-manager` to the webserver directory (can also be `/srv/www/htdocs/` depending on the used system):
+Clone the repository using git and copy the files in the subfolder `intelmq-manager` to the webserver directory (can also be `/srv/www/htdocs/` depending on the used system):
 ```bash
 git clone https://github.com/certtools/intelmq-manager.git /tmp/intelmq-manager
 cp -R /tmp/intelmq-manager/intelmq-manager/* /var/www/html/
@@ -135,6 +120,8 @@ Also, stopping bots does currently not work, see also https://github.com/certtoo
 
 If you can help to fix these issues, please join us!
 
+For RHEL, the packages of CentOS may work better than those for RHEL as there are issues building the packages for RHEL. Help on RHEL is appreciated.
+
 # Security considerations
 
 **Never ever run intelmq-manager on a public webserver without SSL and proper authentication**.
@@ -143,7 +130,7 @@ The way the current version is written, anyone can send a POST request and chang
 
 Therefore you will need authentication and SSL.
 
-Use IntelMQ Manager only from a browser that can only access internal, trusted sites. (Because CSRF development is under way, see [#111](https://github.com/certtools/intelmq-manager/issues/111)).
+Use IntelMQ Manager **only from a browser that can only access internal, trusted sites** (Due to CSRF, development of a fix is under way, see [#111](https://github.com/certtools/intelmq-manager/issues/111)).
 
 In addition, intelmq currently stores plaintext passwords in its configuration files. These can be read via intelmq-manager.
 

diff --git a/docs/Release.md b/docs/Release.md
@@ -1,9 +1,22 @@
 # Release procedure
 
-Make sure the current state is really final ;)
-You can test most of the steps described here locally before doing it real.
+**Table of Contents:**
+- [Documentation](#documentation)
+- [Commit, push, review and merge](#commit-push-review-and-merge)
+- [Tag and release](#tag-and-release)
+- [Packages](#packages)
+- [Announcements](#announcements)
+- [Prepare new version](#prepare-new-version)
 
-Assumption: You are working on branch maintenance, the next version is a bug fix release. For feature releaese it is slightly different.
+
+General assumption: You are working on branch maintenance, the next version is a bug fix release. For feature releases it is slightly different.
+
+## Check before
+
+ * Make sure the current state is really final ;)
+   You can test most of the steps described here locally before doing it real.
+ * Close the milestone on GitHub and move any open issues to the next one.
+ * `docs/INSTALL.md`: Update supported operating systems.
 
 ## Documentation
 
@@ -12,14 +25,17 @@ Assumption: You are working on branch maintenance, the next version is a bug fix
  * `intelmq-manager/php/config.php` and `debian/patches/fix-paths.patch`: Update the version.
  * `debian/changelog`: Insert a new section for the new version with the tool `dch`.
 
-## Commit
-Commit your changes, the message should start with `REL: `. Push and create a pull request from maintenance to master. Someone else should review the changes. Eventually fix them, make sure the `REL: ` is the last commit, you can also push that one at last, after the reviews.
+## Commit, push, review and merge
+
+Commit your changes in a separate branch, the final commit's message should start with `REL: `. Push and create a pull request to maintenance and after that from maintenance to master. Someone else should review the changes. Eventually fix them, make sure the `REL: ` is the last commit, you can also push that one at last, after the reviews.
+
+Why a separate branch? Because if problems show up, you can still force-push to that one, keeping the release commit the latest one.
 
 ## Tag and release
 
 Tag the commit with `git tag -s version HEAD`, merge it into master, push the both branches *and* the tag. The tag is just `a.b.c`, not prefixed with `v` (that was necessary only with SVN a long time ago...).
 
-Go to https://github.com/certtools/intelmq-manager/tags and enter the release notes (changelog) for the new tag, then it's considered a release by github.
+Go to https://github.com/certtools/intelmq-manager/tags and enter the release notes (changelog) for the new tag, then it's considered a *release* by github.
 
 ## Packages
 We are currently using the public Open Build Service instance of openSUSE: http://build.opensuse.org/project/show/home:sebix:intelmq
@@ -39,7 +55,7 @@ For bigger releases, probably also at IHAP, Twitter, etc. Ask your favorite soci
 
 ## Prepare new version
 
-Increase the version in `intelmq-manager/php/config.php` and declare it as alpha version.
+`intelmq-manager/php/config.php` and `debian/patches/fix-paths.patch`: Update the version.
 
 Add a new empty changelog section:
 

diff --git a/intelmq-manager/js/about.js b/intelmq-manager/js/about.js
@@ -13,5 +13,36 @@ function get_versions() {
             alert('error getting version');
         });
 }
+function get_debug() {
+    var section_element = document.getElementById('debugging');
+
+    $.getJSON(MANAGEMENT_SCRIPT + '?scope=debug')
+        .done(function (data) {
+            for (const section in data) {
+                section_heading = document.createElement("h3");
+                section_heading.innerHTML = section;
+                section_element.appendChild(section_heading);
+                table = document.createElement("table");
+                tbody = document.createElement("table");
+
+                for (const element of data[section]) {
+                    row = tbody.insertRow(-1);
+                    cell0 = row.insertCell(0);
+                    cell0.innerHTML = "<pre>" + element[0] + "</pre>";
+                    cell1 = row.insertCell(1);
+                    cell1.innerHTML = "<pre>" + element[1] + "</pre>";
+                }
+                table.appendChild(tbody);
+                section_element.appendChild(table);
+            }
+            $('#debugging-heading').removeClass('waiting');
+        })
+        .fail(function (jqxhr, textStatus, error) {
+            var err = textStatus + ", " + error;
+            console.error( "Request Failed: " + err );
+            alert('Error getting debugging information. Do you have IntelMQ >= 2.2.0?');
+        });
+}
 
 get_versions();
+get_debug();