DEP: Remove URLVir feed and parser

feed is not available anymore
fixes #1537

CHANGELOG.md

@@ -58,6 +58,7 @@ Dropped support for Python 3.4.
 - `intelmq.bots.parsers.microsoft.parser_ctip`: Compatibility for new CTIP data format used provided by the Azure interface.
 - `intelmq.bots.parsers.cymru.parser_cap_program`: Support for `openresolver` type.
 - `intelmq.bots.parsers.github_feed.parser`: Added (PR#1481).
+- `intelmq.bots.parsers.urlvir.parser`: Removed, as the feed is discontinued (#1537).
 
 #### Experts
 - `intelmq.bots.experts.csv_converter`: Added as converter to CSV.
@@ -79,6 +80,7 @@ Dropped support for Python 3.4.
 - Added document on MISP integration possibilities.
 - Feeds:
   - Added "Full Bogons IPv6" feed.
+  - Remove discontinued URLVir Feeds (#1537).
 
 ### Packaging
 - `setup.py` do not try to install any data to `/opt/intelmq/` as the behavior is inconsistent on various systems and with `intelmqsetup` we have a tool to create the structure and files anyway.

NEWS.md

@@ -34,6 +34,10 @@ This also changes the required configuration parameters. The new required parame
 
 The previous parameters `account_name`, `account_key` and `delete` are not supported anymore.
 
+#### URLVir Feeds and Parser
+All URLVir feeds have been discontinued. The URLVir Parser has been removed.
+The `intelmqctl upgrade-config` command warns if you have these feed and the bot in use.
+
 ### Libraries
 
 ### Postgres databases

docs/Feeds.md

@@ -46,7 +46,6 @@ To add feeds to this file add them to `intelmq/etc/feeds.yaml` and then run
 - [Team Cymru](#team-cymru)
 - [Threatminer](#threatminer)
 - [Turris](#turris)
-- [URLVir](#urlvir)
 - [University of Toulouse](#university-of-toulouse)
 - [VXVault](#vxvault)
 - [ViriBack](#viriback)
@@ -1865,52 +1864,6 @@ server {
 * **Configuration Parameters:**
 
 
-# URLVir
-
-## Hosts
-
-* **Public:** yes
-* **Revision:** 2018-01-20
-* **Documentation:** http://www.urlvir.com/
-* **Description:** This feed provides FQDN's or IP addresses for Active Malicious Hosts.
-
-### Collector
-
-* **Module:** intelmq.bots.collectors.http.collector_http
-* **Configuration Parameters:**
-*  * `http_url`: `http://www.urlvir.com/export-hosts/`
-*  * `name`: `Hosts`
-*  * `provider`: `URLVir`
-*  * `rate_limit`: `129600`
-
-### Parser
-
-* **Module:** intelmq.bots.parsers.urlvir.parser
-* **Configuration Parameters:**
-
-
-## IPs
-
-* **Public:** yes
-* **Revision:** 2018-01-20
-* **Documentation:** http://www.urlvir.com/
-* **Description:** This feed provides IP addresses hosting Malware.
-
-### Collector
-
-* **Module:** intelmq.bots.collectors.http.collector_http
-* **Configuration Parameters:**
-*  * `http_url`: `http://www.urlvir.com/export-ip-addresses/`
-*  * `name`: `IPs`
-*  * `provider`: `URLVir`
-*  * `rate_limit`: `129600`
-
-### Parser
-
-* **Module:** intelmq.bots.parsers.urlvir.parser
-* **Configuration Parameters:**
-
-
 # University of Toulouse
 
 ## Blacklist

intelmq/bots/BOTS

@@ -647,11 +647,6 @@
                 "substitutions": " .net;[.]net"
             }
         },
-        "URLVir": {
-            "description": "URLVir Parser is the bot responsible to parse the Export Hosts and Export IP Addresses reports and sanitize the information.",
-            "module": "intelmq.bots.parsers.urlvir.parser",
-            "parameters": {}
-        },
         "VXVault": {
             "description": "VXVault Parser is the bot responsible to parse the report and sanitize the information.",
             "module": "intelmq.bots.parsers.vxvault.parser",

intelmq/etc/feeds.yaml

@@ -181,42 +181,6 @@ providers:
       revision: 2018-02-06
       documentation: https://www.openphish.com/phishing_feeds.html
       public: no
-  URLVir:
-    Hosts:
-      description: This feed provides FQDN's or IP addresses for Active Malicious
-        Hosts.
-      additional_information:
-      bots:
-        collector:
-          module: intelmq.bots.collectors.http.collector_http
-          parameters:
-            http_url: http://www.urlvir.com/export-hosts/
-            rate_limit: 129600
-            name: __FEED__
-            provider: __PROVIDER__
-        parser:
-          module: intelmq.bots.parsers.urlvir.parser
-          parameters:
-      revision: 2018-01-20
-      documentation: http://www.urlvir.com/
-      public: yes
-    IPs:
-      description: This feed provides IP addresses hosting Malware.
-      additional_information:
-      bots:
-        collector:
-          module: intelmq.bots.collectors.http.collector_http
-          parameters:
-            http_url: http://www.urlvir.com/export-ip-addresses/
-            rate_limit: 129600
-            name: __FEED__
-            provider: __PROVIDER__
-        parser:
-          module: intelmq.bots.parsers.urlvir.parser
-          parameters:
-      revision: 2018-01-20
-      documentation: http://www.urlvir.com/
-      public: yes
   Netlab 360:
     Mirai Scanner:
       description: 'This feed provides IP addresses which actively scan for vulnerable

intelmq/lib/upgrades.py

@@ -24,6 +24,7 @@
            'v213_feed_changes',
            'v220_configuration_1',
            'v220_azure_collector',
+           'v220_feed_changes',
            ]
 
 
@@ -435,6 +436,32 @@ def v213_feed_changes(defaults, runtime, harmonization, dry_run):
     return messages + ' Remove affected bots yourself.' if messages else changed, defaults, runtime, harmonization
 
 
+def v220_feed_changes(defaults, runtime, harmonization, dry_run):
+    """
+    Migrates feed configuration for changed feed parameters.
+    """
+    found_urlvir_feed = []
+    found_urlvir_parser = []
+    changed = None
+    messages = []
+    for bot_id, bot in runtime.items():
+        if bot["module"] == "intelmq.bots.collectors.http.collector_http":
+            if "http_url" not in bot["parameters"]:
+                continue
+            if bot["parameters"]["http_url"].startswith("http://www.urlvir.com/export-"):
+                found_urlvir_feed.append(bot_id)
+        elif bot['module'] == "intelmq.bots.parsers.urlvir.parser":
+            found_urlvir_parser.append(bot_id)
+    if found_urlvir_feed:
+        messages.append('A discontinued feed "URLVir" has been found '
+                        'as bot %s.' % ', '.join(sorted(found_urlvir_feed)))
+    if found_urlvir_parser:
+        messages.append('The removed parser "URLVir" has been found '
+                        'as bot %s.' % ', '.join(sorted(found_urlvir_parser)))
+    messages = ' '.join(messages)
+    return messages + ' Remove affected bots yourself.' if messages else changed, defaults, runtime, harmonization
+
+
 UPGRADES = OrderedDict([
     ((1, 0, 0, 'dev7'), (v100_dev7_modify_syntax, )),
     ((1, 1, 0), (v110_shadowserver_feednames, v110_deprecations)),
@@ -449,7 +476,7 @@ def v213_feed_changes(defaults, runtime, harmonization, dry_run):
     ((2, 1, 2), ()),
     ((2, 1, 3), (v213_deprecations, v213_feed_changes)),
     ((2, 1, 4), ()),
-    ((2, 2, 0), (v220_configuration_1, v220_azure_collector)),
+    ((2, 2, 0), (v220_configuration_1, v220_azure_collector, v220_feed_changes)),
 ])
 
 ALWAYS = (harmonization, )

intelmq/tests/lib/test_upgrades.py

@@ -332,6 +332,19 @@
     "module": "intelmq.bots.parsers.nothink.parser",
 },
 }
+V220_FEED = {
+"urlvir-hosts-collector": {
+    "group": "Collector",
+    "module": "intelmq.bots.collectors.http.collector_http",
+    "parameters": {
+        "http_url": "http://www.urlvir.com/export-hosts/",
+    },
+},
+"urlvir-parser": {
+    "group": "Parser",
+    "module": "intelmq.bots.parsers.urlvir.parser",
+},
+}
 
 
 def generate_function(function):
@@ -447,6 +460,17 @@ def test_v213_feed_changes(self):
                          result[0])
         self.assertEqual(V213_FEED, result[2])
 
+    def test_v220_feed_changes(self):
+        """ Test v213_feed_changes """
+        result = upgrades.v220_feed_changes({}, V220_FEED, {}, False)
+        self.assertEqual('A discontinued feed "URLVir" has been found '
+                         'as bot urlvir-hosts-collector. '
+                         'The removed parser "URLVir" has been found '
+                         'as bot urlvir-parser. '
+                         'Remove affected bots yourself.',
+                         result[0])
+        self.assertEqual(V220_FEED, result[2])
+
 
 for name in upgrades.__all__:
     setattr(TestUpgradeLib, 'test_function_%s' % name,