Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feed Docu Update #1537

Closed
chappi6 opened this issue May 15, 2020 · 6 comments
Closed

Feed Docu Update #1537

chappi6 opened this issue May 15, 2020 · 6 comments
Labels
bug Indicates an unexpected problem or unintended behavior documentation: feeds About intelmq/etc/feeds.yaml
Milestone
2.1.4

Comments

@chappi6
Copy link
Contributor

chappi6 commented May 15, 2020

Hi,
I just checked the Feed Urls and found some who are offline. Please update the Feed Documentation.

Ransomware Tracker
was Status: on seems to bee offline
Status: off
https://ransomwaretracker.abuse.ch/feeds/csv/

# Ransomware Tracker has been discontinued on Dec 8th, 2019

Nothink change most of the links they are not availble any more:
http://www.nothink.org/honeypot_dns_attacks.txt
http://www.nothink.org/blacklist/blacklist_snmp_day.txt
http://www.nothink.org/blacklist/blacklist_ssh_day.txt
http://www.nothink.org/blacklist/blacklist_telnet_day.txt
No webpage was found for the web address: http://www.nothink.org/honeypot_dns_attacks.txt

But there are a bunch of new Links available under http://www.nothink.org/

URLVir
I didnt got any response just a blank page.
http://www.urlvir.com/export-hosts/

br
@ghost ghost added this to the 2.1.3 milestone May 15, 2020
@ghost ghost added bug Indicates an unexpected problem or unintended behavior documentation: feeds About intelmq/etc/feeds.yaml labels May 15, 2020
@ghost ghost self-assigned this May 15, 2020
ghost pushed a commit that referenced this issue May 15, 2020
MAINT: Remove Abuse.CH Ransomware Tracker feed
7368848 
cleanup feed in documentation, remove parser
update tests
upgrade-config extensions and tests

fix 1/3 of #1537
@ghost
Copy link

ghost commented May 15, 2020 •
edited by ghost

I just checked the Feed Urls and found some who are offline. Please update the Feed Documentation.

Thanks for checking it!

From time to time I do it too, but the sources are changing quite fast...

Ransomware Tracker
was Status: on seems to bee offline
Status: off
https://ransomwaretracker.abuse.ch/feeds/csv/

# Ransomware Tracker has been discontinued on Dec 8th, 2019

Thanks, done in 7368848

Nothink change most of the links they are not availble any more:
http://www.nothink.org/honeypot_dns_attacks.txt
http://www.nothink.org/blacklist/blacklist_snmp_day.txt
http://www.nothink.org/blacklist/blacklist_ssh_day.txt
http://www.nothink.org/blacklist/blacklist_telnet_day.txt
No webpage was found for the web address: http://www.nothink.org/honeypot_dns_attacks.txt

But there are a bunch of new Links available under http://www.nothink.org/

Is someone able to create configurations for the new feeds, e.g. with the HTML Table parser? That would help me a lot.
https://www.nothink.org/honeypots/honeypot_dns_reflection_attacks.php
https://www.nothink.org/honeypots/honeypot_snmp_attacks.php

There are two feeds with 2019 in the URL:
https://www.nothink.org/honeypots/honeypot_ssh_blacklist_2019.txt
https://www.nothink.org/honeypots/honeypot_telnet_blacklist_2019.txt
I asked the feed provider about the status.

URLVir
I didnt got any response just a blank page.
http://www.urlvir.com/export-hosts/

I asked them about the status, as I am not sure if this is maybe a (temporary) problem on their side.

@ghost
Copy link

ghost commented May 18, 2020

The nothink honeypot feeds have been discontinued for financial reasons. We need to remove them.

ghost pushed a commit that referenced this issue May 18, 2020
MAINT: Remove nothink feeds
8b8c119 
the nothink feeds are discontinued
removes the feed information and the parser
adds docs
add upgrade function

fixes 2/3 of #1537
@ghost
Copy link

ghost commented May 18, 2020

Nothink removed in 8b8c119

@chappi6
Copy link
Contributor Author

chappi6 commented May 19, 2020

Ah to bad....
Are you responsive on your cert address?

@ghost
Copy link

ghost commented May 19, 2020

Are you responsive on your cert address?

Best effort ;)

@ghost ghost modified the milestones: 2.1.3, 2.1.4 May 26, 2020
@ghost
Copy link

ghost commented May 27, 2020 •
edited by ghost

URLVir
I didnt got any response just a blank page.
http://www.urlvir.com/export-hosts/

I asked them about the status, as I am not sure if this is maybe a (temporary) problem on their side.

Got a response now: data feed is discontinued

@ghost ghost closed this as completed in f193b08 Jun 18, 2020
CSIRT-CZ pushed a commit to CZ-NIC/intelmq that referenced this issue Jun 18, 2020
Merge tag '2.2.0'
eb7f746 
2.2.0 Feature release

Dropped support for Python 3.4.

 ### Core
- `__init__`: Changes to the path-handling, see [User Guide, section _/opt and LSB paths_](docs/User-Guide.md#opt-and-lsb-paths) for more information
  - The environment variable `INTELMQ_ROOT_DIR` can be used to set custom root directories instead of `/opt/intelmq/` (certtools#805) in case of non LSB-path installations.
  - The environment variable `ROOT_DIR` can be used to set custom root directories instead of `/` (certtools#805) in case of LSB-path installations.
- `intelmq.lib.exceptions`: Added `MissingDependencyError` for show error messages about a missing library and how to install it (certtools#1471).
  - Added optional parameter `installed` to show the installed version.
  - Added optional parameter `additional_text` to show arbitrary text.
- Adding more type annotations for core libraries.
- `intelmq.lib.pipeline.Pythonlist.sleep`: Drop deprecated method.
- `intelmq.lib.utils`: `write_configuration`: Append a newline at end of configuration/file to allow proper comparisons & diffs.
- `intelmq.lib.test`: `BotTestCase` drops privileges upon initialization (certtools#1489).
- `intelmq.lib.bot`:
  - New class `OutputBot`:
    - Method `export_event` to format/export events according to the parameters given by the user.
  - `ParserBot`: New methods `parse_json_stream` and `recover_line_json_stream`.
  - `ParserBot.recover_line_json`: Fix format by adding a list around the line data.
  - `Bot.send_message`: In debugging log level, the path to which the message is sent is now logged too.

 ### Bots
- Bots with dependencies: Use of `intelmq.lib.exceptions.MissingDependencyError`.

 #### Collectors
- `intelmq.bots.collectors.misp.collector`: Deprecate parameter `misp_verify` in favor of generic parameter `http_verify_cert`.
- `intelmq.bots.collectors.tcp.collector`: Drop compatibility with Python 3.4.
- `intelmq.bots.collectors.stomp.collector`:
  - Check the stomp.py version and show an error message if it does not match.
  - For stomp.py versions `>= 5.0.0` redirect the `stomp.PrintingListener` output to debug logging.
- `intelmq.bots.collectors.microsoft.collector_azure`: Support current Python library `azure-storage-blob>= 12.0.0`, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details.
- `intelmq.bots.collectors.amqp.collector_amqp`: Require `pika` minimum version 1.0.
- `intelmq.bots.collectors.github_api.collector_github_contents_api`: Added (PR#1481).

 #### Parsers
- `intelmq.bots.parsers.autoshun.parser`: Drop compatibility with Python 3.4.
- `intelmq.bots.parsers.html_table.parser`: Drop compatibility with Python 3.4.
- `intelmq.bots.parsers.shadowserver.parser`: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544).
- `intelmq.bots.parsers.taichung.parser`:
  - Migrate to `ParserBot`.
  - Also parse geolocation information if available.
- `intelmq.bots.parsers.cymru.parser_full_bogons`:
  - Migrate to `ParserBot`.
  - Add last updated information in raw.
- `intelmq.bots.parsers.anubisnetworks.parser`: Add new parameter `use_malware_familiy_as_classification_identifier`.
- `intelmq.bots.parsers.microsoft.parser_ctip`: Compatibility for new CTIP data format used provided by the Azure interface.
- `intelmq.bots.parsers.cymru.parser_cap_program`: Support for `openresolver` type.
- `intelmq.bots.parsers.github_feed.parser`: Added (PR#1481).
- `intelmq.bots.parsers.urlvir.parser`: Removed, as the feed is discontinued (certtools#1537).

 #### Experts
- `intelmq.bots.experts.csv_converter`: Added as converter to CSV.
- `intelmq.bots.experts.misp`: Added (PR#1475).
- `intelmq.bots.experts.modify`: New parameter `maximum_matches`.

 #### Outputs
- `intelmq.bots.outputs.amqptopic`:
  - Use `OutputBot` and `export_event`.
  - Allow formatting the routing key with event data by the new parameter `format_routing_key` (boolean).
- `intelmq.bots.outputs.file`: Use `OutputBot` and `export_event`.
- `intelmq.bots.outputs.files`: Use `OutputBot` and `export_event`.
- `intelmq.bots.outputs.misp.output_feed`: Added, creates a MISP Feed (PR#1473).
- `intelmq.bots.outputs.misp.output_api`: Added, pushes to MISP via the API (PR#1506, PR#1536).
- `intelmq.bots.outputs.elasticsearch.output`: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (certtools#1513).

 ### Documentation
- Document usage of the `INTELMQ_ROOT_DIR` environment variable.
- Added document on MISP integration possibilities.
- Feeds:
  - Added "Full Bogons IPv6" feed.
  - Remove discontinued URLVir Feeds (certtools#1537).

 ### Packaging
- `setup.py` do not try to install any data to `/opt/intelmq/` as the behavior is inconsistent on various systems and with `intelmqsetup` we have a tool to create the structure and files anyway.
- `debian/rules`:
  - Provide a blank state file in the package.
- Patches:
  - Updated `fix-intelmq-paths.patch`.

 ### Tests
- Travis: Use `intelmqsetup` here too.
  - Install required build dependencies for the Debian package build test.
  - This version is no longer automatically tested on Python `<` 3.5.
  - Also run the tests on Python 3.8.
  - Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8.
- Added tests for the new bot `intelmq.bots.outputs.misp.output_feed` (certtools#1473).
- Added tests for the new bot `intelmq.bots.experts.misp.expert` (certtools#1473).
- Added tests for `intelmq.lib.exceptions`.
- Added tests for `intelmq.lib.bot.OutputBot` and `intelmq.lib.bot.OutputBot.export_event`.
- Added IPv6 tests for `intelmq.bots.parsers.cymru.parser_full_bogons`.
- Added tests for `intelmq.lib.bot.ParserBot`'s new methods `parse_json_stream` and `recover_line_json_stream`.
- `intelmq.tests.test_conf`: Set encoding to UTF-8 for reading the `feeds.yaml` file.

 ### Tools
- `intelmqctl`:
  - `upgrade-config`:
    - Allow setting the state file location with the `--state-file` parameter.
    - Do not require a second run anymore, if the state file is newly created (certtools#1491).
    - New parameter `no_backup`/`--no-backup` to skip creation of `.bak` files for state and configuration files.
  - Only require `psutil` for the `IntelMQProcessManager`, not for process manager independent calls like `upgrade-config` or `check`.
  - Add new command `debug` to output some information for debugging. Currently implemented:
    - paths
    - environment variables
  - `IntelMQController`: New argument `--no-file-logging` to disable logging to file.
  - If dropping privileges does not work, `intelmqctl` will now abort (certtools#1489).
- `intelmqsetup`:
  - Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions.
  - Call `intelmqctl upgrade-config` and add argument for the state file path (certtools#1491).
- `intelmq_generate_misp_objects_templates.py`: Tool to create a MISP object template (certtools#1470).
- `intelmqdump`: New parameter `-t` or `--truncate` to optionally give the maximum length of `raw` data to show, 0 for no truncating.

 ### Contrib
- Added `development-tools`.
- ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (certtools#1513).
- Malware Name Mapping Downloader:
  - New parameter `--mwnmp-ignore-adware`.
  - The parameter `--add-default` supports an optional parameter to define the default value.

 ### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (certtools#952).
- Corrupt dump files when interrupted during writing (certtools#870).
CSIRT-CZ pushed a commit to CZ-NIC/intelmq that referenced this issue Jun 22, 2020
Merge tag '2.2.0' into test
68269b0 
2.2.0 Feature release

Dropped support for Python 3.4.

 ### Core
- `__init__`: Changes to the path-handling, see [User Guide, section _/opt and LSB paths_](docs/User-Guide.md#opt-and-lsb-paths) for more information
  - The environment variable `INTELMQ_ROOT_DIR` can be used to set custom root directories instead of `/opt/intelmq/` (certtools#805) in case of non LSB-path installations.
  - The environment variable `ROOT_DIR` can be used to set custom root directories instead of `/` (certtools#805) in case of LSB-path installations.
- `intelmq.lib.exceptions`: Added `MissingDependencyError` for show error messages about a missing library and how to install it (certtools#1471).
  - Added optional parameter `installed` to show the installed version.
  - Added optional parameter `additional_text` to show arbitrary text.
- Adding more type annotations for core libraries.
- `intelmq.lib.pipeline.Pythonlist.sleep`: Drop deprecated method.
- `intelmq.lib.utils`: `write_configuration`: Append a newline at end of configuration/file to allow proper comparisons & diffs.
- `intelmq.lib.test`: `BotTestCase` drops privileges upon initialization (certtools#1489).
- `intelmq.lib.bot`:
  - New class `OutputBot`:
    - Method `export_event` to format/export events according to the parameters given by the user.
  - `ParserBot`: New methods `parse_json_stream` and `recover_line_json_stream`.
  - `ParserBot.recover_line_json`: Fix format by adding a list around the line data.
  - `Bot.send_message`: In debugging log level, the path to which the message is sent is now logged too.

 ### Bots
- Bots with dependencies: Use of `intelmq.lib.exceptions.MissingDependencyError`.

 #### Collectors
- `intelmq.bots.collectors.misp.collector`: Deprecate parameter `misp_verify` in favor of generic parameter `http_verify_cert`.
- `intelmq.bots.collectors.tcp.collector`: Drop compatibility with Python 3.4.
- `intelmq.bots.collectors.stomp.collector`:
  - Check the stomp.py version and show an error message if it does not match.
  - For stomp.py versions `>= 5.0.0` redirect the `stomp.PrintingListener` output to debug logging.
- `intelmq.bots.collectors.microsoft.collector_azure`: Support current Python library `azure-storage-blob>= 12.0.0`, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details.
- `intelmq.bots.collectors.amqp.collector_amqp`: Require `pika` minimum version 1.0.
- `intelmq.bots.collectors.github_api.collector_github_contents_api`: Added (PR#1481).

 #### Parsers
- `intelmq.bots.parsers.autoshun.parser`: Drop compatibility with Python 3.4.
- `intelmq.bots.parsers.html_table.parser`: Drop compatibility with Python 3.4.
- `intelmq.bots.parsers.shadowserver.parser`: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544).
- `intelmq.bots.parsers.taichung.parser`:
  - Migrate to `ParserBot`.
  - Also parse geolocation information if available.
- `intelmq.bots.parsers.cymru.parser_full_bogons`:
  - Migrate to `ParserBot`.
  - Add last updated information in raw.
- `intelmq.bots.parsers.anubisnetworks.parser`: Add new parameter `use_malware_familiy_as_classification_identifier`.
- `intelmq.bots.parsers.microsoft.parser_ctip`: Compatibility for new CTIP data format used provided by the Azure interface.
- `intelmq.bots.parsers.cymru.parser_cap_program`: Support for `openresolver` type.
- `intelmq.bots.parsers.github_feed.parser`: Added (PR#1481).
- `intelmq.bots.parsers.urlvir.parser`: Removed, as the feed is discontinued (certtools#1537).

 #### Experts
- `intelmq.bots.experts.csv_converter`: Added as converter to CSV.
- `intelmq.bots.experts.misp`: Added (PR#1475).
- `intelmq.bots.experts.modify`: New parameter `maximum_matches`.

 #### Outputs
- `intelmq.bots.outputs.amqptopic`:
  - Use `OutputBot` and `export_event`.
  - Allow formatting the routing key with event data by the new parameter `format_routing_key` (boolean).
- `intelmq.bots.outputs.file`: Use `OutputBot` and `export_event`.
- `intelmq.bots.outputs.files`: Use `OutputBot` and `export_event`.
- `intelmq.bots.outputs.misp.output_feed`: Added, creates a MISP Feed (PR#1473).
- `intelmq.bots.outputs.misp.output_api`: Added, pushes to MISP via the API (PR#1506, PR#1536).
- `intelmq.bots.outputs.elasticsearch.output`: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (certtools#1513).

 ### Documentation
- Document usage of the `INTELMQ_ROOT_DIR` environment variable.
- Added document on MISP integration possibilities.
- Feeds:
  - Added "Full Bogons IPv6" feed.
  - Remove discontinued URLVir Feeds (certtools#1537).

 ### Packaging
- `setup.py` do not try to install any data to `/opt/intelmq/` as the behavior is inconsistent on various systems and with `intelmqsetup` we have a tool to create the structure and files anyway.
- `debian/rules`:
  - Provide a blank state file in the package.
- Patches:
  - Updated `fix-intelmq-paths.patch`.

 ### Tests
- Travis: Use `intelmqsetup` here too.
  - Install required build dependencies for the Debian package build test.
  - This version is no longer automatically tested on Python `<` 3.5.
  - Also run the tests on Python 3.8.
  - Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8.
- Added tests for the new bot `intelmq.bots.outputs.misp.output_feed` (certtools#1473).
- Added tests for the new bot `intelmq.bots.experts.misp.expert` (certtools#1473).
- Added tests for `intelmq.lib.exceptions`.
- Added tests for `intelmq.lib.bot.OutputBot` and `intelmq.lib.bot.OutputBot.export_event`.
- Added IPv6 tests for `intelmq.bots.parsers.cymru.parser_full_bogons`.
- Added tests for `intelmq.lib.bot.ParserBot`'s new methods `parse_json_stream` and `recover_line_json_stream`.
- `intelmq.tests.test_conf`: Set encoding to UTF-8 for reading the `feeds.yaml` file.

 ### Tools
- `intelmqctl`:
  - `upgrade-config`:
    - Allow setting the state file location with the `--state-file` parameter.
    - Do not require a second run anymore, if the state file is newly created (certtools#1491).
    - New parameter `no_backup`/`--no-backup` to skip creation of `.bak` files for state and configuration files.
  - Only require `psutil` for the `IntelMQProcessManager`, not for process manager independent calls like `upgrade-config` or `check`.
  - Add new command `debug` to output some information for debugging. Currently implemented:
    - paths
    - environment variables
  - `IntelMQController`: New argument `--no-file-logging` to disable logging to file.
  - If dropping privileges does not work, `intelmqctl` will now abort (certtools#1489).
- `intelmqsetup`:
  - Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions.
  - Call `intelmqctl upgrade-config` and add argument for the state file path (certtools#1491).
- `intelmq_generate_misp_objects_templates.py`: Tool to create a MISP object template (certtools#1470).
- `intelmqdump`: New parameter `-t` or `--truncate` to optionally give the maximum length of `raw` data to show, 0 for no truncating.

 ### Contrib
- Added `development-tools`.
- ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (certtools#1513).
- Malware Name Mapping Downloader:
  - New parameter `--mwnmp-ignore-adware`.
  - The parameter `--add-default` supports an optional parameter to define the default value.

 ### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (certtools#952).
- Corrupt dump files when interrupted during writing (certtools#870).
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Indicates an unexpected problem or unintended behavior documentation: feeds About intelmq/etc/feeds.yaml
Projects
None yet
Milestone
2.1.4
Development

No branches or pull requests
1 participant
@chappi6