Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix OTX and Whois data adapter issues #75

Merged
merged 7 commits into from Dec 15, 2017
Merged

Fix OTX and Whois data adapter issues #75

merged 7 commits into from Dec 15, 2017

Conversation

@bernd
Copy link
Member

@bernd bernd commented Dec 7, 2017

This adds a separate OTX data adaper and uses that one for the OTX API lookup tables to fix several issues and actually makes the OTX features work.

It also fixes the Whois data adapter by adding a client with timeouts.

Notes: This needs to be cherry-picked into 2.4 once merged


ATTENTION: Run the following MongoDB shell script via mongo <filename>.js to cleanup the existing lookup table/cache/adapter entries. This is only needed if you are running with a previous 2.4-alpha/beta.

var db = connect("localhost:27017/graylog");

db.cluster_config.remove({type: "org.graylog.plugins.threatintel.migrations.V20170815111700_CreateThreatIntelLookupTables.MigrationCompleted"})
db.cluster_config.remove({type: "org.graylog.plugins.threatintel.migrations.V20170821100300_MigrateOTXAPIToken.MigrationCompleted"})

db.lut_tables.remove({name: {$in: ["otx-ip", "tor-exit-node-list", "spamhaus-drop", "abuse-ch-ransomware-domains", "whois", "abuse-ch-ransomware-ip"]}})
db.lut_caches.remove({name: {$in: ["spamhaus-e-drop-cache", "otx-ip-cache", "whois-cache", "threat-intel-uncached-adapters"]}})
db.lut_data_adapters.remove({name: {$in: ["otx-ip", "whois", "abuse-ch-ransomware-ip", "tor-exit-node", "spamhaus-drop", "abuse-ch-ransomware-domains"]}})
@bernd bernd added this to the 3.0.0 milestone Dec 7, 2017
@ghost ghost assigned bernd Dec 7, 2017
@ghost ghost added the in progress label Dec 7, 2017
@bernd bernd force-pushed the fix-issues branch 2 times, most recently from 86968f7 to 88d916b Dec 14, 2017
bernd added 4 commits Dec 7, 2017
This fixes an issue where the processing blocked because of whois socket
connections without a timeout.

- Use WhoisClient from apache commons-net and configure timeouts for the
  connection
- Introduce config values for read and connect timeout
- Add data adapter documentation
- Add metrics
@bernd bernd force-pushed the fix-issues branch from 88d916b to 81772e5 Dec 15, 2017
@bernd bernd modified the milestones: 3.0.0, 2.4.0 Dec 15, 2017
@bernd bernd changed the title [WIP] Fix issues Fix OTX and Whois data adapter issues Dec 15, 2017
@bernd bernd requested a review from kroepke Dec 15, 2017
@bernd bernd removed their assignment Dec 15, 2017
@bernd bernd added ready-for-review and removed in progress labels Dec 15, 2017
Copy link
Member

@kroepke kroepke left a comment

The OTX adapter isn't restarted when its configuration is changed on the Overview page.
This unfortunately requires manual work in

// check for changes in the configuration and bounce the corresponding adapters if something changed

If the adapter is disabled and a lookup is attempted via the table's detail page, an NPE is thrown. This might be a shortcoming of the overall system, probably nothing specific to the OTX adapter.

bernd added 3 commits Dec 15, 2017
The OTX adapter doesn't do any background checks and the API token is
also not a hard requirement.
@ghost ghost assigned bernd Dec 15, 2017
@bernd
Copy link
Member Author

@bernd bernd commented Dec 15, 2017

@kroepke Good catch!

I removed the config setting to disable the OTX adaper because it doesn't need to be disabled/enabled. There is no background data fetching and also the API key is not strictly required.

Copy link
Member

@kroepke kroepke left a comment

With the latest changes it works for me, too.

@kroepke kroepke merged commit 799c6f6 into master Dec 15, 2017
2 checks passed
2 checks passed
graylog-project/pr Jenkins build graylog-project-pr-snapshot 833 has succeeded
Details
license/cla Contributor License Agreement is signed.
Details
@kroepke kroepke deleted the fix-issues branch Dec 15, 2017
@ghost ghost removed the ready-for-review label Dec 15, 2017
kroepke added a commit that referenced this pull request Dec 15, 2017
* Add OTX data adapter and use it for the OTX lookup tables

* Do not modify method parameter in Domain#prepareDomain()

* Remove outdated OTX API key note from ThreatIntelPluginConfig

* Fix Whois data adapter implementation

This fixes an issue where the processing blocked because of whois socket
connections without a timeout.

- Use WhoisClient from apache commons-net and configure timeouts for the
  connection
- Introduce config values for read and connect timeout
- Add data adapter documentation
- Add metrics

* Remove "disabled" config option for the OTX adapter

The OTX adapter doesn't do any background checks and the API token is
also not a hard requirement.

* Remove unused variables in ThreatIntelPluginConfig.jsx

* Replace HTTP host/port/scheme settings with "api_url"

(cherry picked from commit 799c6f6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants