New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Legacy functions do not work out of the box after upgrade to 2.4 #57

Closed
lennartkoopmann opened this Issue Sep 27, 2017 · 0 comments

Comments

Projects
None yet
4 participants
@lennartkoopmann
Member

lennartkoopmann commented Sep 27, 2017

I am using pipeline rules from before the migration to lookup tables:

rule "Threat Intelligence lookups"
when
  has_field("src_addr") && has_field("dst_addr")
then
  set_fields(threat_intel_lookup_ip(to_string($message.src_addr), "src_addr"));
  set_fields(threat_intel_lookup_ip(to_string($message.dst_addr), "dst_addr"));
end

This leads to all lookups failing and these error messages:

2017-09-27T13:58:45.692-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-ip> does not exist
2017-09-27T13:58:45.692-05:00 WARN  [LookupTableService] Lookup table <spamhaus-drop> does not exist
2017-09-27T13:58:45.693-05:00 WARN  [LookupTableService] Lookup table <alienvault-otx-ip> does not exist
2017-09-27T13:58:45.693-05:00 WARN  [LookupTableService] Lookup table <whois> does not exist
2017-09-27T13:58:45.693-05:00 WARN  [LookupTableService] Lookup table <tor-exit-node-list> does not exist
2017-09-27T13:58:45.693-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-domains> does not exist
2017-09-27T13:58:46.712-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-ip> does not exist
2017-09-27T13:58:46.712-05:00 WARN  [LookupTableService] Lookup table <spamhaus-drop> does not exist
2017-09-27T13:58:46.712-05:00 WARN  [LookupTableService] Lookup table <alienvault-otx-ip> does not exist
2017-09-27T13:58:46.712-05:00 WARN  [LookupTableService] Lookup table <whois> does not exist
2017-09-27T13:58:46.712-05:00 WARN  [LookupTableService] Lookup table <tor-exit-node-list> does not exist
2017-09-27T13:58:46.712-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-domains> does not exist
2017-09-27T13:58:47.688-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-ip> does not exist
2017-09-27T13:58:47.688-05:00 WARN  [LookupTableService] Lookup table <spamhaus-drop> does not exist
2017-09-27T13:58:47.688-05:00 WARN  [LookupTableService] Lookup table <alienvault-otx-ip> does not exist
2017-09-27T13:58:47.688-05:00 WARN  [LookupTableService] Lookup table <whois> does not exist
2017-09-27T13:58:47.688-05:00 WARN  [LookupTableService] Lookup table <tor-exit-node-list> does not exist
2017-09-27T13:58:47.689-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-domains> does not exist
2017-09-27T13:58:48.695-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-ip> does not exist
2017-09-27T13:58:48.695-05:00 WARN  [LookupTableService] Lookup table <spamhaus-drop> does not exist
2017-09-27T13:58:48.695-05:00 WARN  [LookupTableService] Lookup table <alienvault-otx-ip> does not exist
2017-09-27T13:58:48.695-05:00 WARN  [LookupTableService] Lookup table <whois> does not exist
2017-09-27T13:58:48.695-05:00 WARN  [LookupTableService] Lookup table <tor-exit-node-list> does not exist
2017-09-27T13:58:48.695-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-domains> does not exist
2017-09-27T13:58:49.689-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-ip> does not exist
2017-09-27T13:58:49.689-05:00 WARN  [LookupTableService] Lookup table <spamhaus-drop> does not exist
2017-09-27T13:58:49.690-05:00 WARN  [LookupTableService] Lookup table <alienvault-otx-ip> does not exist
2017-09-27T13:58:49.690-05:00 WARN  [LookupTableService] Lookup table <whois> does not exist
2017-09-27T13:58:49.690-05:00 WARN  [LookupTableService] Lookup table <tor-exit-node-list> does not exist
2017-09-27T13:58:49.690-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-domains> does not exist
2017-09-27T13:58:50.691-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-ip> does not exist
2017-09-27T13:58:50.691-05:00 WARN  [LookupTableService] Lookup table <spamhaus-drop> does not exist
2017-09-27T13:58:50.691-05:00 WARN  [LookupTableService] Lookup table <alienvault-otx-ip> does not exist
2017-09-27T13:58:50.691-05:00 WARN  [LookupTableService] Lookup table <whois> does not exist
2017-09-27T13:58:50.691-05:00 WARN  [LookupTableService] Lookup table <tor-exit-node-list> does not exist
2017-09-27T13:58:50.691-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-domains> does not exist
2017-09-27T13:58:51.693-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-ip> does not exist
2017-09-27T13:58:51.693-05:00 WARN  [LookupTableService] Lookup table <spamhaus-drop> does not exist
2017-09-27T13:58:51.693-05:00 WARN  [LookupTableService] Lookup table <alienvault-otx-ip> does not exist
2017-09-27T13:58:51.693-05:00 WARN  [LookupTableService] Lookup table <whois> does not exist
2017-09-27T13:58:51.693-05:00 WARN  [LookupTableService] Lookup table <tor-exit-node-list> does not exist
2017-09-27T13:58:51.693-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-domains> does not exist
2017-09-27T13:58:52.697-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-ip> does not exist
2017-09-27T13:58:52.697-05:00 WARN  [LookupTableService] Lookup table <spamhaus-drop> does not exist
2017-09-27T13:58:52.697-05:00 WARN  [LookupTableService] Lookup table <alienvault-otx-ip> does not exist
2017-09-27T13:58:52.697-05:00 WARN  [LookupTableService] Lookup table <whois> does not exist
2017-09-27T13:58:52.697-05:00 WARN  [LookupTableService] Lookup table <tor-exit-node-list> does not exist
2017-09-27T13:58:52.697-05:00 WARN  [LookupTableService] Lookup table <abuse-ch-ransomware-domains> does not exist

I do see them in lut_tables though:

...
{
        "_id" : ObjectId("59cbf313da4e6a5f9cd778f0"),
        "title" : "Spamhaus DROP",
        "description" : "This is the lookup table for Spamhaus' DROP (Don't Route Or Peer) list, containing netblocks which are \"hijacked\" or leased by professional spam or cyber-crime operations. For more information see https://www.spamhaus.org/drop. This lookup table is used internally by Graylog's Threat Intel Plugin. Do not delete it manually.",
        "name" : "spamhaus-drop",
        "cache" : ObjectId("59cbf313da4e6a5f9cd778e7"),
        "data_adapter" : ObjectId("59cbf313da4e6a5f9cd778ed"),
        "content_pack" : "59cbf312da4e6a5f9cd778e0",
        "default_single_value" : "",
        "default_single_value_type" : "NULL",
        "default_multi_value" : "",
        "default_multi_value_type" : "NULL"
}
{
        "_id" : ObjectId("59cbf313da4e6a5f9cd778f1"),
        "title" : "abuse.ch Ransomware Domains",
        "description" : "This is the lookup table for the abuse.ch ransomware Domain Tracker, listing infrastructure by domain names which are used for ransomware. For more information see https://ransomwaretracker.abuse.ch. This lookup table is use
d internally by Graylog's Threat Intel Plugin. Do not delete it manually.",
        "name" : "abuse-ch-ransomware-domains",
        "cache" : ObjectId("59cbf313da4e6a5f9cd778e2"),
        "data_adapter" : ObjectId("59cbf313da4e6a5f9cd778ec"),
        "content_pack" : "59cbf312da4e6a5f9cd778e0",
        "default_single_value" : "",
        "default_single_value_type" : "NULL",
        "default_multi_value" : "",
        "default_multi_value_type" : "NULL"
}
{
        "_id" : ObjectId("59cbf313da4e6a5f9cd778f2"),
        "title" : "abuse.ch Ransomware IP",
        "description" : "This is the lookup table for the abuse.ch ransomware IP Tracker, listing infrastructure by IP which is used for ransomware. For more information see https://ransomwaretracker.abuse.ch. This lookup table is used internally by
 Graylog's Threat Intel Plugin. Do not delete it manually.",
        "name" : "abuse-ch-ransomware-ip",
        "cache" : ObjectId("59cbf313da4e6a5f9cd778e2"),
        "data_adapter" : ObjectId("59cbf313da4e6a5f9cd778e9"),
        "content_pack" : "59cbf312da4e6a5f9cd778e0",
        "default_single_value" : "",
        "default_single_value_type" : "NULL",
        "default_multi_value" : "",
        "default_multi_value_type" : "NULL"
}
{
        "_id" : ObjectId("59cbf313da4e6a5f9cd778f3"),
        "title" : "Tor Exit Node List",
        "description" : "This is the lookup table for the TOR (The Onion Router) Exit Node List, listing Exit Nodes of the TOR Network . This lookup table is used internally by Graylog's Threat Intel Plugin. Do not delete it manually.",
        "name" : "tor-exit-node-list",
        "cache" : ObjectId("59cbf313da4e6a5f9cd778e4"),
        "data_adapter" : ObjectId("59cbf313da4e6a5f9cd778ea"),
        "content_pack" : "59cbf312da4e6a5f9cd778e0",
        "default_single_value" : "",
        "default_single_value_type" : "NULL",
        "default_multi_value" : "",
        "default_multi_value_type" : "NULL"
}
...

@lennartkoopmann lennartkoopmann added this to the 2.4.0 milestone Sep 27, 2017

@dennisoelkers dennisoelkers self-assigned this Sep 28, 2017

dennisoelkers added a commit to Graylog2/graylog2-server that referenced this issue Sep 29, 2017

Waiting for data adapter/cache creation in bundle importer.
Before this change, lookup data adapters, caches and tables were created
sequentially during content pack import, leading to errors when lookup
tables were created before caches or data adapters were successfully
created and started. Imported lookup tables were usable only after the
server was restarted at least once after a content pack import.

After this change, the bundle importer waits for the successful creation
of data adapters/caches before creating lookup tables.

Fixes Graylog2/graylog-plugin-threatintel#57.

@wafflebot wafflebot bot added the in progress label Sep 29, 2017

joschi added a commit to Graylog2/graylog2-server that referenced this issue Oct 4, 2017

Waiting for data adapter/cache creation in bundle importer (#4197)
Before this change, lookup data adapters, caches and tables were created
sequentially during content pack import, leading to errors when lookup
tables were created before caches or data adapters were successfully
created and started. Imported lookup tables were usable only after the
server was restarted at least once after a content pack import.

After this change, the bundle importer waits for the successful creation
of data adapters/caches before creating lookup tables.

Fixes Graylog2/graylog-plugin-threatintel#57

joschi added a commit to Graylog2/graylog2-server that referenced this issue Oct 4, 2017

Waiting for data adapter/cache creation in bundle importer
Before this change, lookup data adapters, caches and tables were created
sequentially during content pack import, leading to errors when lookup
tables were created before caches or data adapters were successfully
created and started. Imported lookup tables were usable only after the
server was restarted at least once after a content pack import.

After this change, the bundle importer waits for the successful creation
of data adapters/caches before creating lookup tables.

Fixes Graylog2/graylog-plugin-threatintel#57
Refs #4197
(cherry picked from commit 3493509)

dennisoelkers added a commit to Graylog2/graylog2-server that referenced this issue Oct 4, 2017

Waiting for data adapter/cache creation in bundle importer (#4206)
Before this change, lookup data adapters, caches and tables were created
sequentially during content pack import, leading to errors when lookup
tables were created before caches or data adapters were successfully
created and started. Imported lookup tables were usable only after the
server was restarted at least once after a content pack import.

After this change, the bundle importer waits for the successful creation
of data adapters/caches before creating lookup tables.

Fixes Graylog2/graylog-plugin-threatintel#57
Refs #4197
(cherry picked from commit 3493509)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment