METRON-733: Remove Geo database from ParserBolt

[justinleet]

To create the original problem, just run up a parser and make sure there's no geo data on hdfs (by default in /apps/metron/geo).

This PR removes geo from metron-parsers entirely (since it shouldn't be necessary at all, only pulled in when Stellar cares about it).

Testing

To validate this, I basically ran through the squid demo with the geo file missing, which works as expected (no exception thrown). In addition, the ParserBoltTest is updated to not have a reference to the test Geo DB data (and runs fine without it).

To ensure that Stellar GEO_GET works as expected in a parser, quick-dev was spun up. The steps for squid were followed, but with a custom parser config

{
  "parserClassName": "org.apache.metron.parsers.GrokParser",
  "sensorTopic": "squid",
  "parserConfig": {
    "grokPath": "/patterns/squid",
    "patternLabel": "SQUID_DELIMITED",
    "timestampField": "timestamp"
  },
  "fieldTransformations" : [
    {
      "transformation" : "STELLAR"
    ,"output" : [ "geo_test" ]
    ,"config" : {
      "geo_test" : "GEO_GET(ip_dst_addr)"
                }
    }
                           ]
}

Either update global.json with a valid geo.hdfs.file or run /usr/metron/0.3.1/bin/geo_enrichment_load.sh -z node1:2181 -r /apps/metron/geo/default/ to place the file in the default spot (instead of a timestamped stop). This is necessary to ensure that the push doesn't clobber geo configs.

The resulting data in the index includes

{
...
               "geo_test": {
                  "country": "US",
                  "dmaCode": "807",
                  "city": "San Francisco",
                  "postalCode": "94107",
                  "latitude": "37.7697",
                  "location_point": "37.7697,-122.3933",
                  "locID": "5391959",
                  "longitude": "-122.3933"
               },
...
               "ip_dst_addr": "151.101.192.73",
...
}

[ottobackwards]

Does this mean you can get rid of the local geo database in the parsers resource dir?

[justinleet]

Updated to drop the local geo database in parsers

[nickwallen]

The build error does not seem related to your PR. I saw the same failure on #450.

The resource com.fasterxml:oss-parent:pom:28 really does not exist in Maven Central, AFAIK. Hmm.

[ERROR] Failed to execute goal on project metron-enrichment: Could not resolve dependencies 
for project org.apache.metron:metron-enrichment:jar:0.3.1: 
Failed to collect dependencies at com.maxmind.geoip2:geoip2:jar:2.8.0 ->
 com.maxmind.db:maxmind-db:jar:1.2.1 -> 
com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: 
Failed to read artifact descriptor for com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: 
Could not find artifact com.fasterxml:oss-parent:pom:28 in central 
(http://repo.maven.apache.org/maven2) -> [Help 1]

[justinleet]

@nickwallen it is a separate issue. https://issues.apache.org/jira/browse/METRON-734 exists to track and fix it

[ottobackwards]

I am seeing this on my local branch, not the PR On February 22, 2017 at 09:13:12, Nick Allen (notifications@github.com) wrote: The build error does not seem related to your PR. I saw the same failure on #450 <#450>. The resource com.fasterxml:oss-parent:pom:28 really does not exist in Maven Central, AFAIK. Hmm. [ERROR] Failed to execute goal on project metron-enrichment: Could not resolve dependencies for project org.apache.metron:metron-enrichment:jar:0.3.1: Failed to collect dependencies at com.maxmind.geoip2:geoip2:jar:2.8.0 -> com.maxmind.db:maxmind-db:jar:1.2.1 -> com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to read artifact descriptor for com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Could not find artifact com.fasterxml:oss-parent:pom:28 in central (http://repo.maven.apache.org/maven2) -> [Help 1] — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#461 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABD1_98dtsK9jFWAsUhiM3fkRJZzHr_Kks5rfEJ3gaJpZM4MIH4d> .

[cestella]

+1 by inspection, looks good! Thanks @justinleet