New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update geoip for compat with newer ES #1622

Merged
merged 1 commit into from Oct 2, 2016

Conversation

Projects
None yet
1 participant
@msimerson
Member

msimerson commented Sep 30, 2016

Changes proposed in this pull request:

  • adds a geo property to the result store for geoip plugin
  • send the more explicit geo property to ES

Checklist:

  • docs updated

see caution note here: https://www.elastic.co/guide/en/elasticsearch/guide/current/lat-lon-formats.html

@msimerson

This comment has been minimized.

Show comment
Hide comment
@msimerson

msimerson Oct 2, 2016

Member

Background info (for Google and posterity). Without this change, when saving to ES 2.0+, this error message will often be logged:

[ERROR] [$UUID] [log.elasticsearch] [mapper_parsing_exception] failed to parse

If one digs into it further and adds some appropriate logging, they can turn up this little nugget:

[INFO] [$UUID] [log.elasticsearch] { error: 
   { root_cause: [ [Object] ],
     type: 'mapper_parsing_exception',
     reason: 'failed to parse',
     caused_by: 
      { type: 'illegal_argument_exception',
        reason: 'illegal latitude value [120.97210000000001] for p.geoip.ll' } },
  status: 400 }

And that can be explained by a change in how ES parses an array of geo values (inverting lat/long, as explained at the URL above).

Anyone logging to Elasticsearch, who upgrades both Haraka and ES, will need to re-apply the index map template included in the docs for this plugin, so that ES gets the new explicitly defined geo hash.

Member

msimerson commented Oct 2, 2016

Background info (for Google and posterity). Without this change, when saving to ES 2.0+, this error message will often be logged:

[ERROR] [$UUID] [log.elasticsearch] [mapper_parsing_exception] failed to parse

If one digs into it further and adds some appropriate logging, they can turn up this little nugget:

[INFO] [$UUID] [log.elasticsearch] { error: 
   { root_cause: [ [Object] ],
     type: 'mapper_parsing_exception',
     reason: 'failed to parse',
     caused_by: 
      { type: 'illegal_argument_exception',
        reason: 'illegal latitude value [120.97210000000001] for p.geoip.ll' } },
  status: 400 }

And that can be explained by a change in how ES parses an array of geo values (inverting lat/long, as explained at the URL above).

Anyone logging to Elasticsearch, who upgrades both Haraka and ES, will need to re-apply the index map template included in the docs for this plugin, so that ES gets the new explicitly defined geo hash.

@msimerson msimerson merged commit 02fc7c6 into haraka:master Oct 2, 2016

3 checks passed

codecov/patch Coverage not affected when comparing 7ccc199...b7b0d5e
Details
codecov/project 35.83% (+0.00%) compared to 7ccc199
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@msimerson msimerson deleted the msimerson:elasticsearch-geoip branch Oct 2, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment