Use embedded GeoIP database #562
Conversation
|
In Elasticsearch logs: I assume it loaded correct files. |
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
🤖 GitHub commentsTo re-run your PR in the CI, just comment with:
|
| geoIpCountryMmdbPath := filepath.Join(ingestGeoIPDir, "GeoLite2-Country.mmdb") | ||
| err = writeStaticResource(err, geoIpCountryMmdbPath, geoIpCountryMmdb) | ||
| if err != nil { | ||
| return errors.Wrapf(err, "copying GeoIP country database failed (%s)", geoIpCountryMmdbPath) |
There was a problem hiding this comment.
These files should be added only when using 8.x versions of the stack, otherwise we will need to regenerate all integrations targeting 7.x.
There was a problem hiding this comment.
I'm wondering how does it work at the moment.
If you look at the master build: https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Felastic-package/detail/master/249/pipeline/ , it contains the ingest.geoip.downloader.enabled: false and the output is like this:
�[36melasticsearch_1 |�[0m {"type": "server", "timestamp": "2021-10-29T09:06:56,812Z", "level": "INFO", "component": "o.e.i.g.LocalDatabases", "cluster.name": "elasticsearch", "node.name": "8476f291c554", "message": "initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes" }
�[36melasticsearch_1 |�[0m {"type": "server", "timestamp": "2021-10-29T09:06:56,813Z", "level": "INFO", "component": "o.e.i.g.DatabaseRegistry", "cluster.name": "elasticsearch", "node.name": "8476f291c554", "message": "initialized database registry, using geoip-databases directory [/tmp/elasticsearch-9816551266999378429/geoip-databases/ygIzHKcVSRSzStUofwIYLA]" }
In 8.0.0 (this PR) it's like this:
�[36melasticsearch_1 |�[0m {"@timestamp":"2021-10-29T11:25:02.948Z", "log.level": "INFO", "message":"initialized config databases [[GeoLite2-Country.mmdb, GeoLite2-ASN.mmdb, GeoLite2-City.mmdb]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes", "service.name":"ES_ECS","process.thread.name":"main","log.logger":"org.elasticsearch.ingest.geoip.ConfigDatabases","event.dataset":"elasticsearch.server","elasticsearch.node.name":"b8f554d090ec","elasticsearch.cluster.name":"elasticsearch"}
�[36melasticsearch_1 |�[0m {"@timestamp":"2021-10-29T11:25:02.950Z", "log.level": "INFO", "message":"initialized database registry, using geoip-databases directory [/tmp/elasticsearch-3032188866989068903/geoip-databases/fpV1Mx8bTOmtvFA1rgpDTA]", "service.name":"ES_ECS","process.thread.name":"main","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","event.dataset":"elasticsearch.server","elasticsearch.node.name":"b8f554d090ec","elasticsearch.cluster.name":"elasticsearch"}
These files should be added only when using 8.x versions of the stack, otherwise we will need to regenerate all integrations targeting 7.x.
It would be a directory hack to make it running as we use the Docker Compose snapshot, so basically:
ingest-geoip-default and ingest-geoip-8x. The first one would be empty. Otherwise it won't be possible without the blocker I linked in the original issue.
There was a problem hiding this comment.
I'm wondering how does it work at the moment.
No idea what these log lines mean 😬
What I have seen is that in 7.x, if the downloader is disabled, then it uses a set of included databases, with the downloader it downloads a new set and uses it. In 8.0 no set of databases are included, so without databases no enrichment is done and a tag is added to indicate the problem.
It would be a directory hack to make it running as we use the Docker Compose snapshot, so basically:
ingest-geoip-defaultandingest-geoip-8x. The first one would be empty. Otherwise it won't be possible without the blocker I linked in the original issue.
Ok, if this is too hacky maybe we have to wait for #557.
There was a problem hiding this comment.
Let me try with the hack and we'll decide what to do about it.
This reverts commit 0481bb1.
|
This PR requires correct boot of the Elastic stack v8.0.0, so I will wait with merging until we have a healthy pipeline. |
|
/test |
3 participants
Issue: #556