Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full Planet Import only takes 300GB in Elastic Search #349

Open
gaeunfalo16 opened this issue Jun 28, 2024 · 0 comments
Open

Full Planet Import only takes 300GB in Elastic Search #349

gaeunfalo16 opened this issue Jun 28, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@gaeunfalo16
Copy link

Hi All!

I've tried import full planet project and have some questions

Acoording to pelias document, full planet builds weigh in at over 600million documents and require aoubt 450GB.

  • Current full planet builds weigh in at over 600 million documents, and require about 450GB total storage in Elasticsearch.
    But in my case, full planet build weigth in about 700million documents and takes about 290GB.

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .geoip_databases _-aP4esSTaiUV1SHGmIjww 1 0 33 33 31mb 31mb
green open pelias 5x1mTod0STyCld7xCgg0zg 12 0 693269565 173147230 282.4gb 282.4gb

I cannot convince whether importing data is success or not.

Here is my importing log and pelias system env below... I saw "No datapath configured, nothing to do." at last and importing was finished. Does it mean that I configured wrong datapath?

  • datapath: DATA_DIR=./data

info: [admin-lookup:worker] region worker process exiting, stats: {"calls":722842,"hits":433148,"misses":289694}
info: [admin-lookup:worker] country worker process exiting, stats: {"calls":284582,"hits":73725,"misses":210857}
info: [admin-lookup:worker] ocean worker process exiting, stats: {"calls":54997,"hits":33706,"misses":21291}
info: [admin-lookup:worker] locality worker process exiting, stats: {"calls":12549816,"hits":2801773,"misses":9748043}
info: [dbclient-geonames] paused=false, transient=0, current_length=0, indexed=12681984, batch_ok=25364, venue=7066398, locality=5191814, region=3932, batch_retries=0, failed_records=0, neighbourhood=339824, localadmin=32575, country=193, county=46825, persec=7448.4, dependency=42, macroregion=56, macrocounty=320, borough=5
info: [dbclient-geonames] paused=false, transient=0, current_length=0, indexed=12681984, batch_ok=25364, venue=7066398, locality=5191814, region=3932, batch_retries=0, failed_records=0, neighbourhood=339824, localadmin=32575, country=193, county=46825, persec=7448.4, dependency=42, macroregion=56, macrocounty=320, borough=5
time="2024-06-27T14:15:22+09:00" level=warning msg="/home1/irteam/docker/projects/planet/docker-compose.yml: version is obsolete"
warn: [transit] 'pelias.json' config lacks a transit object entry. Transit importer quitting after taking no action
time="2024-06-27T14:15:23+09:00" level=warning msg="/home1/irteam/docker/projects/planet/docker-compose.yml: version is obsolete"
info: [csv-importer] No datapath configured, nothing to do

@gaeunfalo16 gaeunfalo16 added the enhancement New feature or request label Jun 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant