how to download corpus panlex_lite package in nltk in python #1253

udaysai50 · 2016-01-17T11:25:40Z

I am able to download all the packages except the panlex_lite how to download it?

alvations · 2016-01-17T12:41:34Z

Try within python:

>>> import nltk
>>> nltk.download('panlex_lite')

Or on command line:

$ python -m nltk.downloader panlex_lite

Note: It might take some time to download the data.

stevenbird · 2016-01-17T23:13:35Z

Note that you need to install the development version of NLTK in order to do this.

xiaozongyang · 2016-02-29T11:13:18Z

use this url [http://dev.panlex.org/db/panlex_lite.zip] to download it manually.

alvations · 2016-02-29T11:17:13Z

Wait for NLTK v3.2 and please see extensive discussion on #1283

racekiller · 2016-05-21T12:52:58Z

Hi once panlex_lite is downloaded manually where should I put it within nltk_data?
Thanks

stevenbird · 2016-05-21T23:41:36Z

Please see http://www.nltk.org/data.html

xiaozongyang · 2016-05-22T01:38:54Z

corpora, my complete path is /usr/local/share/nltk_data/corpora

------------------ Original ------------------
From: "racekiller"notifications@github.com;
Date: Sat, May 21, 2016 08:53 PM
To: "nltk/nltk"nltk@noreply.github.com;
Cc: "肖宗阳"xiaozy@mails.ccnu.edu.cn; "Comment"comment@noreply.github.com;
Subject: Re: [nltk/nltk] how to download corpus panlex_lite package in nltk inpython (#1253)

Hi once panlex_lite is downloaded manually where should I put it within nltk_data?
Thanks

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

deepp · 2016-08-10T06:53:24Z

Hi,
Does anyone have idea why its downloading so slow ? At my end its showing 20 hours. Rest of the packages have been downloaded.

xiaozongyang · 2016-08-10T08:24:01Z

@deepp I upload this zip file to baidu cloud. Following is the link and password
link: https://pan.baidu.com/s/1kVavU7d password: 7b5n

deepp · 2016-08-10T08:51:29Z

@XiaoZYang Thanks for response I downloaded file manually from your previous response link. Thanks a ton

xiaozongyang · 2016-08-10T08:54:10Z

@deepp pleasure. be glad to help u

fwanghe · 2016-09-03T07:37:29Z

You can download the panlex_lite.zip from https://dev.panlex.org/db/, and put it in "/nltk_data/corpora/"

SissyCat · 2016-09-06T23:54:38Z

While downloading panlex with nltk downloader, my whole system just froze - even the caps lock indicator light on my keyboard wasn't working anymore. I've restarted my computer, tried again and the same thing happened.
Is there a logfile anywhere to provide you with more info on this?
FYI: I'm running idle3/nltk3/python 3.5.2 on KDE Neon on an AMD64 machine.

I'll just download the zip-file manually.

eupherntech · 2016-10-07T11:43:10Z

what to do after downloading the zip of panlex_lite so that rest packages are downloaded when nltk.download('all') is given? so that it skips panlex_lite downloading? i unzipped the zip folder but still when i try to download rest packages it shows downloading panlex_lite... help please.

stevealbertwong · 2016-10-13T08:32:18Z

@eupherntech same issue.

iamprakashom · 2016-11-22T14:43:22Z

I am also facing the same issue.

BTW, downloaded panlex_lite data manually.

cimarie · 2016-11-28T17:40:47Z

@eupherntech @stevealbertwong You could use nltk.download('all', halt_on_error=False), so that after failing to download the package, you will be asked whether you want to retry to download it. Press n and the rest of packages should be downloaded.

JustFly1984 · 2016-12-31T02:41:55Z

Same issue here, even manually it takes up to 8 hours. Do something about it please!

hcharley · 2017-03-04T18:45:43Z

Based on the file mentioned above, it looks like it's a 2.2 GB file. So you might just need to hang tight and wait!

One thing you can do in the meantime to get some more information is to look at the filesize and last modified time of the panlex_lite.zip file in nltk_data/corpora/ like so:

$ ls -lh nltk_data/corpora/ | grep panlex_lite
-rw-r--r--     1 username  1607558449   2.1G Mar  4 10:51 panlex_lite.zip

aetilley · 2017-03-19T20:38:38Z

I'm having the same issue. I have panlex_lite successfully dowloaded (from http://dev.panlex.org/db/panlex_lite.zip) and located in the correct directory, but when nltk.download() is called it tries to download it again. Is there some other file that needs to be updated to show that the corpus is in place?

Please Note: I would try @cimarie 's suggestion, but the problem is that I'm trying to use tox to test a branch before submitting a pull request, and tox calls nltk.download internally, so I don't think I have the ability to include those options.

stevenbird · 2017-04-14T06:22:00Z

I've updated the checksums, so please try again

aetilley · 2017-04-14T22:52:28Z

@stevenbird Which checksums?

Anyway, it does not appear to have worked. nltk.download('all') still tries to download panlex light, even though I have put the file attached to the above link in my ~/nltk_data/corpora folder.

Also of note, the downloader tries to download panlex_swadesh every time (although this is a much shorter download than panlex_lite). I noticed panlex_swadesh.zip is in the corpora folder, and attempting to unzip it manually gives

Arthurs-MacBook-Pro:corpora aetilley$ unzip panlex_swadesh.zip
Archive: panlex_swadesh.zip
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of panlex_swadesh.zip or
panlex_swadesh.zip.zip, and cannot find panlex_swadesh.zip.ZIP, period.

stevenbird · 2017-04-14T23:18:29Z

@aetilley – the checksums are published on this page – may need to "view source".

They are from this file: https://dev.panlex.org/db/panlex_lite-20170401.zip

Unfortunately I don't have the bandwidth to download it.

There's two things you might try. Maybe you already just did the first in which case the second might be worth a shot.

sudo python -m nltk.downloader panlex_lite
cd PATH_TO_NLTK_DATA; wget https://dev.panlex.org/db/panlex_lite-20170401.zip; unzip panlex_lite-20170401.zip

aetilley · 2017-05-08T20:45:21Z

@stevenbird

I'm afraid that after running both of these (both successfully), nltk.download('all') still can't see panlex_lite.

Again, the main problem here is that it makes it difficult to use tox.

So am I the only one having this problem?

alvations · 2017-05-08T23:22:46Z

Is nltk.download('all') the main cause of these problems? If so, then I think nltk/nltk_data#69 would be something to consider.

Otherwise, the workaround is something like:

>>> import nltk
>>> dler = nltk.downloader.Downloader()
>>> dler._update_index()
>>> dler._status_cache['panlex_lite'] = 'installed' # Trick the index to treat panlex_lite as it's already installed.
>>> dler.download('all')

aetilley · 2017-05-09T02:20:00Z

@alvations

More specifically, that nltk.download('all') correctly skips over all other corpora that I already have, but for some reason tries to get panlex_lite each time.

Also that tox calls nltk.download('all'), so it's difficult to test locally before making a pull request.

alvations · 2017-05-09T14:40:24Z

Hopefully, nltk/nltk_data#75 would resolve some of the issues. And after that's merged, users should be able to do nltk.download('all-nltk') instead of nltk.download('all') if they don't want to wait to download the large panlex_lite file.

aetilley · 2017-05-12T19:10:48Z

@alvations

And what will tox call?

Again, I'm happy to download a large file once but the downloader doesn't seem so see that I already have it so it tries to download it every time.

And again, if I'm the only person having this problem, then maybe it's not a problem, but I'm baffled.

stevenbird · 2017-05-12T19:24:05Z

@aetilley: is this still happening? I think it should be fixed now that we've dropped panlex-lite from the NLTK corpus collection.

aetilley · 2017-05-14T20:22:19Z

@stevenbird, @alvations

Yes, tox appears to be working for me now. Sorry, I didn't catch that you had fixed that.

stevenbird closed this as completed Jan 17, 2016

stevenbird mentioned this issue Feb 24, 2016

panlex is offline (404) #1283

Closed

alvations mentioned this issue Dec 2, 2016

panlex_lite is failing #1517

Closed

alvations mentioned this issue May 9, 2017

Solving initial problems alvations/earthy#1

Open

geblanco mentioned this issue Feb 21, 2022

Panlex lite not working #2950

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how to download corpus panlex_lite package in nltk in python #1253

how to download corpus panlex_lite package in nltk in python #1253

udaysai50 commented Jan 17, 2016

alvations commented Jan 17, 2016

stevenbird commented Jan 17, 2016

xiaozongyang commented Feb 29, 2016

alvations commented Feb 29, 2016

racekiller commented May 21, 2016

stevenbird commented May 21, 2016

xiaozongyang commented May 22, 2016

deepp commented Aug 10, 2016 •

edited

xiaozongyang commented Aug 10, 2016

deepp commented Aug 10, 2016

xiaozongyang commented Aug 10, 2016

fwanghe commented Sep 3, 2016

SissyCat commented Sep 6, 2016

eupherntech commented Oct 7, 2016

stevealbertwong commented Oct 13, 2016 •

edited

iamprakashom commented Nov 22, 2016

cimarie commented Nov 28, 2016 •

edited

JustFly1984 commented Dec 31, 2016

hcharley commented Mar 4, 2017 •

edited

aetilley commented Mar 19, 2017

stevenbird commented Apr 14, 2017

aetilley commented Apr 14, 2017 •

edited

stevenbird commented Apr 14, 2017

aetilley commented May 8, 2017 •

edited

alvations commented May 8, 2017 •

edited

aetilley commented May 9, 2017

alvations commented May 9, 2017

aetilley commented May 12, 2017

stevenbird commented May 12, 2017

aetilley commented May 14, 2017

how to download corpus panlex_lite package in nltk in python #1253

how to download corpus panlex_lite package in nltk in python #1253

Comments

udaysai50 commented Jan 17, 2016

alvations commented Jan 17, 2016

stevenbird commented Jan 17, 2016

xiaozongyang commented Feb 29, 2016

alvations commented Feb 29, 2016

racekiller commented May 21, 2016

stevenbird commented May 21, 2016

xiaozongyang commented May 22, 2016

deepp commented Aug 10, 2016 • edited

xiaozongyang commented Aug 10, 2016

deepp commented Aug 10, 2016

xiaozongyang commented Aug 10, 2016

fwanghe commented Sep 3, 2016

SissyCat commented Sep 6, 2016

eupherntech commented Oct 7, 2016

stevealbertwong commented Oct 13, 2016 • edited

iamprakashom commented Nov 22, 2016

cimarie commented Nov 28, 2016 • edited

JustFly1984 commented Dec 31, 2016

hcharley commented Mar 4, 2017 • edited

aetilley commented Mar 19, 2017

stevenbird commented Apr 14, 2017

aetilley commented Apr 14, 2017 • edited

stevenbird commented Apr 14, 2017

aetilley commented May 8, 2017 • edited

alvations commented May 8, 2017 • edited

aetilley commented May 9, 2017

alvations commented May 9, 2017

aetilley commented May 12, 2017

stevenbird commented May 12, 2017

aetilley commented May 14, 2017

deepp commented Aug 10, 2016 •

edited

stevealbertwong commented Oct 13, 2016 •

edited

cimarie commented Nov 28, 2016 •

edited

hcharley commented Mar 4, 2017 •

edited

aetilley commented Apr 14, 2017 •

edited

aetilley commented May 8, 2017 •

edited

alvations commented May 8, 2017 •

edited