Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Earthpy Get_data key for twitter data is incorrect #512

Closed
lwasser opened this issue Mar 25, 2020 · 2 comments
Closed

Earthpy Get_data key for twitter data is incorrect #512

lwasser opened this issue Mar 25, 2020 · 2 comments
Assignees
Labels
bug Something isn't working high-priority

Comments

@lwasser
Copy link

lwasser commented Mar 25, 2020

Describe the bug
If you try to download the twitter-flood data, get_data fails returning:
To me, it looks like it's trying to unzip a file that is not a zip file.

Downloading from https://ndownloader.figshare.com/files/10960175

---------------------------------------------------------------------------
BadZipFile                                Traceback (most recent call last)
<ipython-input-36-45a13e0cd8e9> in <module>
      1 import earthpy as et
----> 2 et.data.get_data("twitter-flood")

~/miniconda3/envs/earth-analytics-python/lib/python3.7/site-packages/earthpy/io.py in get_data(self, key, url, replace, verbose)
    229                 kind=kind,
    230                 replace=replace,
--> 231                 verbose=verbose,
    232             )
    233             data_paths.append(this_path)

~/miniconda3/envs/earth-analytics-python/lib/python3.7/site-packages/earthpy/io.py in _download(self, url, path, kind, replace, verbose)
    277                 f.write(r.content)
    278         else:
--> 279             self._download_and_extract(path, r, kind, verbose)
    280         return path
    281 

~/miniconda3/envs/earth-analytics-python/lib/python3.7/site-packages/earthpy/io.py in _download_and_extract(self, path, r, kind, verbose)
    305         file_like_object = io.BytesIO(r.content)
    306         if kind == "zip":
--> 307             archive = zipfile.ZipFile(file_like_object)
    308         if kind == "tar":
    309             archive = tarfile.open(fileobj=file_like_object)

~/miniconda3/envs/earth-analytics-python/lib/python3.7/zipfile.py in __init__(self, file, mode, compression, allowZip64, compresslevel)
   1256         try:
   1257             if mode == 'r':
-> 1258                 self._RealGetContents()
   1259             elif mode in ('w', 'x'):
   1260                 # set the modified flag so central directory gets written

~/miniconda3/envs/earth-analytics-python/lib/python3.7/zipfile.py in _RealGetContents(self)
   1323             raise BadZipFile("File is not a zip file")
   1324         if not endrec:
-> 1325             raise BadZipFile("File is not a zip file")
   1326         if self.debug > 1:
   1327             print(endrec)

BadZipFile: File is not a zip file

To Reproduce

import earthpy as et
et.data.get_data("twitter-flood")

This data should just download into the twitter-flood folder. But instead it can't seem to download.

@lwasser lwasser added bug Something isn't working high-priority labels Mar 25, 2020
@lwasser
Copy link
Author

lwasser commented Mar 25, 2020

note that i noticed in earthpy the data are classified as a .zip file. So i changed it to "file" but it's still throwing an error. Note @nkorinek when you test this co-flood-extras also has two files. so we will want to ensure that all keys with files that are not zipped work.

Downloading from https://ndownloader.figshare.com/files/10960175

---------------------------------------------------------------------------
IsADirectoryError                         Traceback (most recent call last)
<ipython-input-5-914afbd4a646> in <module>
      2 et.data.get_data("co-flood-extras")
      3 
----> 4 et.data.get_data("twitter-flood")

~/Documents/github/0-python/earthpy/earthpy/io.py in get_data(self, key, url, replace, verbose)
    229                 kind=kind,
    230                 replace=replace,
--> 231                 verbose=verbose,
    232             )
    233             data_paths.append(this_path)

~/Documents/github/0-python/earthpy/earthpy/io.py in _download(self, url, path, kind, replace, verbose)
    274         os.makedirs(op.dirname(path), exist_ok=True)
    275         if kind == "file":
--> 276             with open(path, "wb") as f:
    277                 f.write(r.content)
    278         else:

IsADirectoryError: [Errno 21] Is a directory: '/Users/leahwasser/earth-analytics/data/twitter-flood/.'


@lwasser lwasser changed the title Get_data doesn't download non zip files Earthpy Get_data key for twitter data is incorrect Mar 25, 2020
@lwasser
Copy link
Author

lwasser commented Mar 26, 2020

implemented via #513

@lwasser lwasser closed this as completed Mar 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working high-priority
Projects
None yet
Development

No branches or pull requests

2 participants