Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download() method can blow through memory #289

Open
dopplershift opened this issue Oct 10, 2019 · 1 comment
Open

download() method can blow through memory #289

dopplershift opened this issue Oct 10, 2019 · 1 comment

Comments

@dopplershift
Copy link
Member

Try to include:

from siphon.catalog import TDSCatalog
cat = TDSCatalog('http://thredds.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_0p25deg/catalog.xml')
cat.latest.download()  # Don't do this unless you want to use all memory

The problem is that the implementation of download() is pretty naive:

        if filename is None:
            filename = self.name
        with self.remote_open() as infile:
            with open(filename, 'wb') as outfile:
                outfile.write(infile.read())

which means for the 0.25 GFS, it will download an entire 20-30GB run, and only then write it to disk. Not sure if a solution to #139 would fix this, but it might be better to just immediately fix this by reading and writing in e.g 1M block sizes.

@dopplershift
Copy link
Member Author

shutil.copyfileobj or one of its friends may be helpful here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant