Skip to content

Conversation

@t8y8
Copy link
Collaborator

@t8y8 t8y8 commented Dec 12, 2016

I've got a working solution for #105

Requests provides a way to just get the response headers back and then iterating over the content in chunks. This lets us stream to disk instead of trying to hold large workbooks in memory.

(See http://docs.python-requests.org/en/master/api/#requests.Response.iter_content and http://docs.python-requests.org/en/master/user/advanced/#body-content-workflow)

As a proof of concept I extended our wrapper methods to allow per-call http options (because we don't need to stream every request) and then tested it on workbook download with the script below.

If this approach works, I can clean it up and add it to data source download as well.

But a few questions:

  1. Are we ok with the way I extended our request-get-wrapper function? An alternative is to have a special get_download_request instead
  2. I need to make sure the response always gets closed once we use stream=True I think I'm right but may want to just wrap it all in a context manager or try/except
  3. Preferences on a chunk size? (I did 4MB because it's usually safe)
import tableauserverclient as TSC

authz = TSC.TableauAuth('user', 'pass')
server = TSC.Server("http://server")

server.auth.sign_in(authz)
print("Signed in: " + server.auth_token)

#workbooks = TSC.Pager(server.workbooks)

big_wb = "FIND ONE OR USE CODE TO FIND ONE"

print("Downloading workbook: " + big_wb)

server.workbooks.download(big_wb)

print("DONE")

@t8y8 t8y8 requested review from LGraber and graysonarts December 12, 2016 06:19
@t8y8
Copy link
Collaborator Author

t8y8 commented Dec 13, 2016

Added datasources and wrapped it all in closing() so we can be sure the connection gets closed.

I went ahead and tested on large and small items on an internal server, and realized that the chunk size didn't actually seem to affect the download time that much -- so I went with 1KB, what most examples on the internet do.

I verified python process memory doesn't balloon while downloading a 3GB file, and that it did without the fix.

Side note: Currently download will just overwrite the filename -- not sure if that's what we want as the default behavior or not

@t8y8 t8y8 changed the title [WIP] Fix large downloads Fix large downloads (#105) Dec 13, 2016
@LGraber
Copy link
Contributor

LGraber commented Dec 13, 2016

This is really clean. I like it very much. 🚀

Copy link
Contributor

@LGraber LGraber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice

@t8y8 t8y8 merged commit a7ca709 into tableau:development Dec 14, 2016
@t8y8 t8y8 deleted the 105-fix-large-download branch December 14, 2016 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants