New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix prefix handling, handle throttling exceptions, add multithreading #1

Open

umbrant wants to merge 1 commit into dacort:master from umbrant:master

umbrant commented Mar 12, 2020

There were a few issues I ran into while running this script:

S3 prefix was not handled correctly
Hitting throttling exceptions that were not handled by default, and need to be retried

I also added multi-threading to try and speed it up, since it takes many hours to run against our Athena warehouse right now.


          Fix the script

7d6c28f

umbrant changed the title ~~Improve script and fix bugs~~ Fix prefix handling, handle throttling exceptions, add multithreading

Owner

dacort commented Mar 12, 2020

Nice, thanks @umbrant! I'll try to take a look at this in the next couple of days, but an initial glance looks good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment