Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement asynchronus cursor (refs #19) #21

Merged
merged 13 commits into from Sep 24, 2017
Merged

Conversation

laughingman7743
Copy link
Owner

@laughingman7743 laughingman7743 commented Aug 14, 2017

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import concurrent

from pyathena import connect
from pyathena.async_cursor import AsyncCursor


def done_callback(future):
    result_set = future.result()
    print(result_set.state)
    print(result_set.description)
    for r in result_set:
        print(r)


def main():
    futures = []
    cursor = connect(s3_staging_dir='s3://YOUR_BUCKET/path/to/',
                     region_name='us-west-2',
                     cursor_class=AsyncCursor).cursor(max_workers=10)

    query_id, future = cursor.execute("SELECT * FROM many_rows LIMIT 10")
    print(query_id)
    future.add_done_callback(done_callback)
    futures.append(future)

    query_id, future = cursor.execute("SELECT * FROM one_row_complex")
    print(query_id)
    future.add_done_callback(done_callback)
    futures.append(future)

    query_id, future = cursor.execute("SELECT does_not_exist FROM this_really_does_not_exist")
    print(query_id)
    future.add_done_callback(done_callback)
    futures.append(future)

    # blocking
    for _ in concurrent.futures.as_completed(futures):
        pass
    # shutdown all worker
    cursor.close()


if __name__ == '__main__':
    main()

@codecov-io
Copy link

codecov-io commented Aug 14, 2017

Codecov Report

Merging #21 into master will increase coverage by 1.92%.
The diff coverage is 91.94%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #21      +/-   ##
==========================================
+ Coverage   90.29%   92.22%   +1.92%     
==========================================
  Files           8       12       +4     
  Lines         567      772     +205     
==========================================
+ Hits          512      712     +200     
- Misses         55       60       +5
Impacted Files Coverage Δ
pyathena/connection.py 92% <100%> (+0.51%) ⬆️
pyathena/cursor.py 100% <100%> (+11.88%) ⬆️
pyathena/__init__.py 82.14% <100%> (ø) ⬆️
pyathena/result_set.py 89.18% <89.18%> (ø)
pyathena/common.py 89.9% <89.9%> (ø)
pyathena/model.py 92.15% <92.15%> (ø)
pyathena/async_cursor.py 93.47% <93.47%> (ø)
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 576c568...24ab2b8. Read the comment docs.

@laughingman7743 laughingman7743 mentioned this pull request Aug 14, 2017
@SamProtas
Copy link

👍 Looking forward to this feature. Thanks!

@laughingman7743 laughingman7743 force-pushed the async_cursor branch 4 times, most recently from 0cc8070 to 45c6142 Compare September 18, 2017 06:47
@laughingman7743 laughingman7743 merged commit 37449e5 into master Sep 24, 2017
@laughingman7743 laughingman7743 deleted the async_cursor branch September 24, 2017 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants