python3.7 support #437

smagafurov · 2018-02-14T08:16:55Z

No description provided.

smagafurov · 2018-02-14T08:55:28Z

Problem with python3.7
pypa/setuptools#1257

codecov · 2018-02-14T12:02:35Z

Codecov Report

Merging #437 into master will increase coverage by 0.02%.
The diff coverage is 93.75%.

@@            Coverage Diff             @@
##           master     #437      +/-   ##
==========================================
+ Coverage   93.57%   93.59%   +0.02%     
==========================================
  Files          25       25              
  Lines        3672     3684      +12     
  Branches      194      195       +1     
==========================================
+ Hits         3436     3448      +12     
  Misses        198      198              
  Partials       38       38

Impacted Files	Coverage Δ
tests/test_pool.py	`99.75% <100%> (ø)`	⬆️
aiopg/cursor.py	`98.76% <100%> (ø)`	⬆️
aiopg/sa/result.py	`91.08% <100%> (ø)`	⬆️
aiopg/utils.py	`84.07% <90.9%> (+0.96%)`	⬆️
aiopg/sa/engine.py	`100% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b670ea...b01c4c2. Read the comment docs.

smagafurov · 2018-02-14T13:22:45Z

@asvetlov: fixes for python3.7 done

vir-mir · 2018-02-14T16:38:29Z

@smagafurov you lowered the coverage, could you raise it or leave it as before

smagafurov · 2018-02-15T11:16:06Z

@vir-mir: Sorry but I can't understand how to do this.
See my changes.
May you help me to do this?

vir-mir · 2018-02-15T11:44:03Z

I'll try to look at this week

smagafurov · 2018-02-15T15:24:48Z

@vir-mir: now code coverage is ok

!!! but there is some back incompatibility in python 3.7

This code works in python 3.6

async for value in conn.execute(sql):

But in python 3.7 it throws:

TypeError: 'async for' received an object from __aiter__ that does not implement __anext__: generator

I fix this for python 3.7 as:

async for value in await conn.execute(sql):

See my changes in tests/pep492/test_async_await.py

May be this solution is wrong and we must do something to support python3.7 without this incompatibility.

vir-mir

@asvetlov @jettify how do you think?

vir-mir · 2018-02-16T11:39:42Z

tests/pep492/test_async_await.py

@@ -275,7 +278,10 @@
    result = []
    async with aiopg.sa.create_engine(loop=loop, **pg_params) as engine:
        async with engine.acquire() as conn:
-            async for value in conn.execute(sql):
+            rows = conn.execute(sql)
+            if PY_37:


This is a bad decision. We are here solving the consequences and not the cause. you need to rethink it, you probably need to do this in conn.execute

Can anyone else refactor this hack so we can move forward? I don't know what you mean by "doing this in conn.execute", @vir-mir.

This is a fun one. The examples and documentation show that aiopg.sa.SAConnection.execute is both await-able and can be asynchronously iterated BUT the current implementation should only work by awaiting result before iterating. It just happens to be working without awaiting the result in CPython < 3.7 because it thinks we were supplying an old-style async iterable and that it should await the iterable first like it did pre-3.5.2.

Now that CPython doesn't revert to awaiting when __aiter__ returns an awaitable, we have a couple choices:

Change the documentation, examples, tests to emphasize that execute is not async-for-able but its result is, like async for row in await conn.execute(tbl.select())

or

Leave docs, tests, examples as-is, but make our _SAConnectionContextManagers behave like they should, with their __aiter__ providing something other than a coroutine. This could be the _SAConnectionContextManager instance itself, with some __anext__ implemented, or some other wrapper around the coroutine the manager encapsulates.

This might be the best option given folks have probably already been using for row in conn.execute() in their code.

@smagafurov I've created a possible fix PR to your PR in smagafurov#1.

mikhail-yarosh · 2018-06-28T10:28:34Z

any update on this?

asvetlov · 2018-06-28T10:56:28Z

Please postpone the merge.
I need to put my hands on the PR.

Se7ge · 2018-06-28T10:59:59Z

aiopg/connection.py

@@ -106,7 +106,8 @@ class Connection:

    def __init__(self, dsn, loop, timeout, waiter, echo, **kwargs):
        self._loop = loop
-        self._conn = psycopg2.connect(dsn, async=True, **kwargs)


Here we can use async_ keyword: http://initd.org/psycopg/docs/connection.html#connection.async_

Gr1N · 2018-07-06T18:51:27Z

Hi, any updates and estimates on this? Thanks

…ecute. Supplements pull request aio-libs#437 for aio-libs#436.

The behavior of iteration has changed to where a StopIteration exception should not be raised inside a generator except by something unexpected.

JustinTArthur · 2018-07-20T07:07:13Z

@smagafurov I have two pull requests pending against your PR's branch for fixing some of the remaining issues.

Fix for RuntimeError in Python 3.7+ from unexpected StopIteration raises

Await result before first async iteration in 3.5.2+ in SQL alchemy execute

Resolves Python 3.4 flake8 continuous integration failure.

JustinTArthur · 2018-07-20T19:45:51Z

This last one should address the failing CI checks for 3.4.

This reverts commit 8d82522.

…stopasynciteration Allow QA of StopAsyncIteration lines, give flake8 built-ins hint in Python <3.5

JustinTArthur · 2018-07-20T20:33:54Z

aiopg/utils.py

+                except StopAsyncIteration:
+                    self._obj.close()
+                    self._obj = None
+                    raise


Ideally, we would just make __aiter__ an async generator so we don't have to supply this __anext__ proxying code, but this would break for the small handful of CPython releases >=3.5.2 and <3.6

smagafurov · 2018-07-20T20:57:03Z

.travis.yml

@@ -9,8 +9,10 @@ python:
 - "3.4"
 - "3.5"
 - "3.6"
+- "nightly"


Want to replace "nightly" -> "3.7" but there is issue about Travis and python 3.7: travis-ci/travis-ci#9815

You can add a job with specific settings for that

Nevermind. I did this.

@webknjaz nice. Did you mean to leave in the nightly build? Not opposed to it either way.

Nightly points to 3.8-dev now.

JustinTArthur · 2018-07-24T16:41:02Z

@vir-mir and @asvetlov this should be ready for another pass if you get a moment.

popravich · 2018-08-01T07:13:43Z

LGTM, but I'd rather wait for @asvetlov to share his thoughts on this

txomon · 2018-08-04T04:18:09Z

I have run our battery of integration tests against this branch and has worked good. I see there are a couple of comments open, do they require any further action? We need into 3.7 in my project and right now this is the one dependency keeping us in 3.6, is there anything I can do to push this forward?

vir-mir · 2018-08-10T14:29:54Z

@asvetlov @popravich @jettify if there are no objections, I will wait a few days and salt it.

ibnbay00 · 2018-08-12T16:22:16Z

Please... merge it.. I can't connect using latest psycopg2 :(

python3.7 support

d6f0b0f

smagafurov mentioned this pull request Feb 14, 2018

Python 3.7 compatibility #436

Closed

smagafurov mentioned this pull request Feb 14, 2018

CoroWrapper AsyncPostgresqlConnection.release_cursor ... was never yielded from 05bit/peewee-async#86

Merged

do not test with python nightly

528d889

smagafurov added 4 commits February 15, 2018 16:50

add python nightly (now 3.7-dev) testing

13db903

add forgotten @asyncio.coroutine decorator

97c3e45

tambourine dances trying to prevent lower tests coverage

a475a97

python3.7: __aiter__ MUST return object that implement __anext__

580f306

vir-mir requested changes Feb 16, 2018

View reviewed changes

Se7ge reviewed Jun 28, 2018

View reviewed changes

use psycopg2.connect async_ kwarg

dc516b8

webknjaz assigned asvetlov Jul 10, 2018

JustinTArthur added 2 commits July 19, 2018 18:01

Await result before first async iteration in 3.5.2+ in SQL alchemy ex…

09ab8af

…ecute. Supplements pull request aio-libs#437 for aio-libs#436.

Fix for RuntimeError in Python 3.7+ from unexpected StopIteration raise.

a734ad7

The behavior of iteration has changed to where a StopIteration exception should not be raised inside a generator except by something unexpected.

This was referenced Jul 20, 2018

Fix for RuntimeError in Python 3.7+ from unexpected StopIteration raises smagafurov/aiopg#2

Merged

Await result before first async iteration in 3.5.2+ in SQL alchemy execute smagafurov/aiopg#1

Merged

smagafurov and others added 3 commits July 20, 2018 23:07

Merge pull request #2 from JustinTArthur/fix-iter-exception-bubbling

fb7cbfd

Fix for RuntimeError in Python 3.7+ from unexpected StopIteration raises

Merge pull request #1 from JustinTArthur/fix-saconnect-async-iteration

0a521d5

Await result before first async iteration in 3.5.2+ in SQL alchemy execute

Allow QA of StopAsyncIteration lines, give flake8 built-in hint in <3.5

c5c01cb

Resolves Python 3.4 flake8 continuous integration failure.

JustinTArthur mentioned this pull request Jul 20, 2018

Allow QA of StopAsyncIteration lines, give flake8 built-ins hint in Python <3.5 smagafurov/aiopg#3

Merged

smagafurov and others added 3 commits July 21, 2018 01:08

bare StopAsyncIteration breaks code coverage in python 3.4

8d82522

Revert "bare StopAsyncIteration breaks code coverage in python 3.4"

4528f49

This reverts commit 8d82522.

Merge pull request #3 from JustinTArthur/fix-3.4-builtins-ci-failure-…

e691586

…stopasynciteration Allow QA of StopAsyncIteration lines, give flake8 built-ins hint in Python <3.5

JustinTArthur reviewed Jul 20, 2018

View reviewed changes

JustinTArthur approved these changes Jul 20, 2018

View reviewed changes

smagafurov commented Jul 20, 2018

View reviewed changes

webknjaz added 2 commits July 20, 2018 22:01

Add proper job for 3.7

c84ebca

Add 3.7 job w/o debug

b01c4c2

webknjaz requested review from jettify, popravich, asvetlov and vir-mir July 24, 2018 17:57

vir-mir approved these changes Aug 2, 2018

View reviewed changes

JustinTArthur approved these changes Aug 3, 2018

View reviewed changes

txomon approved these changes Aug 4, 2018

View reviewed changes

vir-mir merged commit 89c95e3 into aio-libs:master Aug 13, 2018

vir-mir pushed a commit to vir-mir/aiopg that referenced this pull request Jan 28, 2019

remover code for aio-libs#437

5f32af7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python3.7 support #437

python3.7 support #437

smagafurov commented Feb 14, 2018

smagafurov commented Feb 14, 2018

codecov bot commented Feb 14, 2018 •

edited

smagafurov commented Feb 14, 2018

vir-mir commented Feb 14, 2018

smagafurov commented Feb 15, 2018

vir-mir commented Feb 15, 2018

smagafurov commented Feb 15, 2018

vir-mir left a comment

vir-mir Feb 16, 2018

Ge0 Jul 9, 2018

JustinTArthur Jul 19, 2018 •

edited

JustinTArthur Jul 19, 2018

mikhail-yarosh commented Jun 28, 2018

asvetlov commented Jun 28, 2018

Se7ge Jun 28, 2018

smagafurov Jun 29, 2018

Gr1N commented Jul 6, 2018

JustinTArthur commented Jul 20, 2018

JustinTArthur commented Jul 20, 2018

JustinTArthur Jul 20, 2018

smagafurov Jul 20, 2018

webknjaz Jul 20, 2018

webknjaz Jul 20, 2018

JustinTArthur Jul 20, 2018

webknjaz Jul 20, 2018

JustinTArthur commented Jul 24, 2018

popravich commented Aug 1, 2018

txomon commented Aug 4, 2018

vir-mir commented Aug 10, 2018

ibnbay00 commented Aug 12, 2018

python3.7 support #437

python3.7 support #437

Conversation

smagafurov commented Feb 14, 2018

smagafurov commented Feb 14, 2018

codecov bot commented Feb 14, 2018 • edited

Codecov Report

smagafurov commented Feb 14, 2018

vir-mir commented Feb 14, 2018

smagafurov commented Feb 15, 2018

vir-mir commented Feb 15, 2018

smagafurov commented Feb 15, 2018

vir-mir left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JustinTArthur Jul 19, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikhail-yarosh commented Jun 28, 2018

asvetlov commented Jun 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Gr1N commented Jul 6, 2018

JustinTArthur commented Jul 20, 2018

JustinTArthur commented Jul 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JustinTArthur commented Jul 24, 2018

popravich commented Aug 1, 2018

txomon commented Aug 4, 2018

vir-mir commented Aug 10, 2018

ibnbay00 commented Aug 12, 2018

codecov bot commented Feb 14, 2018 •

edited

JustinTArthur Jul 19, 2018 •

edited