Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Python 3.5 (for Glitch) #40

Merged
merged 6 commits into from Jul 13, 2019

Conversation

@simonw
Copy link
Owner

commented Jul 13, 2019

No description provided.

@simonw

This comment has been minimized.

Copy link
Owner Author

commented Jul 13, 2019

Tests now run on 3.5, 3.6 AND 3.7. They previously only ran on 3.6 (they were meant to run on 3.7 too but it looks like that part of my configs was faulty).

@simonw

This comment has been minimized.

Copy link
Owner Author

commented Jul 13, 2019

@simonw

This comment has been minimized.

Copy link
Owner Author

commented Jul 13, 2019

I'm going to try and get this working using just the Python standard library, no dependencies at all. Should be a case of running a urllib request from inside a thread.

https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor

@simonw

This comment has been minimized.

Copy link
Owner Author

commented Jul 13, 2019

Could it be as simple as this?

def blocking_fetch(url, data=None):
    # Returns (message, body) tuple
    message = urllib.request.urlopen(url, data)
    body = message.read()
    return message, body

loop = asyncio.get_running_loop()
message, body = (
    await loop.run_in_executor(None, lambda: blocking_fetch("https://simonwillison.net/"))
)
@simonw

This comment has been minimized.

Copy link
Owner Author

commented Jul 13, 2019

Starting container circleci/python:3.5.2
  image cache not found on this host, downloading circleci/python:3.5.2

Error response from daemon: manifest for circleci/python:3.5.2 not found

Looks like Circle CI doesn't provide a 3.5.2 image - https://circleci.com/docs/2.0/docker-image-tags.json suggests they start at 3.5.3

Mozilla_Firefox

@simonw

This comment has been minimized.

Copy link
Owner Author

commented Jul 13, 2019

It looks like I need to have an image that uses pyenv to install 3.5.2 (since that's the version Glitch use).

@simonw

This comment has been minimized.

Copy link
Owner Author

commented Jul 13, 2019

I'm going to run the tests against 3.5 best available version first.

@simonw

This comment has been minimized.

Copy link
Owner Author

commented Jul 13, 2019

I tried this on Glitch (by adding https://github.com/simonw/datasette-auth-github/archive/python-3-5.zip to the requirements.txt on https://glitch.com/edit/#!/datasette-auth-github-demo ) and got the following error:

AttributeError: module 'asyncio' has no attribute 'get_running_loop'
@simonw

This comment has been minimized.

Copy link
Owner Author

commented Jul 13, 2019

That's because get_running_loop was added in 3.7 - but my unit tests (which pass in 3.5) don't exercise that code because the use my mocked HTTP responses instead.

I'll try get_event_loop() instead. Docs say this:

Get the current event loop. If there is no current event loop set in the current OS thread and set_event_loop() has not yet been called, asyncio will create a new event loop and set it as the current one.
Because this function has rather complex behavior (especially when custom event loop policies are in use), using the get_running_loop() function is preferred to get_event_loop() in coroutines and callbacks.
But I think I'll be OK because my usage is pretty straight-forward.

@simonw simonw changed the title Support Python 3.5 Support Python 3.5 (for Glitch) Jul 13, 2019

@simonw

This comment has been minimized.

Copy link
Owner Author

commented Jul 13, 2019

It works! https://datasette-auth-github-demo.glitch.me/

I'm going to skip running the tests on 3.5.2 explicitly, I'm happy with where this is now.

@simonw simonw merged commit b9f71c1 into master Jul 13, 2019

4 checks passed

ci/circleci: build Your tests passed on CircleCI!
Details
ci/circleci: python-3-5 Your tests passed on CircleCI!
Details
ci/circleci: python-3-6 Your tests passed on CircleCI!
Details
ci/circleci: python-3-7 Your tests passed on CircleCI!
Details
@simonw

This comment has been minimized.

Copy link
Owner Author

commented Jul 13, 2019

This is the code I ended up using for my light-weight standard-library-only asyncio HTTP requests:

 async def http_request(url, body=None):
    "Performs POST if body provided, GET otherwise."

    def _request():
        message = urllib.request.urlopen(url, data=body)
        return message.status, tuple(message.headers.raw_items()), message.read()

    loop = asyncio.get_event_loop()
    status_code, headers, body = await loop.run_in_executor(None, _request)
    return Response(status_code, headers, body)


class Response:
    "Wrapper class making HTTP responses easier to work with"

    def __init__(self, status_code, headers, body):
        self.status_code = status_code
        self.headers = headers
        self.body = body

    def json(self):
        return json.loads(self.text)

    @property
    def text(self):
        # Should decode according to Content-Type, for the moment assumes utf8
        return self.body.decode("utf-8")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.