Skip to content

Conversation

sethmlarson
Copy link
Contributor

@sethmlarson sethmlarson commented May 13, 2020

Splitting this PR off into their own separate PRs for reviewability:

Going to continue splitting the individual commits into their own PRs

Async Notes

The work in this PR right now accomplishes:

  • Switches the primary test runner and some fixtures to be pytest. Eventually would like to switch everything to be pytest.
  • Add AIOHttpConnection which uses aiohttp as an HTTP client
  • Add AsyncTransport which is capable of background sniffing with an asyncio.Task
  • Add AsyncConnectionPool which is basically only needed for an async .close() method for connections.
  • Add the [async] extra which will install dependencies required for AsyncElasticsearch to be available. Can be used for upstream dependencies (like elasticsearch-dsl[async] requires elasticsearch[async])
  • Add async implementations of helpers. async_parallel_bulk() isn't implemented currently because it'd require it's own custom implementation, doesn't play nicely with unasync.
  • Change the API generator to generate async APIs first then use unasync to generate sync client APIs and sync helpers. This guarantees that APIs and helpers are in sync with each other.
  • Adds docs for async client and helpers.

Things that are still todo but will be accomplished outside this PR:

  • Update elasticsearch-async repository to point to elasticsearch[async] for 7.x+
  • Implement async_parallel_bulk()
  • Switch sync tests to use pytest fixtures
  • Figure out a better way to "stripe" the async/sync tests across Python 3.6+
    Currently RequestsHttpConnection tests are being directly replaced with AIOHttpConnection tests meaning RequestsHttpConnection isn't tested on Python 3.6+. Isn't too big of an issue.
  • Make PYTHON_CONNECTION_CLASS=AIOHttpConnection test async instead of relying on versions.

Planning on taking a similar approach with adding async support to elasticsearch-dsl except will probably be a lot less code, only needs unasync in a few places. :)

cc @vEpiphyte

@sethmlarson sethmlarson mentioned this pull request May 13, 2020
13 tasks
@sethmlarson
Copy link
Contributor Author

I'm planning on doing a refactor of helpers without unasync as it doesn't make the actual changes easy to review. Planning on pulling out the sync components not tied to either making an API call or handling an iterator which can be async. That'll hopefully make the helpers easier to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant