Initial asyncio support #906
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WhyNotHugo
force-pushed
the
async
branch
2 times, most recently
from
9f84e4d
to
88e9fb0
Compare
graingert
reviewed
Jun 19, 2021
graingert
reviewed
Jun 20, 2021
WhyNotHugo
force-pushed
the
async
branch
7 times, most recently
from
4b6e331
to
0f491a6
Compare
added 3 commits
June 26, 2021 13:02
This is mainly in preparation to moving to an async architecture.
added 4 commits
June 26, 2021 13:40
Add asyncio to the storage backends and most of the codebase. A lot of it merely uses asyncio APIs, but still doesn't actually run several things concurrently internally. Further improvements will be added on top of these changes Thanks to Thomas Grainger (@graingert) for a few useful pointers related to asyncio.
Closed
sumnerevans
added a commit
to sumnerevans/nixpkgs
that referenced
this pull request
Jan 11, 2023
This makes the "google" storage work again after the upstream asyncio refactor: pimutils/vdirsyncer#906 Signed-off-by: Sumner Evans <me@sumnerevans.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So far, all storage test pass with xandikos (a few more with radicale) with the
davstorages. Haven't gotten to thegooglestorage.All
systemandunittests pass. Manually experimenting withsync,discoverandmetasyncthe basics all seem intact, but this still needs more thorough testing.Non-dav storages have not really been ported, they just "work with the new async interface", but don't really operate async, so don't really have any performance changes.
For dav storages, preliminary results show that this is much faster than multi-threaded support, but there's still a lot of work to be done to really unlock all the benefits we can get from using
asyncio. There's some reduction in code complexity by trimming out the threading support too.I'm not 100% happy about passing a
TCPConnectoraround so much, but it makes limiting the maximum amount of connections very easily at a global level.I'll likely need some special wrapper to limit the amount of open files in a similar way (with a limit and a queue).
Missing
googlestorage toasyncio. Still need to find a library for aiohttp + oauth.googlestorage testing. There's no tests for that, so need to set some up to make sure this still works.etesyncis still tied to run as a module rather than standalone, and won't work with our dependencies.Other changes
sha256fingerprints for certificates are not supported.sha1andmd5are insecure and have been dropped.