Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use async request library #10

Closed
Tracked by #11
avaldebe opened this issue Jan 6, 2022 · 5 comments · Fixed by #12
Closed
Tracked by #11

use async request library #10

avaldebe opened this issue Jan 6, 2022 · 5 comments · Fixed by #12

Comments

@avaldebe
Copy link
Collaborator

avaldebe commented Jan 6, 2022

Hi John,

Have you considered to use an asynchronous request library?
Replacing requests by aiohttp would give you concurrent downloads,
like in the 3rd implementation shown on this article.

Would you accept a PR?
I have no experience with async python, but I'm willing to give it a try.

Cheers,
Álvaro.

@JohnPaton
Copy link
Owner

Hey, thanks for checking out airbase! I've had something like this in mind for ages but just never got around to it. If I had done it myself I would probably go the concurrent.futures route, but I would certainly be open to a properly async PR if it remained backwards compatible!

@avaldebe
Copy link
Collaborator Author

avaldebe commented Jan 7, 2022

[...] I would certainly be open to a properly async PR if it remained backwards compatible!

Would need to bump the minimum python version up to Python 3.7 in order to use aiohttp:
But I guess that would not be a problem, since older versions are out of support.
If so, would it be OK to take it one step further and bump it up to 3.8?

@JohnPaton
Copy link
Owner

Given that this is often used in an academic setting, older Python versions are more common than one might hope 😬 I'd prefer 3.7 for that reason.

@avaldebe
Copy link
Collaborator Author

I went a bit out of scope on PR #11.
Needed to type annotate quite a bit in order to figure find missing awaits and such errors.
I also added configurations for my standard python dev tools.
I'm happy to remove to revert the changes you find unnecessary .

@avaldebe
Copy link
Collaborator Author

Hi @JohnPaton

After adding type hints to all the functions, mypy found the following inconsistency.

request = AirbaseRequest() has request.country = None and request._country_list = [None], which will would crash on line 321.

airbase/airbase/airbase.py

Lines 313 to 324 in 8420903

self._country_list = util.string_safe_list(country)
self._shortpl_list = util.string_safe_list(shortpl)
self._download_links = []
for c in self._country_list:
for p in self._shortpl_list:
self._download_links.append(
util.link_list_url(
c, p, year_from, year_to, source, update_date
)
)

I'm aware that normal usage of the library would not generate such a request, but I think it would be better to remove the default value for country from AirbaseRequest.init`.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants