Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use async io where possible to improve runtime performance #163

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

netomi
Copy link

@netomi netomi commented Nov 29, 2023

This PR fixes #162 .

Things that this PR will modify:

  • use aiohttp whereever appropriate to improve performance
  • cleanups of existing code where appropriate (e.g. get_pypi_data_from_purl which returned duplicate data for each package)

@netomi netomi marked this pull request as ready for review November 30, 2023 12:31
@TG1999
Copy link
Member

TG1999 commented Nov 30, 2023

@netomi thanks! please check the errors in the CI and also do not forget to regen the tests just in case dependencies in tests have been updated https://github.com/nexB/python-inspector#testing

@netomi
Copy link
Author

netomi commented Nov 30, 2023

There are a couple of things that I still need to fix, but the failing CI runs are unrelated to this PR afaict. They also exist for other PRs and look like that the setup is broken.

@TG1999
Copy link
Member

TG1999 commented Nov 30, 2023

@netomi please rebase your PR with latest main, tests are fixed in this PR #165

Signed-off-by: Thomas Neidhart <thomas.neidhart@gmail.com>
Signed-off-by: Thomas Neidhart <thomas.neidhart@gmail.com>
Signed-off-by: Thomas Neidhart <thomas.neidhart@gmail.com>
Signed-off-by: Thomas Neidhart <thomas.neidhart@gmail.com>
@netomi
Copy link
Author

netomi commented Nov 30, 2023

ok so there are some unit test failure that I need to address. Most of them are from the fact that python-inspector returned duplicate package entries in its result up to now, which I fixed, so the expected test results need to be adjusted to take this change into account.

@netomi
Copy link
Author

netomi commented Nov 30, 2023

I could fix most tests (weird thing is that 4 tests fail locally when run in pycharm, but work on the command line so that might be a pycharm problem), there are some weird test failures on macos and windows only. Do you have any idea about that?

@netomi
Copy link
Author

netomi commented Dec 1, 2023

still need to take a look at etc/scripts if the there also need to be adjustments after the changes.

@pombredanne
Copy link
Member

@netomi BTW, do not bother with the etc/scripts ... these come from the shared skeleton repo at https://github.com/nexB/skeleton and should not be of concern for you.

@sschuberth
Copy link
Contributor

Can this be moved forward, please? 😬

@TG1999
Copy link
Member

TG1999 commented Jan 26, 2024

@sschuberth tests are falling, tests needs to be regen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use asyncio to gain performance
4 participants