Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PyPI checker and issue creator #22807

Draft
wants to merge 30 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
77a5c27
Start creating PyPI checker
anthonykim1 Jan 30, 2024
3f25b72
add to run pypi_notifier with token
anthonykim1 Jan 30, 2024
e0eb242
allow access to path to access requirement.txt
anthonykim1 Jan 30, 2024
e29d80b
get dictionary of package and version
anthonykim1 Jan 30, 2024
6e518ab
add function to get latest package version
anthonykim1 Jan 30, 2024
dd49237
add function to fetch all latest version
anthonykim1 Jan 30, 2024
b368b6c
function to denote version difference
anthonykim1 Jan 30, 2024
36aeb6b
use GH_REPO to create issue
anthonykim1 Jan 30, 2024
3cc92d7
how do I get pytest version
anthonykim1 Jan 30, 2024
ab3291f
commit using importlib_metadata
anthonykim1 Jan 31, 2024
95dc6f7
change to every monday as suggested
anthonykim1 Feb 1, 2024
37205b1
improve on parsing requirements
anthonykim1 Feb 1, 2024
bb82ac9
why is there merge conflict
anthonykim1 Feb 2, 2024
116d075
Transfer to more targeted notifier
anthonykim1 Feb 2, 2024
0b81409
check with pre pip
anthonykim1 Feb 2, 2024
70e66ec
lint prettier
anthonykim1 Feb 27, 2024
ddd54d4
add more description
anthonykim1 Feb 27, 2024
9e6295d
remove unused import
anthonykim1 Feb 27, 2024
9b3f792
comment unused var
anthonykim1 Feb 27, 2024
d98f24f
comment unused
anthonykim1 Feb 27, 2024
0e0163c
remove unused
anthonykim1 Feb 27, 2024
14a48c6
remove nox, run ruff
anthonykim1 Feb 29, 2024
14e7069
add types
anthonykim1 Feb 29, 2024
c2a08c4
install test requirement and use sys.executable
anthonykim1 Feb 29, 2024
b947a5d
change the issue body
anthonykim1 Mar 1, 2024
ee433f4
remove pre tag as recommended
anthonykim1 Mar 1, 2024
5c0f0a1
replace python with sys.executable
anthonykim1 Mar 1, 2024
c5763f8
prepare for case that script is wrong
anthonykim1 Mar 2, 2024
568c1b1
Update comment
anthonykim1 Mar 2, 2024
e20d903
remove unused run sort import
anthonykim1 Mar 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/workflows/notifier.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Packages Update Check

on:
schedule:
# Run at UTC 00:00 Mon
- cron: '0 0 * * 1'

jobs:
check-PyPI:
name: Check for package updates on PyPI and create issue if tests fail
runs-on: ubuntu-latest
permissions:
issues: write

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Use Python 3.12
uses: actions/setup-python@v5
with:
python-version: 3.12

- name: Update pip, install wheel
run: python -m pip install -U pip wheel PyGithub
shell: bash

- name: Install test requirements
run: python -m pip install --upgrade -r build/test-requirements.txt

- name: Run Checker
run: python ./.github/workflows/pypi_notifier.py
env:
GITHUB_ACCESS_TOKEN: ${{ secrets.GITHUB_TOKEN }}
81 changes: 81 additions & 0 deletions .github/workflows/pypi_notifier.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
import importlib.metadata
import os
import subprocess
from typing import List, Optional

import github
anthonykim1 marked this conversation as resolved.
Show resolved Hide resolved
import pytest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is pytest imported here? it won't even be installed when this script is actually called.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we would need this for importlib.metadata.version("pytest") to correctly fetch the version of pytest that is imported.
Should I add pytest in additional to python -m pip install -U pip wheel PyGithub or do we not need it at all??

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In your pipeline you are not installing pytest. import pytest will immediately fail. This works for you locally because you have pytest installed locally. Try running this from a fresh environment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need pytest then you need to install all requirements before running this script. In this script you can later update the installed pytest to whatever version you want.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see... the intent of importing pytest and running importlib.metadata.version("pytest") was to get the version of pytest that our python extension was using...

Now I'm little lost on how to correctly get this value.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I install test-requirement.txt and then use importlib to dynaimically get it since test-requirement.txt does not pin to specific version of pytest.

import requests
from packaging import version

GH = github.Github(os.getenv("GITHUB_ACCESS_TOKEN"))
GH_REPO = GH.get_repo(os.getenv("GITHUB_REPOSITORY"))


def fetch_all_package_versions(package_name: str) -> Optional[List[str]]:
url = f"https://pypi.org/pypi/{package_name}/json"
response = requests.get(url)

if response.status_code == 200:
data = response.json()
versions = data["releases"].keys()
# Sort by packaging library and we have version objects.
return sorted(versions, key=lambda v: version.parse(v))
else:
print(f"Failed to fetch data for {package_name}")
return None


def main() -> None:
# Check Pytest version for Python Repo,
# If there is new version, run Python test,
# If Python tests fail, then create issue
# OR We still notify to be safe.
latest_pytest_version: str = fetch_all_package_versions("pytest")[-1]
our_pytest: str = importlib.metadata.version("pytest")

if latest_pytest_version is None:
issue_body: str = (
"Failed to fetch latest Pytest version in Python extension Repository"
)
GH_REPO.create_issue(
title="Packages may need to be updated", body=issue_body, labels=["debt"]
)

if latest_pytest_version != our_pytest:
issue_body: str = "Pytest may need to be updated:\n"
GH_REPO.create_issue(
title="Packages may need to be updated", body=issue_body, labels=["debt"]
)

# Note: --pre will not work if requirements uses hashes.
# We should already be using latest Pytest, but can force via -m pip install pytest==<version> .
# Check to see if those tests pass, if they do not, then create an issue.
subprocess.run(
[
"sys.executable",
"-m",
"pip",
"install",
"--upgrade",
"-r",
"build/test-requirements.txt",
],
check=False,
)
subprocess.run(["sys.executable", "-m", "build/requirements.txt"])
# Run all tests in pythonFiles/tests/run_all.py using subprocess
test_exit_code: subprocess.CompletedProcess = subprocess.run(
["sys.executable", "pythonFiles/tests/run_all.py"]
)
if test_exit_code != 0:
issue_body = "Tests failed with newest Pytest version. Please check for compatibility, or regression."
GH_REPO.create_issue(
title="Tests failed with newest Pytest version",
body=issue_body,
labels=["debt"],
)


if __name__ == "__main__":
main()