Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3.5.0 Release #276

Merged
merged 9 commits into from Jan 13, 2024
Merged

3.5.0 Release #276

merged 9 commits into from Jan 13, 2024

Conversation

CleanCut
Copy link
Owner

@CleanCut CleanCut commented Dec 5, 2023

Based on the advice in #273, I have been working on prepping a 3.5.0 release which primarily represents a nice checkpoint release that is the last release supporting Python 3.6 and Python 3.7.

I branched off of ff8d950 for this release. We'll need to resolve some conflicts with main AFTER release and before merging this into main.

I've run into a bunch of blockers:

  • venv is built-in and should be preferred over installing virtualenv
  • The directory name for installing-in-place starts with green-
  • pypy3 can't install one of black's dependencies
  • The test_finalizer test was not succeeding on macOS
  • Coverage seems to be broken (some part of make release) on Python 3.11 on macOS -- I haven't drilled into this, yet.

@sodul I have invited you as a collaborator. It won't let me view your permissions until you accept, but I intend to give you write permission so you can create branches directly on the repository, and push commits to existing branches like this one if you want to (hint, hint 😉 😆).

@sodul
Copy link
Collaborator

sodul commented Dec 6, 2023

@CleanCut can you point me to the issue with Python 3.11 on macOS ? I pulled the 3.5.0 branch on our side and the GH action stage is passing as expected.

image

@CleanCut
Copy link
Owner Author

CleanCut commented Dec 6, 2023

It's a problem that pops up on my local mac when I run make release. Dang, that makes it nearly impossible for you to debug if it doesn't do it in CI as well. Sorry about that.

I'll see what I can figure out the next time I have some free time. Dunno exactly when that will be.

@sodul
Copy link
Collaborator

sodul commented Dec 7, 2023

I've switched my local virtualenv to a fresh Python 3.11.6 and on the 3.5.0 branch. I got this error:

make release
...
source venv-installed/bin/activate; green -vvvv green
Green 3.5.0, Coverage 7.3.2, Python 3.11.6
...
Ran 233 tests in 9.006s using 10 processes

OK (passes=231, skips=2)

(test-installed) completed

make test-versions
# Run the in-place stub under all python versions in the path
./test_versions
Identified python versions: 3.11 3.12 
...
[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip
pyenv: python3.12: command not found

The `python3.12' command exists in these Python versions:
  3.12.0

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
Please install pip under python3.12
make[1]: *** [test-versions] Error 5
make: *** [test-local] Error 2

The workaround was for me was to temporarily force PYTHON_VERSIONS=3.11 in the test_versions script.

Then I got this, which is what I suppose you are referring to:

Ran 233 tests in 6.038s using 10 processes

Coverage of 35% is below minimum level of 100%
FAILED (passes=231, skips=2)
Coverage of 35% is below minimum level of 100%
make: *** [sanity-checks] Error 1

I get loads of these errors:

/Users/stephane/.pyenv/versions/green311/lib/python3.11/site-packages/coverage/report_core.py:115: CoverageWarning: Couldn't parse '/private/var/folders/z7/tq_3ftkd2wv74hzkjgvv2kcr0000gp/T/tmpqwwtok45/tmpcwan2e4g/tmpyje6dije/tmp_a2qbahv/existing_module.py': No source for code: '/private/var/folders/z7/tq_3ftkd2wv74hzkjgvv2kcr0000gp/T/tmpqwwtok45/tmpcwan2e4g/tmpyje6dije/tmp_a2qbahv/existing_module.py'. (couldnt-parse)

The problem seems to be that the temp folder has been cleaned up by the time coverage.py is trying to read from it.

I'm able to see the 35% with just make test-coverage.

---------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                             14610   9519    35%

Ran 233 tests in 5.667s using 10 processes

OK (passes=231, skips=2)

(test-coverage) completed

I'll look into it.

@sodul
Copy link
Collaborator

sodul commented Dec 7, 2023

I simplified the command to reproduce to: PYTHONPATH="." python3 -m green.cmdline -r -vvvv green

And got the coverage errors. Rolling back coverage.py to 6.5.0 from September 2022 seems to fix the issue. The latest version is 7.3.2.

With 6.5.0 I get:

Name                    Stmts   Miss  Cover   Missing
-----------------------------------------------------
green/cmdline.py           44      0   100%
green/command.py           38      0   100%
green/config.py             0      0   100%
green/djangorunner.py      60      0   100%
green/exceptions.py         2      0   100%
green/junit.py            119      0   100%
green/loader.py           261      0   100%
green/output.py           101      0   100%
green/process.py           53      0   100%
green/result.py           380      0   100%
green/runner.py            70      0   100%
green/suite.py             88      0   100%
green/version.py            0      0   100%
-----------------------------------------------------
TOTAL                    1216      0   100%

Ran 233 tests in 6.260s using 10 processes

OK (passes=231, skips=2)

With 6.5.0 (the last 6.x) it works, with 7.0.0 it fails. So the issue is that we are incompatible with coverage 7.x. I'll have to read the release notes and see what we need to do in order to be forward compatible.

@sodul
Copy link
Collaborator

sodul commented Dec 7, 2023

I got some results. It turns out that for coverage 7.x to report just like it did with prior versions we need to pass what should be included. In our own wrapper for green we actually always pass --include-pattern= and this seems to do the trick here.

I think there is a larger 'compatibility' bug to address which is to ensure we pass a new parameter when calling coverage.coverage() in process.py, but since this is not a direct regression in the green codebase I would just fix the release code of green to pass the pattern, then have a bug for proper coverage 7 compatibility.

> PYTHONPATH="." python3 -m green.cmdline -r -vvvv --include-pattern='green/*' green 
Green 3.5.0, Coverage 7.3.2, Python 3.11.6

...
Name                     Stmts   Miss  Cover   Missing
------------------------------------------------------
green/cmdline.py            44      0   100%
green/command.py            38      0   100%
green/config.py              0      0   100%
green/djangorunner.py       60      0   100%
green/exceptions.py          2      0   100%
green/junit.py             119      0   100%
green/loader.py            261      0   100%
green/output.py            101      0   100%
green/process.py            53      0   100%
green/result.py            380      0   100%
green/runner.py             70      0   100%
green/suite.py              88      0   100%
green/test/__init__.py       0      0   100%
green/version.py             0      0   100%
------------------------------------------------------
TOTAL                     1216      0   100%

Ran 233 tests in 5.616s using 10 processes

OK (passes=231, skips=2)

…ith coverage 7.x. Validated with coverage 6.5.0 and 7.3.2.
@sodul
Copy link
Collaborator

sodul commented Dec 7, 2023

Setting the coverage source allowed coverage.py to focus only on the files in the green directory. This works with v6.x and v7.x.

Note that I put the option in pyproject.toml, but you need either python 3.11+ (toml was added to the standard library in 3.11), or have coverage[toml]' installed on older python releases. See https://coverage.readthedocs.io/en/7.3.2/config.html#configuration-reference.

I thing this short fix is good enough to get green v3.5.0 out.

@sodul
Copy link
Collaborator

sodul commented Dec 7, 2023

I took a look at the GH actions to figure out why the coverage works and this seems to be because the coverage action uses coveralls which explicitly requires coverage 6.x.

https://github.com/clumio-oss/green/actions/runs/7119558135/job/19384933814#step:8:15

Collecting coverage!=6.0.*,!=6.1,!=6.1.1,<7.0,>=4.1 (from coveralls)
  Downloading coverage-6.5.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (217 kB)

@CleanCut
Copy link
Owner Author

I managed to steal a few minutes during lunch today. You nearly got us there! Now we're getting past that and crashing on the pypy3 test. I have to get back to my day job, so I haven't been able to dig into this, but I hope it's as simple as installing some toml package in the pypy3 virtual environment. 🤞🏻

+ PYTHONPATH=.
+ venvpypy3/bin/python -m green.cmdline -k green
..........................................ssssssss.................................................................s................................................................EE............................s.....................s

Error in green.test.test_result.TestGreenTestRunCoverage.test_coverage
  File "/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/unittest/case.py", line 624, in run
    self.setUp()
  File "/Users/nathan/proj/green/green/test/test_result.py", line 1096, in setUp
    include=self.args.include_patterns,
  File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/control.py", line 292, in __init__
    context=context,
  File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/config.py", line 597, in read_coverage_config
    config_read = config.from_file(fname, warn, our_file=our_file)
  File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/config.py", line 297, in from_file
    files_read = cp.read(filename)
  File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/tomlconfig.py", line 67, in read
    raise ConfigError(msg.format(filename))
coverage.exceptions.ConfigError: Can't read 'pyproject.toml' without TOML support. Install with [toml] extra

Error in green.test.test_result.TestGreenTestRunCoverage.test_quiet_coverage
  File "/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/unittest/case.py", line 624, in run
    self.setUp()
  File "/Users/nathan/proj/green/green/test/test_result.py", line 1096, in setUp
    include=self.args.include_patterns,
  File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/control.py", line 292, in __init__
    context=context,
  File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/config.py", line 597, in read_coverage_config
    config_read = config.from_file(fname, warn, our_file=our_file)
  File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/config.py", line 297, in from_file
    files_read = cp.read(filename)
  File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/tomlconfig.py", line 67, in read
    raise ConfigError(msg.format(filename))
coverage.exceptions.ConfigError: Can't read 'pyproject.toml' without TOML support. Install with [toml] extra

Ran 233 tests in 4.151s using 32 processes

FAILED (errors=2, passes=220, skips=11)
+ exit 3
make[1]: *** [test-versions] Error 3
make: *** [test-local] Error 2

@sodul
Copy link
Collaborator

sodul commented Dec 15, 2023

Ah this is because coverage.py requires to be installed as coverage[toml] on python versions prior to 3.11. This is because toml is not a standard library as of Python 3.11. See my prior comment with the link to the coverage.py documentation.

This requirement is documented here: https://coverage.readthedocs.io/en/7.3.2/config.html#configuration-reference

For coverage.py to be able to read toml files on older versions of python as 3.11 added toml to the standard library.
Copy link
Collaborator

@sodul sodul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CleanCut
Copy link
Owner Author

pypy3 tests error and hang. (sigh) Maybe I should just drop support for pypy3. Thoughts?

@sodul
Copy link
Collaborator

sodul commented Dec 22, 2023

I do not use pypy since it is always a few years behind cPython. Currently pypy supports 3.9, 3.10 and still 2.7 somehow. Fortunately a quick pyenv install -l shows that pypy3.6-7.3.3 to pypy3.10-7.3.13 are available to install. I'll try these 2 versions locally and see if I can reproduce. Which command are they failing on, I tried the GH Action CI on our side and all of this seems to work, maybe we could add pypy as a target in the CI Actions. I would only target the linux version though.

@sodul
Copy link
Collaborator

sodul commented Dec 22, 2023

It seems to work with pypy 3.10.13:

> python --version
Python 3.10.13 (f1607341da97ff5a1e93430b6e8c4af0ad1aa019, Sep 28 2023, 20:47:55)
[PyPy 7.3.13 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2.5)]

> make test-coverage
./g 3 -r -vvvv green
Green 3.5.0, Coverage 7.3.4, Python 3.10.13
...
Ran 233 tests in 10.399s using 10 processes

OK (passes=231, skips=2)

(test-coverage) completed

I do get an error with pypy 3.9.17:

> python --version                        
Python 3.9.17 (3f3f2298ddc56db44bbdb4551ce992d8e9401646, Jun 15 2023, 11:14:28)
[PyPy 7.3.12 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2.5)]

> make test-coverage 
./g 3 -r -vvvv green
Green 3.5.0, Coverage 7.3.4, Python 3.9.17

Traceback (most recent call last):
  File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/runpy.py", line 200, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/stephane/repos/clumio-oss/green/green/cmdline.py", line 103, in <module>
    sys.exit(main())
  File "/Users/stephane/repos/clumio-oss/green/green/cmdline.py", line 99, in main
    return _main(argv, testing)
  File "/Users/stephane/repos/clumio-oss/green/green/cmdline.py", line 67, in _main
    result = run(test_suite, stream, args, testing)
  File "/Users/stephane/repos/clumio-oss/green/green/runner.py", line 91, in run
    parallel_targets = toParallelTargets(suite, args.targets)
  File "/Users/stephane/repos/clumio-oss/green/green/loader.py", line 336, in toParallelTargets
    proto_test_list = toProtoTestList(suite)
  File "/Users/stephane/repos/clumio-oss/green/green/loader.py", line 320, in toProtoTestList
    toProtoTestList(i, test_list, doing_completions)
  File "/Users/stephane/repos/clumio-oss/green/green/loader.py", line 312, in toProtoTestList
    getattr(suite, exception_method.strip())()
  File "/Users/stephane/repos/clumio-oss/green/green/loader.py", line 78, in testFailure
    raise ImportError(message)
ImportError: Failed to import green.test.test_djangorunner computed from filename /Users/stephane/repos/clumio-oss/green/green/test/test_djangorunner.py
Traceback (most recent call last):
  File "/Users/stephane/repos/clumio-oss/green/green/loader.py", line 52, in loadFromModuleFilename
    __import__(dotted_module)
  File "/Users/stephane/repos/clumio-oss/green/green/test/test_djangorunner.py", line 8, in <module>
    from green import djangorunner
  File "<frozen importlib._bootstrap>", line 1058, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/stephane/repos/clumio-oss/green/green/djangorunner.py", line 73, in <module>
    from django.test.runner import DiscoverRunner
  File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/site-packages/django/test/__init__.py", line 3, in <module>
    from django.test.client import AsyncClient, AsyncRequestFactory, Client, RequestFactory
  File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/site-packages/django/test/client.py", line 12, in <module>
    from asgiref.sync import sync_to_async
  File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/site-packages/asgiref/sync.py", line 130, in <module>
    class AsyncToSync(Generic[_P, _R]):
  File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/typing.py", line 277, in inner
    return func(*args, **kwds)
  File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/typing.py", line 997, in __class_getitem__
    raise TypeError(
TypeError: Parameters to Generic[...] must all be type variables

make: *** [test-coverage] Error 1

But it seems more of an issue of Django being incompatible with pypy and not a problem with green itself.

I would not make this a release blocker, pypy is not guaranteed to be 100% compatible, and I think the 3.5.0 branch is ok to release as it is. We can still claim that it was tested with pypy pypy3.10-7.3.13.

@sodul
Copy link
Collaborator

sodul commented Dec 22, 2023

I tested targeting pypy in GH Actions and other than pypy 3.6 and 3.7 not being installable, everything looks good.

I've updated the 3.5.0 branch to target pypy 3.8, 3.9 and 3.10 as they worked just fine. No need to drop support for pypy, I just think there is some local issue on your laptop.
https://github.com/CleanCut/green/actions/runs/7310069963

@coveralls
Copy link

Coverage Status

coverage: 97.592% (-2.4%) from 100.0%
when pulling c3ebfb5 on 3.5.0
into 4f6ab77 on main.

* 'main' of https://github.com/CleanCut/green:
  ignore installed versions of green in folders that look like green-*
  bump for version 4.0.0
  update changelog
  Lift minimum supported Python version from 3.6 to 3.8
  Address 'addDuration' warnings in python 3.12.
  don't try to find Python 2's readfp method
  update CI python versions

# Conflicts:
#	.github/workflows/ci.yml
#	CHANGELOG.md
#	green/VERSION
@sodul
Copy link
Collaborator

sodul commented Jan 13, 2024

Merging with the 3.12 errors since the branch is known to not be compatible. Main will be ok.

@sodul sodul merged commit 186a6db into main Jan 13, 2024
13 of 15 checks passed
@sodul sodul deleted the 3.5.0 branch January 13, 2024 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants