Add support for python 3.11 #403
Conversation
|
All checks technically pass, but I think this should be held up until after #400 is merged and we can ignore python 3.6. There is a latent problem (examine the commit history for this branch LOL) wherein after I update If somebody who still has poetry 1.1.5 (or earlier) installed wants to try a As it stands, |
rbailey-godaddy
force-pushed
the
action-versions
branch
from
7df622d
to
81d1e03
Compare
|
Rebased and cleaned up for a python3.6-free world. No problems with dependency bumps, although pylint came up with a few new callouts. I verified the pygit2 bump does indeed allow us to bypass compilation in the docker image, and Dockerfile is streamlined accordingly. |
rbailey-godaddy
force-pushed
the
action-versions
branch
from
34e67d6
to
2c4729d
Compare
rbailey-godaddy
changed the title
|
There is a subtle problem baked into this PR somewhere. By way of a baseline, the recently-issued 3.3.1 docker container works as expected (and is based on python 3.10). The container generated here (based on python 3.11) is bogus -- I have extended the CI workflow to include a functionality test that catches this problem (and would have caught the 3.3.0 problem that needed 3.3.1 for a fix, if we'd had this at the time). The issue is that this: import pygit2
foo = pygit2.Repository(".")works with the old image but fails with "no repository found" with the new image. I have not yet figured out why...
I manually created a bare virtual environment with python 3.11 and pygit2 1.11.1 and the test code above works, so whatever the problem is doubtless will turn out to be something subtle. When the docker CI test passes, we should be good to go. |
|
I still haven't found a fix for the misbehavior, but here is a very suggestive test case: That is,
Not shown, I embedded rsync so I could copy This suggests the problem is somehow with the directory mapping, but that's really bizarre. (I am experimenting in WSL.) I have tried backing off to base image I am completely flummoxed by what is going on. I would be interested to hear if anybody can reproduce these results, or obtain contrary results (that would point the finger at my screwed-up environment). |
|
It looks like our problem may be a regression in pygit2 1.10.0. Here is the simple test to reproduce:
|
|
Further bisection using the technique I just outlined narrows the problem space to somewhere between pygit2 1.9.1 (which works) and pygit2 1.9.2 (which fails) |
|
Veeeerrrry interestingly, the failure message in a manual diagnostic for 1.9.2 is much more informative than later versions: The delta in pygit2 between 1.9.1 and 1.9.2 is extremely small, and the only consequential change in this context is a version bump from libgit2 from 1.4.2 to 1.4.3. Therefore, I'm now investigating what happened between those two releases. |
|
You just can't win. I added commit cb2cd60 which addresses the change of behavior in libgit2 that was breaking this PR, but now the linter actions all fail with some weird error, even though they all pass locally. In any case, I am moving this PR out of draft as I believe it now is ready for merging. |
* actions/checkout bumped to v3.1.0 * actions/cache bumped to v3.0.11 * actions/setup-python bumped to v4.3.0
docker/setup-buildx-action now v2.2.1 (was v1.6.0)
* Adjust pygit2 requirements to match latest support matrix * Bump poetry to latest stable version
Force pip to install a version of poetry new enough to handle setuptools correctly, instead of whatever old crummy version it happens to find lying around in cache.
So forcing a latest-and-greatest poetry for python 3.6 doesn't work because the latest and greatest version isn't great enough. Fall back to letting pip install the poetry version it wants, but then additionally install setuptools explicitly; this way, if poetry fumbles and thinks it won't be needed, it's okay because we already installed it.
Forcing setuptools installation manually before running poetry is ineffective, because the old poetry for python 3.6 removes the "unnecessary" package. In the key area, namely the python unit test section of tox.ini, try telling poetry we want setuptools also to see if we can avoid this behavior. FWIW, I am beginning to suspect this entire situation is due to rebuilding poetry.lock with a new poetry version and then letting an old poetry version try to interpret it. The problem only arises with python 3.6, because other pythons support versions new enough to not be braindamaged.
This reverts commit 9189332.
This reverts commit a0b087f.
This reverts commit 9ab0d6d.
* Add 3.11 to various test matrix * Bump Dockerfile base from 3.10 to 3.11
* Drop references to python 3.6 in test matrices * Incorporate all of the bot PR version bumps * Skip GitPython over all of the yanked versions
* Bump pylint to latest version -- eliminates numerous false positive detections when run with python 3.11 * Bump minimum python to 3.7.2 to placate pylint after update * Remove exclusions for deprecated/deleted pylint checks * Minor reworking of some tests to eliminate new pylint complaints
We need the latest version for python 3.11 support, so bump version but pin python 3.7 to previous version, which is the last one that supports the old python.
We've had several instances where the docker build is "successful" but the resulting container image is completely brain-damaged. Add a step that exercises it by running a scan of the working directory used to build the container.
* Instead of having poetry tell pip how to install packages, just make it do the job itself. * Since we don't consume tarball dists, don't generate them. If you are wondering why we don't just do a single `poetry install` to do all of the prerequisites *AND* tartufo itself, it's because it doesn't quite work. I think there are linkages to the original content in /app that isn't carried through to the final image; making that work more sanely is left as an exercise for the student.
Latest tagged version understands our updated signatures
Consolidate steps so we actually have the image available for testing
Hopefully this will save the image so we can test it
libgit2 version 1.4.3 introduced a number of new sanity checks aimed at thwarting attacks where an untrusted entity could inject configuration files into a repository that might be read and acted upon by an unwary consumer. By default, it refuses to read a repository that is not owned by the current user. This change became visible in pygit2 1.9.2, which bumped its internal libgit2 linkage from 1.4.2 to 1.4.3. Initially, 1.9.2 was kind enough to report explicitly when raising a GitError that "repository path ... is not owned by current user" -- but later versions did not include this information, making it very difficult to determine why the "[r]epository not found at ..." error was occurring. A new option, `GIT_OPT_SET_OWNER_VALIDATION`, was added to libgit2 to allow users to opt out of this behavior, but it was not included in pygit2 1.9.2; luckily by the time we get to pygit2 1.11.x, it is visible. This commit explicitly forces `GIT_OPT_SET_OWNER_VALIDATION` to `0`, restoring compatibility with pygit2 1.9.1 and earlier. While there are many cases where the default behavior might be appropriate, as a practical manner we must opt out to support use from a docker container, where the owner of the host files will almost never be the same as the process running tartufo inside the container. It's much more straightforward to preserve existing behavior in all cases. If you are a paranoid, do not run scans of repositories you do not own. Note that remote repositories were not affected by this change, because they always caused a clone operation where the resulting working directory is created (and therefore owned) by the current user.
Update github actions to latest released versions, to see if this fixes inscrutable linter workflow failures
rbailey-godaddy
force-pushed
the
action-versions
branch
from
ce886c2
to
20fede1
Compare
`whitelist_externals` -> `allowlist_externals`
To help us get this pull request reviewed and merged quickly, please be sure to include the following items:
PR Type
What kind of change does this PR introduce?
Backward Compatibility
Is this change backward compatible with the most recently released version? Does it introduce changes which might change the user experience in any way? Does it alter the API in any way?
Issue Linking
What's new?
Update actions to latest versions
Update dependencies
poetry updateThe pygit2 version is the first one that supports python 3.11. An explicit option change was made to maintain compatibility with the behavior of pygit2 version 1.9.1 and earlier (used by existing tartufo versions), with respect to allowing repositories owned by other users to be scanned.
The
calculate_entropy()method was changed to static, as it does not utilize any instance or class data and the latest pylint was offended by the prospect of cachingselfinstances.