pygit2 raises KeyError: 'the requested type does not match the type in the ODB'

[dstromberg]

Hello people.

I'm working on porting some cpython2.7 code to cpython3.10 - single codebase. I'm using pygit2 0.28.2 on cpython 2.7, and pygit2 1.9.2 on cpython3.10, at least for now.

I'm getting an error (-3) back from:
err = C.git_remote_push(self._remote, refspecs, opts) ...and payload.check_error(err) is mapping that to:
KeyError: 'the requested type does not match the type in the ODB'

That error only surfaces on cpython3.10, not cpython2.7.

I'm afraid I don't know what to make of the error. I googled for around 45 minutes, and didn't find much.

Here's the full traceback:

Traceback (most recent call last):
    File "/app/shared/common/git/handlers.py", line 488, in Push
        remote.push(temp3, callbacks=self.callbacks)
    File "/usr/local/lib/python3.10/site-packages/pygit2/remote.py", line 257, in push
        payload.check_error(err)
    File "/usr/local/lib/python3.10/site-packages/pygit2/callbacks.py", line 93, in check_error
        check_error(error_code)
    File "/usr/local/lib/python3.10/site-packages/pygit2/errors.py", line 56, in check_error
        raise KeyError(message)'
KeyError: 'the requested type does not match the type in the ODB'

Can anyone please give me a nudge in the right direction? What types is it complaining about?

Thanks!

PS: Does anyone know of a debugger that can singlestep from CPython into C extension modules?

[dstromberg]

Is it possible that pygit2 0.28.2 would 'force' always, while pygit2 1.9.2 will only force by request?

[dstromberg]

I appear to be using:
-rwxr-xr-x 1 root root 4243128 Sep 14 17:01 /usr/local/lib/python3.10/site-packages/pygit2.libs/libgit2-b1d381ba.so.1.4.3

[jdavid]

If it's a regression, works with 0.28.2 but does not with 1.9.2, then try to identify in which version the regression appeared.
In https://www.pygit2.org/install.html#version-numbers you will find a compatibility list.
Verify as well whether it works or not with the latest version, 1.10.1

[dstromberg]

It might be a regression.

I tried 1.0.0, which failed to compile with libgit2 0.99.0 and 0.28.5. It wants 0.28.*, but ld gave: multiple definition of `OdbBackendType'

I also tried 1.0.1, which compiled smoothly but is SIBABRT'ing at runtime.

Any other suggestions?

[dstromberg]

Hello again.

I tried the following versions on CPython 3.10, with results as described:

1.10.1 - KeyError: 'the requested type does not match the type in the ODB'
1.9.2 - KeyError: 'the requested type does not match the type in the ODB'
1.5.0 - KeyError: 'the requested type does not match the type in the ODB'
1.1.0 - KeyError: 'the requested type does not match the type in the ODB'
1.0.1 - KeyError: 'the requested type does not match the type in the ODB'
0.28.2 - KeyError: 'the requested type does not match the type in the ODB'

pygit2 0.28.2 is the one we're using successfully on CPython 2.7, albeit with libgit2 0.99.0 - which feels strange to me. I only tried pygit2 0.28.2 with libgit2 0.28.5 on CPython 3.10.

Any other suggestions?

BTW, with the really old libgit's, I needed to download the source tarball, python setup.py build install, and:
root@tact_pub_api:/usr/local/lib/python3.10/site-packages/pygit2# mv _libgit2.abi3.so _libgit2.so

Thanks!

[jdavid]

If I understand:

• 0.28.2 with Python 2.7 works
• 0.28.2 with Python 3.X does not

Correct?

It's not possible to build pygit2 0.28 with libgit2 0.99 as there is a version check in src/types.h
Probably you are using the binary wheel, so the bundled libgit2 0.28

On the other hand, there is no binary wheel in 0.28 for Python 3.10, so probably you built it
from the source package; 0.28 has binary wheels for 2.7 and 3.4 to 3.7

What OS are you using?

[dstromberg]

Yes, 0.28.2 works with Python 2.7 but not Python 3.10. Python 2.7 clicks along, but 3.10 gives "KeyError: 'the requested type does not match the type in the ODB'".

One of our Dockerfiles is going out of its way to manually build GIT2LIB_VER=0.99.0 from GIT2LIB_URL=https://github.com/libgit2/libgit2/archive/ I'm not certain it's being used I guess. It looks like it's really using:
-rwxr-xr-x 1 root staff 5780912 Oct 3 15:24 libgit2-96e6b8cb.so.0.28.2

Yes, I built the older pygit2's from their respective source packages for Python 3.10.

Our OS is Debian 11.2.

Thanks!

[jdavid]

Okay, could you try 0.28.2 with Python 3.7 and Python 3.4, using the binary wheels?

[dstromberg]

We have been targeting 3.10, but I'll see what I can do.

[dstromberg]

We have a 78 line, m4-preprocessed requirements.txt. Right now, it is preprocessed to allow python2 and python3 from the same m4 input. I might be able to change it to allow python2, python3.10, python3.7 and python3.4, but it's probably going to take a couple of days.

Is there anything else we can try?

[jdavid]

It's to narrow the issue.
Or maybe you have a reproducible test?

[dstromberg]

I understand - narrowing the issue is good. We had problems with 3.9 recently - 3.7 will likely have the same problem.

We have a way of reproducing the problem on demand, but it's part of a large, internal-only software system with many dependencies.

[dstromberg]

We have Python using one version of libgit2, and C++ using another. If they both use the same git repo(s), could that cause the problem we're seeing?

[jdavid]

Maybe... right now I don't have any clue.

The the requested type does not match the type in the ODB error is raised by libgit2, but there are several places where it's raised.

Do you use a custom backend or is it a regular Git repository?

[dstromberg]

Using pygit2 0.28.2 on CPython 3.10 isn't working well enough, so we're back on this.

[dstromberg]

I don't see anything about a custom backend in the code.

[dstromberg]

I created an SSCCE, and ran it on both pygit2 0.28.2 and pygit2 1.10.1.

It fails on both, but 1.10.1 gives a more useful error message that points me at:
https://github.blog/2021-09-01-improving-git-protocol-security-github/

This bit of that URL appears to be especially relevant:

libgit2 and other Git clients
If you’re using libgit2 or another piece of code using libssh2, we recommend you use libssh2 1.9.0 or newer and an ECDSA key, since it does not yet support RSA with SHA-2. Similarly, the Go SSH client also doesn’t yet support RSA with SHA-2, so we recommend using an Ed25519 key there.

Is there a way to use RSA with SHA-2 on pygit2 yet?

If we use 'ssh-keygen -t ecdsa', should we be able to use pygit2 0.28.2 as an interim measure?

Thanks!

[dstromberg]

I went ahead and tried ecdsa keys with a git@github URL, and it didn't appear to help.

On a whim, I tried an https:// URL, and it worked with ECDSA or RSA, easily. But a git@github.com (ssh) URL continues to have problems. Oddly, I had removed my RSA key from github, but RSA continued to work.

[jdavid]

The wheels are bundled with libssh2 1.10.0 since pygit2 1.7.0; before it was libssh2 1.9.0, since 1.3.0 I think.
The problem may be in openssl instead, since the version bundled is relatively old.

If you are using the Linux wheels, instead of building from source, then please try the wheels from the CI, see https://github.com/libgit2/pygit2/actions/runs/3279576093
These bundle a newer version of openssl.