gevent._greenlet.pxd no longer provided in wheel (1.5 regression) #1568
Hello up there. I've tried to
It still works fine if I
Here is the place in pygolang that depends on
Is it please possible to restore providing the
`pip install pygolang` output
The text was updated successfully, but these errors were encountered:
The pxd files were never a documented interface. They're an internal implementation detail, subject to change at any time, even between minor releases, to the point where they might even come and go over time. That's especially true of the ones that have been added recently such as greenlet, queue, etc. It was an accident that they were ever in a distribution to start with, and a packaging update in 1.5a4 removed them.
I'd really like to avoid having to support them. Can you just not use them, like you don't on PyPy?
@jamadden thanks for feedback. While I understand the desire to minimize the surface of provided public API, there are reasons on pygolang side to prefer Cython interface compared to pure Python: the ultimate goal of pygolang is to provide its service in completely
Regarding benchmarking at Python vs Cython-level - it is just that the benchmarks currently exist only in Python form. When used at Cython level (pygolang can be already used like this even with
Said that isn't gevent/_greenlet.pxd, that we are talking about, providing Cython-level annotations for public gevent/greenlet.py? If i understand correctly the
At least when using
Once again, I understand the desire to avoid exposing things as API to users, but given the above rationales and that it already used to work out of the box for at least ~ 1 year, maybe we could do something that it is ok for both gevent and pygolang?
(*) performance tests were done with caring about measurement stability.
Let me try to be more clear. The definitions in the .pxd files are not public, documented or supported. They may change at any time. And when they do, any project that had consumed them must be recompiled or it is highly likely to be broken. This could be the easy thing where an extension module won't even load because of unresolved symbols, but it's at least as likely to simply crash. The way that Cython lays out a class's vtable is fragile from the ABI point of view, and methods invoked through it use hard-coded offsets; any change in that vtable invalidates all previously compiled code. Calling a method is likely to jump to code expecting different parameters or jump into garbage.
If you wish to take that risk for the sake of performance, and wish to restrict users to exactly the version of gevent that your extension was compiled with, that's totally fine. But I don't want to encourage that because I don't have the time to deal with the inevitable bug reports; I choose to discourage that by not distributing the pxd files.
Perhaps you can copy the definitions you wish to use ("vendor" them)?
Starting from gevent >= 1.5 '*.pxd' files for gevent API are no longer provided, at least in released gevent wheels. This broke pygolang: Error compiling Cython file: ------------------------------------------------------------ ... # Gevent runtime uses gevent's greenlets and semaphores. # When sema.acquire() blocks, gevent switches us from current to another greenlet. IF not PYPY: from gevent._greenlet cimport Greenlet ^ ------------------------------------------------------------ golang/runtime/_runtime_gevent.pyx:28:4: 'gevent/_greenlet.pxd' not found Since gevent upstream refuses to restore Cython level access, let's fix the build by using gevent bits via Python-level. Even when used via py import gevent-1.5 brings speed improvement compared to gevent-1.4 (used via cimport): (on firstname.lastname@example.orgGHz, gevent runtime) gevent-1.4 gevent-1.5 (cimport) (py import) name old time/op new time/op delta pyx_select_nogil 9.47µs ± 0% 8.74µs ± 0% -7.70% (p=0.000 n=10+9) pyx_go_nogil 14.3µs ± 1% 12.0µs ± 1% -16.52% (p=0.000 n=10+10) pyx_chan_nogil 7.10µs ± 1% 6.32µs ± 1% -10.89% (p=0.000 n=10+10) go 16.0µs ± 2% 13.4µs ± 1% -16.37% (p=0.000 n=10+10) chan 7.50µs ± 0% 6.79µs ± 0% -9.53% (p=0.000 n=10+10) select 10.8µs ± 1% 10.0µs ± 1% -6.78% (p=0.000 n=10+10) Using gevent-1.5 could have been even faster via cimport (it is still possible to compile and test against gevent installed in development mode via `pip install -e` because pxd files are there in gevent worktree and tarball): gevent-1.5 gevent-1.5 (py import) (cimport) name old time/op new time/op delta pyx_select_nogil 8.74µs ± 0% 7.90µs ± 1% -9.60% (p=0.000 n=9+10) pyx_go_nogil 12.0µs ± 1% 11.2µs ± 2% -6.35% (p=0.000 n=10+10) pyx_chan_nogil 6.32µs ± 1% 5.89µs ± 0% -6.80% (p=0.000 n=10+9) go 13.4µs ± 1% 12.4µs ± 1% -7.54% (p=0.000 n=10+9) chan 6.79µs ± 0% 6.42µs ± 0% -5.47% (p=0.000 n=10+10) select 10.0µs ± 1% 9.4µs ± 1% -6.39% (p=0.000 n=10+10) but we cannot use cimport to access gevent-1.5 universally, since pxd are not shipped in gevent wheel releases. In the future we might want to change plain version check into compile time check whether gevent/_greenlet.pxd is actually present or not and use faster access if yes. Requesting gevent to be installed in non-binary form might be also an option worth trying. However plain version check should be ok for now.  gevent/gevent#1568
Argh! I myself spent time struggling with the binary compatibility issue today. I made what I thought was a trivial change (for speed!) in one pxd file, rebuilt, and ran tests. They passed. Except one that hardcore crashed the interpreter in a
I truly sympathize with the desire for maximum performance, but the safety just isn’t there. It took me longer than I want to admit to figure out what was happening, and that was in a fully controlled, single project environment with debug flags enabled.
As these internal optimizations are obviously still in flux and likely to stay that way I can’t in good conscience foist these types of issues onto users or other maintainers. Thank you for understanding.
@jamadden, thanks for feedback. I understand you. With the given state of lack of reliable Cython dependency tracking, and gevent pxd bits being still in flux, it probably indeed makes more sense not to expose pxd API to users in general, at least for now.
The problem of unreliable dependency tracking on Cython side is old and, sadly, well-known. I myself am used to do
Even if we are disabling pxd interface for gevent for now, I have to share that in my company gevent is regularly criticized for being slow. It is good to see gevent increasingly improving from release to release (thanks for your hard work on this!), but loosing any part of potential speedup is pity, since the competition is in order of magnitude, not percents:
In my view, compared to other ways, gevent/greenlet/greenstack/... provide big advantage by preserving backward compatibility for existing codebases. This gives ability to switch existing programs into lightweight-threads mode without major rewrite and to further enhance them incrementally instead of rewriting everything in one go.
Once again, I understand current situation and the rationale for decision to keep pxd in private.
For non-pure-python mode there is also a lot of confusion of whether to use
See also cython/cython#3208.