[python] Support pyroaring without BitMap64#8420
Conversation
fa57cb5 to
86f5d31
Compare
| pyroaring<=0.3.3; python_version < "3.7" | ||
| pyroaring<=0.4.5; python_version == "3.7" | ||
| pyroaring>=1.0.0; python_version >= "3.8" | ||
| pyroaring>=1.0.0 |
There was a problem hiding this comment.
Paimon still advertises Python 3.6/3.7 support (python_requires=">=3.6" and classifiers), but pyroaring>=1.0.0 only publishes wheels for cp38+ on PyPI, so Python 3.6/3.7 users fall back to building the sdist. That is a regression from the previous 0.3.x/0.4.x pins, which had cp36/cp37 wheels, and it is especially risky for slim/container installs without a compiler toolchain. Please either raise Pypaimon's supported Python version to 3.8+, or add CI/build support that proves the source build works on 3.6/3.7.
There was a problem hiding this comment.
您好,这个问题已经在原来的PR里讨论过不止一次了。你提供的两个方案,1,升级 Pypaimon 最低依赖到 3.8,在原来的 PR 被您亲自否定了;2,增加 CI 验证源码构建是否在 3.6 / 3.7 下是否构建通过,实际上就是原来 PR 里做的主要工作,并且也已经验证了 pyroaring 可以在 3.6 / 3.7 下通过,但是你要求拆分 PR,当前 PR 就是拆分出来的部分。
不过为了避免更多讨论,我先调研一下在 3.6 / 3.7 下不依赖 pyroaring 来实现 btree global index 并且保持兼容是否可行。感谢 review。
Hello, this issue has already been discussed more than once in the original PR.
Regarding the two options you suggested:
Raising the minimum Pypaimon dependency to Python 3.8 was explicitly rejected by you in the original PR.
Adding CI to verify whether the source build passes on Python 3.6 / 3.7 is essentially the main work that was done in the original PR. It has also already been verified that pyroaring can build successfully on Python 3.6 / 3.7. However, you asked for the PR to be split, and the current PR is the extracted part of that work.
That said, to avoid further discussion, I will first investigate whether it is feasible to implement the B-tree global index on Python 3.6 / 3.7 without depending on pyroaring, while still preserving compatibility. Thank you for the review.
cca6d63 to
266e370
Compare
|
@aisk What is the purpose of creating this PR? I see that you banned 3.6 again before. Do you really want to improve 3.6 or do something about it? |
|
If you have nothing to do, then keep the status quo. |
|
I am closing this PR, please create a PR according to your business needs with a clear purpose, I am not sure what you are completing. |
The purpose is at the original PR description, you can see it at the top of this page, and this is the Chinese translation: 创建这个 PR的动机在这个 PR 的描述里,你可以在页面顶部看到它,这是对应的中文翻译:
Can I re-open this PR now? Or should I create a issue for this problem then re-open this PR? 我能重新打开这个 PR 吗?还是应该创建一个关于这个问题的 issue 再重新打开这个 PR? |
For Python 3.6/3.7, the pyroaring versions specified in requirements.txt do not include BitMap64. As a result, importing or using PyPaimon code paths that create RoaringBitmap64 can fail on those Python versions. Requiring pyroaring 1.0.0+ would provide BitMap64, but those versions do not publish Python 3.6/3.7 wheels, so users would need to build pyroaring from source.
At @JingsongLi's request, this PR adds a fallback for environments where pyroaring.BitMap64 is unavailable. RoaringBitmap64 still uses BitMap64 when present; otherwise it stores values in chunks keyed by the high 32 bits, with each chunk backed by a normal 32-bit pyroaring.BitMap. The dependency ranges and CI installs are also adjusted so Python 3.6/3.7 use pyroaring versions with published wheels, while Python 3.8+ keeps using pyroaring>=1.0.0.