Skip to content

[python] Support pyroaring without BitMap64#8420

Closed
aisk wants to merge 2 commits into
apache:masterfrom
aisk:fix-pyroaring-bitmap64-py36
Closed

[python] Support pyroaring without BitMap64#8420
aisk wants to merge 2 commits into
apache:masterfrom
aisk:fix-pyroaring-bitmap64-py36

Conversation

@aisk

@aisk aisk commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

For Python 3.6/3.7, the pyroaring versions specified in requirements.txt do not include BitMap64. As a result, importing or using PyPaimon code paths that create RoaringBitmap64 can fail on those Python versions. Requiring pyroaring 1.0.0+ would provide BitMap64, but those versions do not publish Python 3.6/3.7 wheels, so users would need to build pyroaring from source.

At @JingsongLi's request, this PR adds a fallback for environments where pyroaring.BitMap64 is unavailable. RoaringBitmap64 still uses BitMap64 when present; otherwise it stores values in chunks keyed by the high 32 bits, with each chunk backed by a normal 32-bit pyroaring.BitMap. The dependency ranges and CI installs are also adjusted so Python 3.6/3.7 use pyroaring versions with published wheels, while Python 3.8+ keeps using pyroaring>=1.0.0.

@aisk aisk force-pushed the fix-pyroaring-bitmap64-py36 branch from fa57cb5 to 86f5d31 Compare July 1, 2026 16:27
Comment thread paimon-python/dev/requirements.txt Outdated
pyroaring<=0.3.3; python_version < "3.7"
pyroaring<=0.4.5; python_version == "3.7"
pyroaring>=1.0.0; python_version >= "3.8"
pyroaring>=1.0.0

@JingsongLi JingsongLi Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Paimon still advertises Python 3.6/3.7 support (python_requires=">=3.6" and classifiers), but pyroaring>=1.0.0 only publishes wheels for cp38+ on PyPI, so Python 3.6/3.7 users fall back to building the sdist. That is a regression from the previous 0.3.x/0.4.x pins, which had cp36/cp37 wheels, and it is especially risky for slim/container installs without a compiler toolchain. Please either raise Pypaimon's supported Python version to 3.8+, or add CI/build support that proves the source build works on 3.6/3.7.

@aisk aisk Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

您好,这个问题已经在原来的PR里讨论过不止一次了。你提供的两个方案,1,升级 Pypaimon 最低依赖到 3.8,在原来的 PR 被您亲自否定了;2,增加 CI 验证源码构建是否在 3.6 / 3.7 下是否构建通过,实际上就是原来 PR 里做的主要工作,并且也已经验证了 pyroaring 可以在 3.6 / 3.7 下通过,但是你要求拆分 PR,当前 PR 就是拆分出来的部分。

不过为了避免更多讨论,我先调研一下在 3.6 / 3.7 下不依赖 pyroaring 来实现 btree global index 并且保持兼容是否可行。感谢 review。

Hello, this issue has already been discussed more than once in the original PR.
Regarding the two options you suggested:
Raising the minimum Pypaimon dependency to Python 3.8 was explicitly rejected by you in the original PR.
Adding CI to verify whether the source build passes on Python 3.6 / 3.7 is essentially the main work that was done in the original PR. It has also already been verified that pyroaring can build successfully on Python 3.6 / 3.7. However, you asked for the PR to be split, and the current PR is the extracted part of that work.

That said, to avoid further discussion, I will first investigate whether it is feasible to implement the B-tree global index on Python 3.6 / 3.7 without depending on pyroaring, while still preserving compatibility. Thank you for the review.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已更新。

Updated.

@aisk aisk changed the title [python] Require pyroaring>=1.0.0 so BitMap64 is available on Python 3.6/3.7 [python] Support pyroaring without BitMap64 Jul 3, 2026
@aisk aisk force-pushed the fix-pyroaring-bitmap64-py36 branch from cca6d63 to 266e370 Compare July 3, 2026 16:39
@JingsongLi

Copy link
Copy Markdown
Contributor

@aisk What is the purpose of creating this PR? I see that you banned 3.6 again before. Do you really want to improve 3.6 or do something about it?

@JingsongLi

Copy link
Copy Markdown
Contributor

If you have nothing to do, then keep the status quo.

@JingsongLi

Copy link
Copy Markdown
Contributor

I am closing this PR, please create a PR according to your business needs with a clear purpose, I am not sure what you are completing.

@JingsongLi JingsongLi closed this Jul 4, 2026
@aisk

aisk commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

@aisk What is the purpose of creating this PR? I see that you banned 3.6 again before. Do you really want to improve 3.6 or do something about it?

The purpose is at the original PR description, you can see it at the top of this page, and this is the Chinese translation:

创建这个 PR的动机在这个 PR 的描述里,你可以在页面顶部看到它,这是对应的中文翻译:

对于 Python 3.6/3.7,requirements.txt 中指定的 pyroaring 版本并不包含 BitMap64。因此,在这些 Python 版本下,导入或使用会创建 RoaringBitmap64 的 PyPaimon 代码路径时可能会失败。要求使用 pyroaring 1.0.0+ 可以提供 BitMap64,但这些版本没有发布适用于 Python 3.6/3.7 的 wheel,因此用户需要从源码构建 pyroaring

根据 @JingsongLi 的要求,取消 PyPaimon 对 Python3.6 / 3.7 的支持,或者直接要求这两个版本的用户需要本地安装 GCC 等编译器而在本地构建 pyroaring 都被否定了,因此只能通过在 3.6 / 3.7 只下创建 fallback 来解决这个问题。

本 PR 为 pyroaring.BitMap64 不可用的环境增加了一个回退实现。RoaringBitmap64BitMap64 可用时仍会使用它;否则,会按照高 32 位对数值进行分块,并使用普通的 32 位 pyroaring.BitMap 存储每个分块。依赖版本范围和 CI 安装逻辑也进行了相应调整:Python 3.6/3.7 使用已经发布 wheel 的 pyroaring 版本,而 Python 3.8+ 继续使用 pyroaring>=1.0.0

Can I re-open this PR now? Or should I create a issue for this problem then re-open this PR?

我能重新打开这个 PR 吗?还是应该创建一个关于这个问题的 issue 再重新打开这个 PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants