Discussing the future of pypdfium2's conda packages

[mara004]

This is a re-posting of mindee/doctr#2127 for the broader pypdfium2 users community to note. Feedback welcome!

---

Hi,

pypdfium2 maintainer here again.

I came to ask if we could re-evaluate the conda packaging situation together, seeing as it was you / the doctr project that had originally wished for conda packages of pypdfium2 (cf. #77, mindee/doctr#113, CC @frgfm).

I've been pondering for a while what might be the best way forward:

1. Try to maintain pypdfium2's conda packaging just the way it is?
2. Revise (rewrite) our conda packaging code?
3. Actually deprecate/remove it altogether in favor of PyPI, to keep focused on the core project?

I'll try to explain in due course why we're making these considerations.
First, there have been a number of changes (improvements) that I'd like to acknowledge, which prompted me to revisit pypdfium2's conda packaging:

• The problem that had afflicted us in #113 back in 2022/23, with conda not supporting platform-specific but interpreter independent python packages at that time, seems to have been addressed by CEP 20, which actually introduced a python_version_independent setting. (The corresponding conda-build PR Add an option to create arch specific, python version independent pkgs conda/conda-build#5456 got merged by the end of Feb 2025.)
• The pypdfium2 project is now capable of producing its own pdfium builds, and no longer solely depends on external pdfium-binaries. In particular, using a union of different build strategies allows us to cover more platforms than we could with one strategy alone.
• Conda finally seems to be gaining better integration with the PyPI world via the (official) conda-pypi bridge beta.

Now, our current approach to conda packaging has some limitations that I've been uneasy about for a while:

• As a result of outsourcing the pdfium dependency, to work around the then lack of python version independent packages, pypdfium2's conda packaging is split in 3 packages over 2 custom channels: pypdfium2_helpers, which depends on pypdfium2_raw (pypdfium2-team channel), which depends on pdfium-binaries (bblanchon's channel). (cf. Conda package bblanchon/pdfium-binaries#119)
• This split adds complexity, and it means our conda packages are limited to the pdfium-binaries. They are cut off from the other build strategies.
• Consequently, we have less freedom as to what builds we use, and no direct control over things like release schedule, platforms included, etc. Doing all packaging at our end would grant us more flexibility and room for tweaks.
• The additional source of trouble here is that anaconda.org (officially) have a tight per package storage limit of 3 GB for user repos. This is generally a problem with binary packages, so I had to ask bblanchon to go for a slower release schedule, and we even agreed to reduce the number of platforms uploaded to conda. (cf. Reduce conda release frequency? bblanchon/pdfium-binaries#146)
  Now, shifting packaging to our side would allow us to do things like occasionally making full releases with more platforms, or making an out-of-order release if necessary, while not producing builds at times we don't actually need to. This is not really an option with upstream packaging, and I do not want to annoy the upstream author with these things any more than I already did.
  I recently learned that one can, apparently, overrun the (official) 3 GB limit until an (unofficial) hard limit of 10 GB, but if that will remain, and whether an increase would be granted or not, we do not know.
• Another thing to note is that pdfium-binaries conda builds are packaged with a script written from scratch off the conda package specification, not with the usual tools like conda build.
  bblanchon resolved to do this due to (understandable) dissatisfaction with conda package tooling and platform issues we ran into. As a result, however, the packages might be slightly different than what conda build produces.
  For instance, the package names are like pdfium-binaries: 153.0.7988.0- bblanchon – the version comes without the usual hash/id as in pip: 26.1.2-pyh0d26453_0 defaults, or pypdfium2_helpers: 5.13.0-py_0 pypdfium2-team.
  While technically specification conforming, it might be the only package using that scheme.
  This is just to point out there are subtle differences, and we can only hope those differences won't cause issues at some point.

Apart from that, some more fundamental problems remain, regardless of how we arranged our builds:

• Custom channels don't integrate well with the conda ecosystem (acknowledged in feat: Adds conda recipe & corresponding CI jobs mindee/doctr#1414 (comment), upstream issue is Allow to specify channels in a recipe conda/conda-build#532).
• In particular, packages in a custom channel are inaccessible to any dependents within the conda-forge/anaconda ecosystems themselves. Were doctr packaged at conda-forge, our builds in a custom channel would not be much use, since conda-forge is hermetic (think of it like a Linux distribution's repository).

Finally, I'd like to echo that it has always been possible to pip install pypdfium2 in a conda env, and given that pypdfium2 has no mandatory runtime dependencies, there is not really any question of breaking the conda env (cf. our readme section).
That would not have allowed for depending on pypdfium2 in a conda package recipe, but with the complications around custom channels, I'm not sure it makes much of a difference whether you ask users to manage custom channels, or make pypdfium2 optional at install time and ask them to add it with pip instead.
Or you might use an environment file, which (I think) can include pip packages.

That all said, I wanted to ask if you're still interested in conda packages of pypdfium2 at all?
I see now that mentions of conda seem to have disappeared from this repository – commit mindee/doctr@a054bda removed doctr's conda packaging I think? Or has it been shifted somewhere else?
In general, download count of pypdfium2 from conda seems to be low compared to PyPI.

Given that, I'm not even sure this is the right place to post, but I still wanted to record these thoughts, and ask for your opinion, given that you're the project that initially requested it.
I think I'll also re-post this at pypdfium2's own discussions tracker to receive user feedback if conda packaging is still desired, and what route we should take. (-> #480)

Thanks!

---

Disclaimer: This post is entirely human-written – apologies it got so long. Anyway, I never use gen AI to write comments, let alone code, and AI content is banned from the pypdfium2 organization altogether.

[mara004]

#481 proposes to remove pypdfium2's conda packaging and dedicated setup code. It would be a relief to get rid of.

[mara004]

#481 has been merged following consultation with doctr folks.