Installation of mozci via pip is broken

[bxbrenden]

Installation of mozci via pip is broken

Issue Description

If a user exactly follows the instructions in the README for this repo, they will get an error once they try importing Push from mozci.push:

from mozci.push import Push

When they do so, they'll receive the following error:

ModuleNotFoundError: No module named 'zstandard'

TL;DR

There's an issue with the way mozci is being packaged and published to PyPI with poetry. It should make zstandard mandatory as already specified in pyproject.toml, but for some reason it doesn't. We need to fix the packaging issue or change the installation instructions.

I saw zstandard mentioned in Issue 545, but it's already marked as resolved and appears to relate to Docker versions of the tool.

Discussion

It appears that the mozci hosted on PyPI is packaged using poetry. If I look in the pyproject.toml file for the repo, I do see zstandard in the dependencies:

[tool.poetry.dependencies]
...
zstandard = {version = "~0"}

Additionally, if I use poetry install from the root of the cloned repo, it installs zstandard as you might expect.

However, a simple pip installation of mozci doesn't work as instructed, and that's what's written in the README.

So, whichever command is being used to package mozci, e.g. poetry build, it is not including zstandard in the dependencies. It is, however, including zstandard in the optional features' dependencies, and you can prove it by running this command in a fresh venv:

pip install 'mozci[cache]']

The METADATA file that gets put into the wheel distribution file for this package shows that zstandard is only required for the optional features:

# Partial contents of METADATA
Provides-Extra: cache
Provides-Extra: cache-memcached
Provides-Extra: cache-redis
Provides-Extra: cache-s3
Provides-Extra: cache-seeded-file
Requires-Dist: zstandard (>=0,<1) ; extra == "cache" or extra == "cache-seeded-file"

Steps to Reproduce

1. Create a clean Python virtual environment:

python3 -m venv venv
source venv/bin/activate

2. Use the virtual environment's version of pip to install mozci as instructed:

pip install mozci

3. Run an interactive python3 interpreter in the venv:

python3

4. In the interactive python3 interpreter, run the import command from the README example:

>>> from mozci.push import push

You will get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bhyde/git/mozci-venv/venv/lib/python3.10/site-packages/mozci/__init__.py", line 2, in <module>
    from mozci.configuration import config  # noqa
  File "/home/bhyde/git/mozci-venv/venv/lib/python3.10/site-packages/mozci/configuration.py", line 227, in <module>
    config = Configuration()
  File "/home/bhyde/git/mozci-venv/venv/lib/python3.10/site-packages/mozci/configuration.py", line 155, in __init__
    self.cache = CustomCacheManager(self._config["cache"])
  File "/home/bhyde/git/mozci-venv/venv/lib/python3.10/site-packages/mozci/configuration.py", line 97, in __init__
    self.register_serializer("compressedpickle", CompressedPickleSerializer())
  File "/home/bhyde/git/mozci-venv/venv/lib/python3.10/site-packages/mozci/util/cache_stores.py", line 238, in __init__
    import zstandard
ModuleNotFoundError: No module named 'zstandard'

[ahal]

It's listed as an extra dependency:
https://github.com/mozilla/mozci/blob/master/pyproject.toml#L50

So you can pip install mozci[cache-seeded-file] to get it. The reasoning is that zstandard is a natively compiled package and can be a massive PITA in dependency lock files and the like, and it is only needed if you are using a specific caching strategy. So we don't include it by default.