Change default cython flags to have smaller compiled files

[peterroelants]

Pip install results in a much larger lib artefact compared to conda install (both in a conda environment):

• pip install pydantic: 80M
• conda install pydantic -c conda-forge: 6.6M

I don't know if this is expected or not. I encountered this while trying to minimize the size of a Docker container I'm building, and was surprised that pydantic took up 80M when installed with pip.

I wasn't sure to file this as bug or not, but given the extreme difference in size I thought there might be something going wrong with the pip install.

I've added a full list of the files in site-packages:

Pip install file sizes

> du -sh /opt/conda/lib/python3.8/site-packages/pydantic/* | sort -h
0	/opt/conda/lib/python3.8/site-packages/pydantic/py.typed
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/json.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/version.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/generics.py
12K	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.py
12K	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.py
12K	/opt/conda/lib/python3.8/site-packages/pydantic/typing.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/errors.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/networks.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/color.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/validators.py
24K	/opt/conda/lib/python3.8/site-packages/pydantic/utils.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/types.py
32K	/opt/conda/lib/python3.8/site-packages/pydantic/fields.py
40K	/opt/conda/lib/python3.8/site-packages/pydantic/main.py
40K	/opt/conda/lib/python3.8/site-packages/pydantic/schema.py
196K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so
332K	/opt/conda/lib/python3.8/site-packages/pydantic/__pycache__
516K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.cpython-38-x86_64-linux-gnu.so
552K	/opt/conda/lib/python3.8/site-packages/pydantic/version.cpython-38-x86_64-linux-gnu.so
684K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.cpython-38-x86_64-linux-gnu.so
948K	/opt/conda/lib/python3.8/site-packages/pydantic/json.cpython-38-x86_64-linux-gnu.so
1.4M	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.cpython-38-x86_64-linux-gnu.so
1.5M	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.cpython-38-x86_64-linux-gnu.so
1.6M	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.cpython-38-x86_64-linux-gnu.so
1.7M	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.cpython-38-x86_64-linux-gnu.so
2.0M	/opt/conda/lib/python3.8/site-packages/pydantic/typing.cpython-38-x86_64-linux-gnu.so
2.1M	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.cpython-38-x86_64-linux-gnu.so
2.7M	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.cpython-38-x86_64-linux-gnu.so
3.4M	/opt/conda/lib/python3.8/site-packages/pydantic/color.cpython-38-x86_64-linux-gnu.so
4.0M	/opt/conda/lib/python3.8/site-packages/pydantic/networks.cpython-38-x86_64-linux-gnu.so
5.1M	/opt/conda/lib/python3.8/site-packages/pydantic/validators.cpython-38-x86_64-linux-gnu.so
5.2M	/opt/conda/lib/python3.8/site-packages/pydantic/utils.cpython-38-x86_64-linux-gnu.so
6.2M	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.cpython-38-x86_64-linux-gnu.so
6.4M	/opt/conda/lib/python3.8/site-packages/pydantic/errors.cpython-38-x86_64-linux-gnu.so
6.4M	/opt/conda/lib/python3.8/site-packages/pydantic/fields.cpython-38-x86_64-linux-gnu.so
8.0M	/opt/conda/lib/python3.8/site-packages/pydantic/schema.cpython-38-x86_64-linux-gnu.so
8.5M	/opt/conda/lib/python3.8/site-packages/pydantic/main.cpython-38-x86_64-linux-gnu.so
11M	/opt/conda/lib/python3.8/site-packages/pydantic/types.cpython-38-x86_64-linux-gnu.so

Conda install file sizes:

> du -sh /opt/conda/lib/python3.8/site-packages/pydantic/* | sort -h
0	/opt/conda/lib/python3.8/site-packages/pydantic/py.typed
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/json.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/version.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/generics.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/typing.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/errors.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/networks.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/validators.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/color.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/utils.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/types.py
32K	/opt/conda/lib/python3.8/site-packages/pydantic/fields.py
36K	/opt/conda/lib/python3.8/site-packages/pydantic/main.py
36K	/opt/conda/lib/python3.8/site-packages/pydantic/schema.py
48K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so
80K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.cpython-38-x86_64-linux-gnu.so
84K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.cpython-38-x86_64-linux-gnu.so
88K	/opt/conda/lib/python3.8/site-packages/pydantic/version.cpython-38-x86_64-linux-gnu.so
116K	/opt/conda/lib/python3.8/site-packages/pydantic/json.cpython-38-x86_64-linux-gnu.so
144K	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.cpython-38-x86_64-linux-gnu.so
156K	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.cpython-38-x86_64-linux-gnu.so
164K	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.cpython-38-x86_64-linux-gnu.so
188K	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.cpython-38-x86_64-linux-gnu.so
188K	/opt/conda/lib/python3.8/site-packages/pydantic/typing.cpython-38-x86_64-linux-gnu.so
192K	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.cpython-38-x86_64-linux-gnu.so
280K	/opt/conda/lib/python3.8/site-packages/pydantic/errors.cpython-38-x86_64-linux-gnu.so
296K	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.cpython-38-x86_64-linux-gnu.so
308K	/opt/conda/lib/python3.8/site-packages/pydantic/networks.cpython-38-x86_64-linux-gnu.so
324K	/opt/conda/lib/python3.8/site-packages/pydantic/__pycache__
356K	/opt/conda/lib/python3.8/site-packages/pydantic/color.cpython-38-x86_64-linux-gnu.so
360K	/opt/conda/lib/python3.8/site-packages/pydantic/utils.cpython-38-x86_64-linux-gnu.so
452K	/opt/conda/lib/python3.8/site-packages/pydantic/fields.cpython-38-x86_64-linux-gnu.so
456K	/opt/conda/lib/python3.8/site-packages/pydantic/validators.cpython-38-x86_64-linux-gnu.so
472K	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.cpython-38-x86_64-linux-gnu.so
480K	/opt/conda/lib/python3.8/site-packages/pydantic/types.cpython-38-x86_64-linux-gnu.so
572K	/opt/conda/lib/python3.8/site-packages/pydantic/main.cpython-38-x86_64-linux-gnu.so
572K	/opt/conda/lib/python3.8/site-packages/pydantic/schema.cpython-38-x86_64-linux-gnu.so

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())" from conda install:

root@c6b3b43340a6:/# python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.6.1
            pydantic compiled: True
                 install path: /opt/conda/lib/python3.8/site-packages/pydantic
               python version: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0]
                     platform: Linux-5.8.0-38-generic-x86_64-with-glibc2.10
     optional deps. installed: []

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())" from pip install:

root@c6b3b43340a6:/# python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.7.3
            pydantic compiled: True
                 install path: /opt/conda/lib/python3.8/site-packages/pydantic
               python version: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0]
                     platform: Linux-5.8.0-38-generic-x86_64-with-glibc2.10
     optional deps. installed: []

conda info:

     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 4.9.2
    conda-build version : not installed
         python version : 3.8.5.final.0
       virtual packages : __glibc=2.28=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/conda  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.25.1 CPython/3.8.5 Linux/5.8.0-38-generic debian/10 glibc/2.28
                UID:GID : 0:0
             netrc file : None
           offline mode : False

[peterroelants]

I noticed there is a version difference, however, even with conda install pydantic=1.7.3 -c conda-forge I only get a 7.0M artefact.

[PrettyWood]

Hello @peterroelants and thanks for reporting.
Your issue comes from compiled files with cython (the .so files).
Can you remove everything and try again with SKIP_CYTHON=1 pip install pydantic?

[peterroelants]

I uninstalled the conda pydantic and installed it again with SKIP_CYTHON=1 pip install pydantic. After pip install I get a 80M large /opt/conda/lib/python3.8/site-packages/pydantic/ folder. The files seem to be the same as the one I posted in "Pip install file sizes" above.

Could it be that cython is not skipped?

[PrettyWood]

Can you try clone the repo and build it with the SKIP_CYTHON env variable ?
If you download from pypi, the flag will be useless since it already contains the pre-compiles binaries!
Or you can simply remove all the *.so and use only the *.py files.

---

On linux the compiled files are big because a lot of optimizations are done to improve perf but at the cost of quite some space.
We could also maybe improve the cythonize parameters. Currently we use only the -O3 flag but maybe -Ozor others would be possible and better?

[peterroelants]

Note that the files in the conda build are the same, it also contains the .so files, they are just of smaller size.

What is the sequence of commands I need to run to build the repo?

[peterroelants]

You might be right that it's related to the cython flags, see this comment: cython/cython#2102 (comment)

[PrettyWood]

I finally got my hands on a ubuntu and tested with default CFLAGS and -Os -g0.

(env) ~/pydantic-test/pydantic$ du -sh build
13M	build
(env) ~/pydantic-test/pydantic$ du -sh build-original
74M	build-original

You can try it out yourself by just cloning it, change in setup.py -> os.environ['CFLAGS'] = '-Os -g0'
and run python setup.py build_ext

Now I don't know if we should make the change directly in pydantic as -O3 optimizes performance over size, which makes sense to me. And you can always cythonize yourself. Maybe there is a trade-off in the flags to make or just add some documentation

[peterroelants]

Thanks for verifying.

I see that -Os optimizes for size. From the gcc docs: "-Os enables all -O2 optimizations except those that often increase code size.".
-g0 just seems to turn of debugging.

However, I haven't been able to reproduce your build results. python setup.py build_ext doesn't produce a build folder.
When using python setup.py build install && pip install . I'm only gettiung a 712K folder for pydantic in site-packages. I'm using conda from the continuumio/miniconda3:4.9.2 container to run these experiments.

[vivainio]

This is currently in "feedback wanted" state. What kind of feedback is wanted?

The big binaries are problematic enough in that they warranted a special warning here: https://awslabs.github.io/aws-lambda-powertools-python/utilities/parser/

Even if the optimized binaries were 2x faster, they wouldn't be worth the size increase in modern workloads (lambdas, containers etc) - and people needing to extract the last bit of performance should be able to compile their own version of pydantic

[jvuori]

This has already been discussed here: #1325. The .so files were an issue in AWS Lambda already back then but now the situation is even worse as the files have grown a lot. (I actually already hit the limits in one of my AWS Lambda deployments and after investigating why the package was so big I found this discussion.) In order FastAPI to become the next recommended framework for serverless APIs this major dependency should be more lightweight (i.e. smaller). Installing packages straight from Git repository doesn't sound right.