Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide pre-build Python wheels #913

Closed
jancespivo opened this issue Jun 11, 2021 · 7 comments
Closed

Provide pre-build Python wheels #913

jancespivo opened this issue Jun 11, 2021 · 7 comments
Assignees
@danielrbradley
Labels
impact/performance Something is slower than expected kind/enhancement Improvements or new features language/python resolution/fixed This issue was fixed

Comments

@jancespivo
Copy link

For example python package uses about 655MB of space. Container images with pulumi consume most of our container registry storage and also network traffic is affected when a new container is created (and downloaded). It also takes huge amount of time in our CI pipelines.

Thx for consideration
@jancespivo jancespivo added the kind/enhancement Improvements or new features label Jun 11, 2021
@mikhailshilkov mikhailshilkov added the impact/performance Something is slower than expected label Jun 11, 2021
@mikhailshilkov
Copy link
Member

Thank you for opening an issue to track this.

Do you have ideas that could help us optimize the size without loosing functionality?

@jancespivo
Copy link
Author

I have some ideas:

  • Split the package to multiple ones with the same namespace (to keep the API the same). The current package would be only container that depends on all of the new ones for backward compatibility. It is workaround but there is certainly no project that uses the whole API.
  • Use zipped python packages - unfortunately with LazyLoader (that pulumi for python uses) it is possible only from python 3.10 I tested it and the resulted file had 145MB with almost no performance hit.

As a workaround we are deleting unused parts after installation (basically the result is the same as the first approach)

@Gerrit-K
Copy link

Gerrit-K commented Nov 22, 2021 •
edited

I was trying out the azure-python example today using pulumi new azure-python and was confused that it downloads basically every version since 0.9.0 when installing the latest version (see output below). I was about to open an issue for that, but I guess it's related to this issue, isn't it?

As a developer this feels wrong. And as someone who uses and loves pulumi (with TypeScript) and tries to spread the word among his (Python-loving) colleagues it's especially painful, since they first have to download unnecessary dependencies for like 20 minutes for just a 1 minute demo afterwards.

pulumi new azure-python output

 
This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name: (pulumi-python-azure)
project description: (A minimal Azure Native Python Pulumi program)
Created project 'pulumi-python-azure'

stack name: (dev)
Created stack 'dev'
Enter your passphrase to protect config/secrets:
Re-enter your passphrase to confirm:

Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
azure-native:location: The Azure location to use: (WestUS) Westeurope
Saved config

Creating virtual environment...
Finished creating virtual environment
Updating pip, setuptools, and wheel in virtual environment...
Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.3.1)
Requirement already satisfied: setuptools in ./venv/lib/python3.9/site-packages (58.5.3)
Collecting setuptools
  Downloading setuptools-59.2.0-py3-none-any.whl (952 kB)
     |████████████████████████████████| 952 kB 2.0 MB/s
Collecting wheel
  Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 58.5.3
    Uninstalling setuptools-58.5.3:
      Successfully uninstalled setuptools-58.5.3
Successfully installed setuptools-59.2.0 wheel-0.37.0
Finished updating
Installing dependencies in virtual environment...
Collecting pulumi<3.0.0,>=2.0.0
  Downloading pulumi-2.25.2-py2.py3-none-any.whl (147 kB)
     |████████████████████████████████| 147 kB 2.0 MB/s
Collecting pulumi-azure-native>=0.7.0
  Downloading pulumi_azure_native-1.47.0.tar.gz (38.9 MB)
     |████████████████████████████████| 38.9 MB 32.5 MB/s
  Preparing metadata (setup.py) ... done
Collecting grpcio!=1.30.0,>=1.9.1
  Downloading grpcio-1.42.0-cp39-cp39-macosx_10_10_x86_64.whl (4.0 MB)
     |████████████████████████████████| 4.0 MB 1.1 MB/s
Collecting dill>=0.3.0
  Downloading dill-0.3.4-py2.py3-none-any.whl (86 kB)
     |████████████████████████████████| 86 kB 4.3 MB/s
Collecting pyyaml>=5.3.1
  Downloading PyYAML-6.0-cp39-cp39-macosx_10_9_x86_64.whl (197 kB)
     |████████████████████████████████| 197 kB 5.4 MB/s
Collecting six>=1.12.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting protobuf>=3.6.0
  Downloading protobuf-3.19.1-cp39-cp39-macosx_10_9_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 4.1 MB/s
Collecting semver>=2.8.1
  Downloading semver-2.13.0-py2.py3-none-any.whl (12 kB)
Collecting parver>=0.2.1
  Downloading parver-0.3.1-py2.py3-none-any.whl (14 kB)
Collecting pulumi-azure-native>=0.7.0
  Downloading pulumi_azure_native-1.46.0.tar.gz (38.8 MB)
     |████████████████████████████████| 38.8 MB 3.0 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.45.0.tar.gz (39.2 MB)
     |████████████████████████████████| 39.2 MB 3.4 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.44.0.tar.gz (39.2 MB)
     |████████████████████████████████| 39.2 MB 1.4 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.43.0.tar.gz (39.0 MB)
     |████████████████████████████████| 39.0 MB 98 kB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.42.0.tar.gz (38.9 MB)
     |████████████████████████████████| 38.9 MB 2.6 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.41.0.tar.gz (38.4 MB)
     |████████████████████████████████| 38.4 MB 236 kB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.40.0.tar.gz (38.2 MB)
     |████████████████████████████████| 38.2 MB 3.9 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.39.0.tar.gz (38.1 MB)
     |████████████████████████████████| 38.1 MB 4.8 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.38.0.tar.gz (38.1 MB)
     |████████████████████████████████| 38.1 MB 630 kB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.37.0.tar.gz (38.0 MB)
     |████████████████████████████████| 38.0 MB 750 kB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.36.0.tar.gz (37.8 MB)
     |████████████████████████████████| 37.8 MB 4.2 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.35.0.tar.gz (37.7 MB)
     |████████████████████████████████| 37.7 MB 3.2 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.34.0.tar.gz (37.0 MB)
     |████████████████████████████████| 37.0 MB 3.8 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.33.0.tar.gz (36.9 MB)
     |████████████████████████████████| 36.9 MB 4.4 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.32.0.tar.gz (36.9 MB)
     |████████████████████████████████| 36.9 MB 2.4 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.31.0.tar.gz (36.8 MB)
     |████████████████████████████████| 36.8 MB 36.9 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.30.0.tar.gz (36.5 MB)
     |████████████████████████████████| 36.5 MB 3.4 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.29.0.tar.gz (36.5 MB)
     |████████████████████████████████| 36.5 MB 2.6 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.28.0.tar.gz (36.1 MB)
     |████████████████████████████████| 36.1 MB 3.8 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.26.0.tar.gz (35.9 MB)
     |████████████████████████████████| 35.9 MB 3.8 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.25.0.tar.gz (35.9 MB)
     |████████████████████████████████| 35.9 MB 2.6 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.24.0.tar.gz (35.7 MB)
     |████████████████████████████████| 35.7 MB 3.0 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.22.0.tar.gz (34.6 MB)
     |████████████████████████████████| 34.6 MB 3.0 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.21.0.tar.gz (34.5 MB)
     |████████████████████████████████| 34.5 MB 4.4 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.20.0.tar.gz (34.2 MB)
     |████████████████████████████████| 34.2 MB 3.2 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.19.0.tar.gz (33.9 MB)
     |████████████████████████████████| 33.9 MB 3.0 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.18.0.tar.gz (33.8 MB)
     |████████████████████████████████| 33.8 MB 3.9 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.17.0.tar.gz (33.8 MB)
     |████████████████████████████████| 33.8 MB 3.6 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.16.0.tar.gz (33.2 MB)
     |████████████████████████████████| 33.2 MB 2.9 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.15.0.tar.gz (32.9 MB)
     |████████████████████████████████| 32.9 MB 4.4 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.14.0.tar.gz (32.5 MB)
     |████████████████████████████████| 32.5 MB 7.4 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.13.0.tar.gz (32.4 MB)
     |████████████████████████████████| 32.4 MB 3.9 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.12.0.tar.gz (32.3 MB)
     |████████████████████████████████| 32.3 MB 3.8 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.11.0.tar.gz (32.2 MB)
     |████████████████████████████████| 32.2 MB 2.1 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.10.0.tar.gz (32.2 MB)
     |████████████████████████████████| 32.2 MB 4.5 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.9.0.tar.gz (31.9 MB)
     |████████████████████████████████| 31.9 MB 2.2 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.8.0.tar.gz (31.7 MB)
     |████████████████████████████████| 31.7 MB 3.9 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.7.0.tar.gz (31.4 MB)
     |████████████████████████████████| 31.4 MB 4.8 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.6.0.tar.gz (30.6 MB)
     |████████████████████████████████| 30.6 MB 3.9 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.5.0.tar.gz (29.6 MB)
     |████████████████████████████████| 29.6 MB 2.0 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.4.0.tar.gz (29.5 MB)
     |████████████████████████████████| 29.5 MB 3.9 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.3.0.tar.gz (29.7 MB)
     |████████████████████████████████| 29.7 MB 5.2 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.2.0.tar.gz (29.6 MB)
     |████████████████████████████████| 29.6 MB 5.7 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.1.0.tar.gz (29.1 MB)
     |████████████████████████████████| 29.1 MB 2.1 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.0.1.tar.gz (29.1 MB)
     |████████████████████████████████| 29.1 MB 2.0 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-1.0.0.tar.gz (29.1 MB)
     |████████████████████████████████| 29.1 MB 4.3 MB/s
  Preparing metadata (setup.py) ... done
  Downloading pulumi_azure_native-0.9.0.tar.gz (28.4 MB)
     |████████████████████████████████| 28.4 MB 3.3 MB/s
  Preparing metadata (setup.py) ... done
Collecting attrs>=19.2
  Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting arpeggio~=1.7
  Downloading Arpeggio-1.10.2-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 5.0 MB/s
Building wheels for collected packages: pulumi-azure-native
  Building wheel for pulumi-azure-native (setup.py) ... done
  Created wheel for pulumi-azure-native: filename=pulumi_azure_native-0.9.0-py3-none-any.whl size=49982591 sha256=74ac46bd52c9b8d8ad44d32649078e512e97f8cc4e106e8ba38c87dd73a295f3
  Stored in directory: /Users/.../Library/Caches/pip/wheels/7e/0d/fa/6bee9b138d37692b296fea15830be647c5486c21afca703674
Successfully built pulumi-azure-native
Installing collected packages: six, semver, pyyaml, protobuf, grpcio, dill, attrs, arpeggio, pulumi, parver, pulumi-azure-native
Successfully installed arpeggio-1.10.2 attrs-21.2.0 dill-0.3.4 grpcio-1.42.0 parver-0.3.1 protobuf-3.19.1 pulumi-2.25.2 pulumi-azure-native-0.9.0 pyyaml-6.0 semver-2.13.0 six-1.16.0
Finished installing dependencies
Your new project is ready to go! ✨

To perform an initial deployment, run 'pulumi up'

@danielrbradley danielrbradley changed the title Package size is huge Reduce SDK package size Mar 3, 2022
@danielrbradley
Copy link
Member

Starting phase 1 of this work here: #1547

@RobbieMcKinstry RobbieMcKinstry mentioned this issue Sep 15, 2022
Open
This was referenced Nov 28, 2022
Closed
Closed
@RobbieMcKinstry RobbieMcKinstry mentioned this issue Dec 6, 2022
Open
@sanmai-NL
Copy link

sanmai-NL commented Jun 22, 2023 •
edited

@danielrbradley I noticed that you only publish source distributions, rather than pre-built wheels on PyPI:

Source Distribution
pulumi_azure_native-1.103.0.tar.gz (64.0 MB view hashes)
Uploaded Jun 3, 2023 source

Offering wheels would address the performance efficiency requirements this issue presupposes.

@danielrbradley
Copy link
Member

Version 2.0.0 of the provider has now been released. This significantly reduces the size of all the SDKs.

I'm going to re-purpose this issue around providing pre-built wheels on PyPI.

@danielrbradley danielrbradley changed the title Reduce SDK package size Provide pre-build Python wheels Jul 17, 2023
@danielrbradley danielrbradley self-assigned this Nov 21, 2023
@danielrbradley danielrbradley added the resolution/fixed This issue was fixed label Nov 21, 2023
@danielrbradley
Copy link
Member

This was completed in:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danielrbradley danielrbradley

Labels
impact/performance Something is slower than expected kind/enhancement Improvements or new features language/python resolution/fixed This issue was fixed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@danielrbradley @jancespivo @mikhailshilkov @sanmai-NL @Gerrit-K