pip grpcio install is (still) very slow #22815
Comments
|
I hit the same issue and noticed that pip was downloading the source and building instead of getting the pre-built wheel. It looks like all the pre-built wheels should be found: https://pypi.org/project/grpcio/1.28.1/#files This is not the case. I changed version to 1.27.2 and it found the wheel just fine. So this is possibly a bug in PyPi indexes? The release of grpc==1.28.1 was 27 days ago. |
|
Please upgrade pip. We upgrade our binary wheel to |
|
I also noticed this a while ago when testing grpcio 1.27 releases and learned it the hard way. Wanted to leave some archeology trails here for whoever is interested. :)
|
|
Thanks for gathering the links @gertvdijk. Closing this issue. |
|
I'm still experiencing the same issue. Here's how to reproduce: |
|
@rtshadow PyPI only support source wheels or binary wheels that comply with manylinux standard. But alpine is not included by manylinux. So, for Python packages with C extension (like gRPC, numpy), |
|
To elaborate on that, for @rtshadow: the manylinux2010 standard for wheels (PEP-571) assumes the presence of basic but specific runtime libraries. The most essential, glibc, is not present in the minimal Alpine image of Python. Instead, they use musl as the alternative c library to cut down image size even further. This is explained on the Docker Hub Python image description (see excerpt below), but I think those lines could be improved to include what this means with respect to compatibility of wheels. None of this is specific to gRPC, by the way.
Also, if you want to be sure you want to use wheels only and let pip fail on that, there's an option for that. |
|
@lidizheng @gertvdijk - that makes sense. Thank you for the explanation. |
Here is the issue related to long grpcio build time grpc/grpc#22815
* Setup Kafka * Change Kafka URL host and enable print logs * Change location-grpc-service image base to cut down build time Here is the issue related to long grpcio build time grpc/grpc#22815 * Add troubleshoot log * Change into simpler value for troubleshooting purpose * Add try-except for troubleshooting purpose * Remove unnecessary error * Add more information about the error * Fix formatting error * Add more information about error related for troubleshooting purpose * Change into sequence of octets * Change the request value into bytes * Setup location consumer * Decode message value * Decode bytes into Dictionary * Fix no attribute error due incorrect key access * Add insert new record logic to PostgresSQL * Add PIP upgrade * Add additional parameter * add python parameter * Change base image and dependencies * Add try-except and move the save logic into the loop * Remove finally in try-except block * Add keyboard interrupt and add mock value for coordinate * Add rollback for failed command * Fix no "count" attribute error * Calculate the coordinate based on the latitude and longitude * Add POINT function * Insert Geom value with ST_GeomFromText * Convert string into float for latitude and longitude * Remove geoalchemy libraries * Remove unnecessary logs
Programming Language & gRPC version?Python, gRPC v1.41.1 Operating System?Ubuntu 16.04.7 (Xenial) LTS, it's an online virtual device provided by Lanqiao. Runtime / Compiler Version?Python 3.5.2 Steps to reproduce?sudo pip3 install -U pip # Using a self-hosted PyPI mirror, really fast!
sudo pip3 install tensorflow # Really fast downloading, but REALLY SLOW builing wheels ...
# Tensorflow v2.x needs grpcio as a kind of dependencies.What did I expect to see?
What did I see instead?
|
Oof. This version of Ubuntu is EoL since end of April 2021 and is only supported through a commercial subscription.
This version of Python is very old and EoL. It's unlikely that software vendors are producing wheels for this version. Try to upgrade your Python version. I can recommend pyenv to install different Python versions on your system. Then use a python-venv with a newer interpreter to install grpc in. Alternatively, use a (Docker) container with a more modern OS & Python to run and install your software in. |
Sorry, I can't achieve that ... Although Lanqiao online virtual device does provide us
I can't sure if Lanqiao V-devs installed Pyenv as default. But I found that Lanqiao is using Anaconda 3 "base" environment as global intepreter. I'm using Miniconda 3 (conda v4.10.1) and Conda virtual environment (Python 3.7.11) on my OWN device (Microsoft Windows 10 Home China 21H1). Is it also acceptable to use Conda for installing different Python versions? 😕
Lanqiao DOES provide some V-devs containing Docker or Kubernates installations, but they are isolated for different usage ... (They are only available if you're actually learning Docker or K8s courses) For the above exact course, it DOESN'T provide Docker or K8s. 😞 I have no authorization to modify the default configurations of Lanqiao V-devs. I think report it to Lanqiao Official Team and mention this issue may be the best way. They may can update the OS & Python version and solve the entire problem. I'll try to ask them for further help. Thanks for your instructions, @gertvdijk ❤️ |
|
Lanqiao Official Team just pay attention to this issue 1 hour ago.Most of the courses are sharing the same environment so switching Python intepreter to a new version may affect all of them. But after 30 minutes tests & evaluations, they finally update to Python v3.8.10 and solve the problem. The installation time of Tensorflow v2.x immediately reduce to around 4 minites (downloading 489.6MB Thank you again, @gertvdijk ❤️ Installation Logs on Lanqiao V-devsConversation logs with Lanqiao Offical Team (Simplified Chinese) |
@przemekpastuszka przemekpastuszka |
|
@TalhaNaeem1 Please scroll up to read my earlier comment; you're using an alternative Python-Docker distribution (Alpine) for which no grpcio wheels are published. (But even if there were, your version is pinned to a version predating the standard described below.) You can see it here on PyPI; cp3X with Also, consider upgrading to a stable release of grpcio, your version is nearly 2 years old. Similar for the Python 3.7, but okay. I.e. change your steps to: HTH |
|
Well. Thank you @gertvdijk :) |
- swaps out `django-push-notifications` for [`fcm-django`](https://github.com/grafana/fcm-django). Again.. this is a fork of the parent repo for exactly the same reason.. the migrations point to `auth_user` without letting us use our own user model, this has been patched in the `grafana` fork. The reason why we are using `fcm-django` vs `django-push-notifications` is that the latter does not support the new FCM API, only the "legacy" API. The legacy FCM API does not support certain push notification settings that we would like to use. - modifies the iOS/Android specific push notification settings - adds a `flower` pod in the `docker-compose-developer.yml`, useful for debugging tasks locally - sets the mobile app verification token TTL to 5 minutes when developing locally. The default of 1 minute makes working with device emulators really tricky.. This PR also swaps out the base image in `engine/Dockerfile` from `python:3.9-alpine3.16` to `python:3.9-slim-buster`. As to why.. in short, with the introduction of the `fcm-django` library there is now a peer-dependency on [`grpcio`](https://github.com/grpc/grpc) (which is used by `firebase_admin`.. which I am using in this PR to interact directly with Firebase Cloud Messaging (FCM)). `grpcio` does not publish wheels (read: compiled binaries) for the Alpine distro. It does publish wheels for Debian and hence `pip install -r requirements.txt` does not need to build this library from the source distribution. This is a [known "issue"](grpc/grpc#22815 (comment)) and the recommended solution in the community is to.. not use alpine. These were the numbers, when building the image locally, in terms of image size and build time: | | Local image size (uncompressed | Build time (may differ based on your network speed) | | ------------------------- | -------------------------------------- | ---------- | | `python:3.9-alpine3.16` | 785MB | 320s | | `python:3.9-slim-buster` | 1.05GB | 90s | Co-authored-by: Salvatore Giordano <salvatoregiordanoo@gmail.com>
|
On Fedora f37 with poetry install https://github.com/openai/chatgpt-retrieval-plugin it requires pip install grpcio==1.47.5 and it is very long because it is building it |
|
@giuliohome Similar as my previous comment here with the links to pypi, it's pretty easy to see... You're trying to install an older version which does not have Python 3.11 binaries. Just upgrade to one that has it if you require Python 3.11, or use a Python interpreter version 3.10.x (using pyenv or something like that). 1.47.5 only has cp310 wheels:
1.53.0 has cp311 wheels:
|
|
@gertvdijk thanks, yes I know, the problem is that Milvus is not yet ready for python 3.11 but I have solved using Redis (at least for now). OK, everything clarified and your screenshots (with explanation) are very helpful to visually confirm what was written in the logs. |
|
I just tried to install in my machine with python 3.10 and the build took 35 mins... (mac M1, 16G RAM) |
What version of gRPC and what language are you using?
Python, grpc==1.28.1
What operating system (Linux, Windows,...) and version?
Ubuntu 18.04
What runtime / compiler are you using (e.g. python version or version of gcc)
Python 3.6.9
What did you do?
Please provide either 1) A unit test for reproducing the bug or 2) Specific steps for us to follow to reproduce the bug. If there’s not enough information to debug the problem, gRPC team may close the issue at their discretion. You’re welcome to re-open the issue once you have a reproduction.
Install
grpciousing python pip, in a docker containerWhat did you expect to see?
Getting the package installed a lot quicker
What did you see instead?
Very slow install, build from source I guess. For me it takes over 3 minutes!
Anything else we should know about your project / environment?
Using a brand spanking new threadripper system with fast CPU
The following Dockerfile can be used to reproduce
This is the same issue as #12992, which as been locked
The text was updated successfully, but these errors were encountered: