New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discussion and troubleshooting on PyPI (pip) installation #8671

Open
szha opened this Issue Nov 15, 2017 · 31 comments

Comments

Projects
None yet
@szha
Member

szha commented Nov 15, 2017

PyPI for Apache MXNet installation has been available for a while and has been the most important channel for distribution. I'm opening this issue to collect feedbacks, suggestions and issue reports for releases and nightly pre-releases.

Current Status

Supported variants:

  • mxnet on Windows, Linux, and Mac OSX, with CPU-only without MKL-DNN support.
  • mxnet-cu75 on Linux, supports CUDA-7.5. (discontinued in 1.2)
  • mxnet-cu80 on Windows and Linux, supports CUDA-8.0.
  • mxnet-cu90 on Windows and Linux, supports CUDA-9.0.
  • mxnet-cu91 on Windows and Linux, supports CUDA-9.1. (will discontinue in 1.3)
  • mxnet-cu92 on Windows and Linux, supports CUDA-9.2.
  • mxnet-mkl on Windows, Linux, and Mac OSX, with CPU-only MKLDNN support.
  • mxnet-cu75mkl on Linux, supports CUDA-7.5 and MKLDNN support. (discontinued in 1.2)
  • mxnet-cu80mkl on Windows and Linux, supports CUDA-8.0 and MKLDNN support.
  • mxnet-cu90mkl on Windows and Linux, supports CUDA-9.0 and MKLDNN support.
  • mxnet-cu91mkl on Windows and Linux, supports CUDA-9.1 and MKLDNN support. (will discontinue in 1.3)
  • mxnet-cu92mkl on Windows and Linux, supports CUDA-9.2 and MKLDNN support.

Compilation Flags

Flags mxnet mxnet-mkl mxnet-cu75 mxnet-cu75mkl mxnet-cu80 mxnet-cu80mkl mxnet-cu90 mxnet-cu90mkl mxnet-cu91 mxnet-cu91mkl
USE_OPENMP* 1 1 1 1 1 1 1 1 1 1
USE_DIST_KVSTORE 1 1 1 1 1 1 1 1 1 1
USE_MKLDNN 0 1 0 1 0 1 0 1 0 1
USE_BLAS apple for OSX, openblas for Linux
USE_LAPACK 1 1 1 1 1 1 1 1 1 1
USE_NVRTC 0 0 1 1 1 1 1 1 1 1
USE_NCCL** 0 0 1 1 1 1 1 1 1 1
USE_CUDNN*** 0 0 1 1 1 1 1 1 1 1
USE_S3 1 1 1 1 1 1 1 1 1 1
USE_HDFS 0 0 0 0 0 0 0 0 0 0
USE_OPENCV 1 1 1 1 1 1 1 1 1 1
USE_SIGNAL_HANDLER 1 1 1 1 1 1 1 1 1 1
USE_PROFILER 0 0 0 0 0 0 0 0 0 0
DEV 0 0 0 0 0 0 0 0 0 0
DEBUG 0 0 0 0 0 0 0 0 0 0

Notes:

* `USE_OPENMP` only applies to Linux

** USE_NCCL is supported since release version 1.0.0.post2, nightly since 1.0.1b20171228. Version: 2.1.2-1

*** USE_CUDNN version: 7.0.5.15-1 for cu80/cu90/cu91, 6.0.21-1 for cu75.

Usage

Release

For release, users can install it via

MXNET_VARIANT=mxnet-cu90 MXNET_VERSION=1.0.0.post2 pip install --upgrade ${MXNET_VARIANT}==${MXNET_VERSION}

Be sure to update pip (setuptools) to the most recent version, by pip install --upgrade pip.

Nightly

For nightly pre-release, users can install it via pip install --pre --upgrade. Be sure to update pip (setuptools) to the most recent version, by pip install --upgrade pip.

Report issue

For bug submissions on issues found in the pre-releases, please attach:

  • MXNet pip package version
  • Environment report:
What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.

  • Problem code
    • Code snippet to reproduce the issue
    • Code should use dummy data (e.g. mx.nd.random.uniform), unless bug is only triggered with specific data value, in which case that specific value should be attached.
@szha

This comment has been minimized.

Show comment
Hide comment
@szha
Member

szha commented Nov 30, 2017

@rahul003

This comment has been minimized.

Show comment
Hide comment
@rahul003

rahul003 Dec 8, 2017

Contributor

Is there a way to see the build flags used to build the pip packages?

Contributor

rahul003 commented Dec 8, 2017

Is there a way to see the build flags used to build the pip packages?

@szha

This comment has been minimized.

Show comment
Hide comment
@szha

szha Dec 8, 2017

Member

I will add a section for the flags

Member

szha commented Dec 8, 2017

I will add a section for the flags

@szha

This comment has been minimized.

Show comment
Hide comment
@szha
Member

szha commented Dec 13, 2017

@szha szha self-assigned this Dec 13, 2017

@szha szha added the pip label Dec 14, 2017

@adaaaaaa

This comment has been minimized.

Show comment
Hide comment
@adaaaaaa

adaaaaaa Jan 2, 2018

头像换了,不好看,还是原来的猪头像好看。。。蛤蛤。。。

adaaaaaa commented Jan 2, 2018

头像换了,不好看,还是原来的猪头像好看。。。蛤蛤。。。

@adaaaaaa

This comment has been minimized.

Show comment
Hide comment
@adaaaaaa

adaaaaaa Jan 3, 2018

https://pypi.python.org/pypi/mxnet-cu91/
Not Found
Sorry, the page you're looking for couldn't be found.
Perhaps check the URL?

adaaaaaa commented Jan 3, 2018

https://pypi.python.org/pypi/mxnet-cu91/
Not Found
Sorry, the page you're looking for couldn't be found.
Perhaps check the URL?

@szha

This comment has been minimized.

Show comment
Hide comment
@szha

szha Jan 3, 2018

Member

@adaaaaaa cu91 versions are WIP.

Member

szha commented Jan 3, 2018

@adaaaaaa cu91 versions are WIP.

@adaaaaaa

This comment has been minimized.

Show comment
Hide comment
@adaaaaaa

adaaaaaa Jan 5, 2018

cu90和cu91有没有benchmark,对比一下?

adaaaaaa commented Jan 5, 2018

cu90和cu91有没有benchmark,对比一下?

@adaaaaaa

This comment has been minimized.

Show comment
Hide comment
@adaaaaaa

adaaaaaa Jan 24, 2018

求cu91的更一下win版本的?

adaaaaaa commented Jan 24, 2018

求cu91的更一下win版本的?

@szha

This comment has been minimized.

Show comment
Hide comment
@szha

szha Jan 24, 2018

Member

pinging @yajiedesign

Member

szha commented Jan 24, 2018

pinging @yajiedesign

@adaaaaaa

This comment has been minimized.

Show comment
Hide comment
@adaaaaaa

adaaaaaa Jan 30, 2018

你@了一个不存在的人?

adaaaaaa commented Jan 30, 2018

你@了一个不存在的人?

@yzzymt

This comment has been minimized.

Show comment
Hide comment
@yzzymt

yzzymt Feb 28, 2018

realese date for cu91 for win? Xiexie

yzzymt commented Feb 28, 2018

realese date for cu91 for win? Xiexie

@KishoreKarunakaran

This comment has been minimized.

Show comment
Hide comment
@KishoreKarunakaran

KishoreKarunakaran Mar 1, 2018

@szha when can we get cu91 for win ?

KishoreKarunakaran commented Mar 1, 2018

@szha when can we get cu91 for win ?

@szha

This comment has been minimized.

Show comment
Hide comment
@szha

szha Mar 1, 2018

Member

@yajiedesign is working on it.

Member

szha commented Mar 1, 2018

@yajiedesign is working on it.

@rahul003

This comment has been minimized.

Show comment
Hide comment
@rahul003

rahul003 May 5, 2018

Contributor

@szha Can we turn on the USE_LIBJPEG_TURBO flag. I find that it helps improve the speed of IO pipeline significantly?

Results for Resnet50 v1 Imagenet, 480px resized data, batch size 1920, float16, symbolic, p3.16x
Current package: 3600samples/sec
With libjpeg-turbo: 4600samples/sec

As an example, Here's how I used LIBJPEG_TURBO on ubuntu.

sudo apt-get install autoconf automake libtool nasm
JPEG_TURBO_VERSION=1.5.2 && \
wget -q -O - https://github.com/libjpeg-turbo/libjpeg-turbo/archive/${JPEG_TURBO_VERSION}.tar.gz | tar -xzf - && \
cd libjpeg-turbo-${JPEG_TURBO_VERSION} && \
autoreconf -fiv && \
./configure --enable-shared --prefix=/usr 2>&1 >/dev/null && \
sudo make -j"$(nproc)" install 2>&1 >/dev/null && \
rm -rf libjpeg-turbo-${JPEG_TURBO_VERSION}

Flags
USE_LIBJPEG_TURBO=1 USE_LIBJPEG_TURBO_PATH=/usr

Contributor

rahul003 commented May 5, 2018

@szha Can we turn on the USE_LIBJPEG_TURBO flag. I find that it helps improve the speed of IO pipeline significantly?

Results for Resnet50 v1 Imagenet, 480px resized data, batch size 1920, float16, symbolic, p3.16x
Current package: 3600samples/sec
With libjpeg-turbo: 4600samples/sec

As an example, Here's how I used LIBJPEG_TURBO on ubuntu.

sudo apt-get install autoconf automake libtool nasm
JPEG_TURBO_VERSION=1.5.2 && \
wget -q -O - https://github.com/libjpeg-turbo/libjpeg-turbo/archive/${JPEG_TURBO_VERSION}.tar.gz | tar -xzf - && \
cd libjpeg-turbo-${JPEG_TURBO_VERSION} && \
autoreconf -fiv && \
./configure --enable-shared --prefix=/usr 2>&1 >/dev/null && \
sudo make -j"$(nproc)" install 2>&1 >/dev/null && \
rm -rf libjpeg-turbo-${JPEG_TURBO_VERSION}

Flags
USE_LIBJPEG_TURBO=1 USE_LIBJPEG_TURBO_PATH=/usr

@szha

This comment has been minimized.

Show comment
Hide comment
@szha

szha May 5, 2018

Member

@rahul003 thanks for the suggestion. I will certainly take a look. For these dependencies, my approach have been to statically link them, so some steps will be different.

Member

szha commented May 5, 2018

@rahul003 thanks for the suggestion. I will certainly take a look. For these dependencies, my approach have been to statically link them, so some steps will be different.

@xinyu-intel

This comment has been minimized.

Show comment
Hide comment
@xinyu-intel

xinyu-intel May 21, 2018

Contributor

Is mxnet-mkl 1.2.0 ready for windows?

Contributor

xinyu-intel commented May 21, 2018

Is mxnet-mkl 1.2.0 ready for windows?

@fhieber

This comment has been minimized.

Show comment
Hide comment
@fhieber

fhieber May 21, 2018

Contributor

Are the pip distributions compiled with USE_DIST_KVSTORE = 1 and if not, why?

Contributor

fhieber commented May 21, 2018

Are the pip distributions compiled with USE_DIST_KVSTORE = 1 and if not, why?

@szha

This comment has been minimized.

Show comment
Hide comment
@szha

szha May 21, 2018

Member

@fhieber Yes, they should have USE_DIST_KVSTORE=1 already. Did you have trouble using the distributed kvstore feature?

Member

szha commented May 21, 2018

@fhieber Yes, they should have USE_DIST_KVSTORE=1 already. Did you have trouble using the distributed kvstore feature?

@fhieber

This comment has been minimized.

Show comment
Hide comment
@fhieber

fhieber May 21, 2018

Contributor

@szha Oh great, thanks! I haven't tried it in a while but remembered that last time I checked the distributions were compiled without it. Glad to hear its enabled by default now!

Contributor

fhieber commented May 21, 2018

@szha Oh great, thanks! I haven't tried it in a while but remembered that last time I checked the distributions were compiled without it. Glad to hear its enabled by default now!

@szha

This comment has been minimized.

Show comment
Hide comment
@szha

szha May 21, 2018

Member

@fhieber no problem. I added the corresponding row in the table.

Member

szha commented May 21, 2018

@fhieber no problem. I added the corresponding row in the table.

@eric-haibin-lin

This comment has been minimized.

Show comment
Hide comment
@eric-haibin-lin

eric-haibin-lin Jun 18, 2018

Contributor

@szha what version of opencv is used for pip?

Contributor

eric-haibin-lin commented Jun 18, 2018

@szha what version of opencv is used for pip?

@szha

This comment has been minimized.

Show comment
Hide comment
@szha
Member

szha commented Jun 19, 2018

@yajiedesign

This comment has been minimized.

Show comment
Hide comment
@yajiedesign

yajiedesign Jun 20, 2018

Contributor

@szha the windows mkl is compiled,is update?

Contributor

yajiedesign commented Jun 20, 2018

@szha the windows mkl is compiled,is update?

@MoritzMaxeiner

This comment has been minimized.

Show comment
Hide comment
@MoritzMaxeiner

MoritzMaxeiner Aug 14, 2018

@szha What malloc are you using for pip? tcmalloc, jemalloc, or default?

MoritzMaxeiner commented Aug 14, 2018

@szha What malloc are you using for pip? tcmalloc, jemalloc, or default?

@szha

This comment has been minimized.

Show comment
Hide comment
@szha

szha Aug 14, 2018

Member

@MoritzMaxeiner default malloc so far. I intended to use jemalloc by building it from source but our makefile logic for jemalloc prevents a static link. https://github.com/apache/incubator-mxnet/blob/master/Makefile#L254-L278 (e.g. it forces to look in specific paths for libjemalloc.so).

Member

szha commented Aug 14, 2018

@MoritzMaxeiner default malloc so far. I intended to use jemalloc by building it from source but our makefile logic for jemalloc prevents a static link. https://github.com/apache/incubator-mxnet/blob/master/Makefile#L254-L278 (e.g. it forces to look in specific paths for libjemalloc.so).

@MoritzMaxeiner

This comment has been minimized.

Show comment
Hide comment
@MoritzMaxeiner

MoritzMaxeiner Aug 14, 2018

@szha Thank you very for the info, I suspected as much.

OT: I was just making sure, since I use MXNet from a shared object (dynamically loaded at runtime by another executable that I can't change to tcmalloc or jemalloc, which links dynamically against OpenCV), and using tcmalloc in MXNet leads to these issues with the LD_PRELOAD workaround not working (freezing the executable), so I was wondering if the prebuilt binary versions had some workaround for that.

MoritzMaxeiner commented Aug 14, 2018

@szha Thank you very for the info, I suspected as much.

OT: I was just making sure, since I use MXNet from a shared object (dynamically loaded at runtime by another executable that I can't change to tcmalloc or jemalloc, which links dynamically against OpenCV), and using tcmalloc in MXNet leads to these issues with the LD_PRELOAD workaround not working (freezing the executable), so I was wondering if the prebuilt binary versions had some workaround for that.

@cconvey

This comment has been minimized.

Show comment
Hide comment
@cconvey

cconvey Oct 4, 2018

@szha : I'm trying to locally reproduce the mxnet and mxnet-mkl wheels that end up on PyPI. (Linux and OS X only.) Regarding the process you guys use:

  1. Is there anything noteworthy about the platforms and/or steps used to prepare the wheels?

  2. What tests (if any) do you run to verify that a wheel is good prior to upload?

Thanks!

cconvey commented Oct 4, 2018

@szha : I'm trying to locally reproduce the mxnet and mxnet-mkl wheels that end up on PyPI. (Linux and OS X only.) Regarding the process you guys use:

  1. Is there anything noteworthy about the platforms and/or steps used to prepare the wheels?

  2. What tests (if any) do you run to verify that a wheel is good prior to upload?

Thanks!

@szha

This comment has been minimized.

Show comment
Hide comment
@szha

szha Oct 4, 2018

Member

@cconvey I use ubuntu 14.04 as the environment because I want the GLIBC to be old enough for portability. I statically link all the dependencies too. For verification, if I changed any dependency, I do a manual verification of the new wheel. For all wheels, I run unit-test as sanity test before any upload.

Member

szha commented Oct 4, 2018

@cconvey I use ubuntu 14.04 as the environment because I want the GLIBC to be old enough for portability. I statically link all the dependencies too. For verification, if I changed any dependency, I do a manual verification of the new wheel. For all wheels, I run unit-test as sanity test before any upload.

@cconvey

This comment has been minimized.

Show comment
Hide comment
@cconvey

cconvey Oct 15, 2018

@szha When building the wheels, do you use the provided Makefiles or CMake?

edit for clarification: Do you use the script tools/pip_package/make_pip_package.sh to build the wheels that end up on PyPI?

cconvey commented Oct 15, 2018

@szha When building the wheels, do you use the provided Makefiles or CMake?

edit for clarification: Do you use the script tools/pip_package/make_pip_package.sh to build the wheels that end up on PyPI?

@szha

This comment has been minimized.

Show comment
Hide comment
@szha

szha Oct 15, 2018

Member

I'm using Makefile along with a separate set of scripts.

Member

szha commented Oct 15, 2018

I'm using Makefile along with a separate set of scripts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment