Conda Environment Install Issue

[kleingeo]

Trying to get DeepSpeed installed for local use with a Conda environment, but it seems that DeepSpeed in not installing to the environment itself. After building the wheel DeepSpeed is not installing into the proper Conda conda environment location. Apex is installing in the proper environment location. Unclear why DeepSpeed is not working but Apex is.

[ShadenSmith]

Hi there! We're in the process of rewriting our installation scripts (that were previously only used within Docker containers) and hoping to also release a conda package in short time. These sorts of issues should be fixed at that point.

[msdejong]

The installation scripts perform sudo -H pip installs, which install system wide. I replaced those with normal pip installs and it installed into the current environment without problems.

[kleingeo]

When I installed it I did that in the install script (for both the deep speed, apex and requirements). However, there were still issues in that DeepSpeed would not install to the right environment location. Looking at the installation a little more, this seemed more likely an issue with the wheel created for DeepSpeed in the install.sh file. I was able to get it working by forcing pip to install DeepSpeed into the correct location (the same location that Apex was correctly installed to).

[ShadenSmith]

We have a now have a conda package uploaded and we appreciate any feedback!

We have versions compiled for cudatoolkit versions 10.0 and 10.1 To install along with pytorch and other dependencies that are in the conda-forge channel:

conda install deepspeed cudatoolkit=10.1 -c deepspeed -c pytorch -c conda-forge

[ShadenSmith]

The repo's install.sh should respect the environment by default now (sudo is opt-in). Please let me know if the issue persists.

[kleingeo]

Using the conda install, deepspeed shows up when I run conda list but it is not available when trying to import in python.

[ShadenSmith]

Hi @kleingeo, thanks for the report. I can see that on my end now as well. Not sure what happened...I'm looking into it.

Interestingly, the deepspeed entry point looks fine and is found in my $PATH after installation. And I can see the DeepSpeed library installed under ~/miniconda3/envs/test/lib/python3.7/site-packages/deepspeed/ (where test is my conda environment name), and also see the expected ~/miniconda3/lib/python3.7/site-packages in my sys.path...so I'm not sure why the deepspeed library is not importable.

[kleingeo]

Yes, I remember having this problem a lot when trying to install deepspeed normally with the install.sh file. With a normal python virtual env it works, but for some reason with Conda, it consistently tries to install to another location. The only thing I found to work was to force pip (when using conda) to force the install location to where the install.sh file installs Apex.

[jdongca2003]

@ShadenSmith , it is easier to install deepspeed via your conda command than 'install.sh' (prone to fail). In the deepspeed channel, only early-version deepspeed exists.

conda search -f deepspeed -c deepspeed
Loading channels: done
deepspeed 0.1.0 py3.6_cuda10.0.130_0 deepspeed
deepspeed 0.1.0 py3.6_cuda10.1.243_0 deepspeed
deepspeed 0.1.0 py3.7_cuda10.0.130_0 deepspeed
deepspeed 0.1.0 py3.7_cuda10.1.243_0 deepspeed

When do you plan to release new conda version of deepspeed with Zero2?

Thanks

[ShadenSmith]

Hi @jdongca2003, I have some time to dedicate to the DeepSpeed's conda infrastructure now that the v0.2 release is complete. I'm looking at improved packages (per the above bug report) and automating the package build process.

[jdongca2003]

@ShadenSmith Thanks. I tested your conda deepspeed package on https://github.com/microsoft/DeepSpeedExamples/tree/master/cifar.
It failed on Tesla K80 and I got the following error mesage:
"
THCudaCheck FAIL file=csrc/fused_adam_cuda_kernel.cu line=135 error=209 : no kernel image is available for execution on the device
Traceback (most recent call last):
File "cifar10_deepspeed.py", line 178, in
model_engine.step()
File "/home/dong/miniconda3/envs/deepspeed/lib/python3.7/site-packages/deepspeed/pt/deepspeed_light.py", line 692, in step
self.optimizer.step()
File "/home/dong/miniconda3/envs/deepspeed/lib/python3.7/site-packages/apex/optimizers/fused_adam.py", line 146, in step
group['weight_decay'])
RuntimeError: cuda runtime error (209) : no kernel image is available for execution on the device at csrc/fused_adam_cuda_kernel.cu:135"

But it worked well on Tesla P4. Probably deepspeed does not support old GPU architecture.

[analog75]

In V100, Same error with THCudaChecker happens!!

[ConnollyLeon]

@ShadenSmith Thanks. I tested your conda deepspeed package on https://github.com/microsoft/DeepSpeedExamples/tree/master/cifar.
It failed on Tesla K80 and I got the following error mesage:
"
THCudaCheck FAIL file=csrc/fused_adam_cuda_kernel.cu line=135 error=209 : no kernel image is available for execution on the device
Traceback (most recent call last):
File "cifar10_deepspeed.py", line 178, in
model_engine.step()
File "/home/dong/miniconda3/envs/deepspeed/lib/python3.7/site-packages/deepspeed/pt/deepspeed_light.py", line 692, in step
self.optimizer.step()
File "/home/dong/miniconda3/envs/deepspeed/lib/python3.7/site-packages/apex/optimizers/fused_adam.py", line 146, in step
group['weight_decay'])
RuntimeError: cuda runtime error (209) : no kernel image is available for execution on the device at csrc/fused_adam_cuda_kernel.cu:135"

But it worked well on Tesla P4. Probably deepspeed does not support old GPU architecture.

Hi @jdongca2003 ,
I encounter the same problem as you describe when using Tesla K80. And I found it work normally when applying them on Tesla V100. Have you solved this problem?

@ShadenSmith Could you please explain why this happen? Dose deepspeed not support Tesla K80?

Thanks.

[loadams]

Hi, closing this issue as it is stale with respect to Cuda/Torch/DeepSpeed versions. However, we now provide an environment.yml for ease of building in conda, that is located at the root of our repo!