A few minor fixes #12

r9y9 · 2018-01-28T07:02:29Z

Please see commit messages for details.

this avoids creating an unnecessary directory `warp_ctc` in the top of the pytorch_binding Quote from: http://pytorch.org/docs/master/ffi.html package : set to True to build in package mode (for modules meant to be installed as pip packages

SeanNaren · 2018-02-16T11:28:51Z

Thanks man!

SeanNaren · 2018-02-16T11:58:06Z

@r9y9 I've made a PR to revert the changes, I'm seeing some errors in the building of the library, could you help debug this? How do you currently build this lib?

r9y9 · 2018-02-16T15:42:37Z

I used python setup.py develop. What errors are you seeing?

SeanNaren · 2018-02-16T18:10:12Z

Doing a python setup.py install and then running the tests end with a strange error message about lib, do you not see that on your side? I'll open up a new PR that reverts the revert :D

r9y9 · 2018-02-19T03:47:11Z

I'm currently in a business trip and cannot have access to my PC (with GTX 1080 Ti). I will try to investigate when I get back.

r9y9 · 2018-02-24T08:26:40Z

Sorry for late; I'm back. I cannot reproduce. Could you tell me what exactly you are seeing?

CUDA_HOME=/usr/local/cuda python setup.py install

generating build/warpctc_pytorch/_warp_ctc/__warp_ctc.c
regenerated: 'build/warpctc_pytorch/_warp_ctc/__warp_ctc.c'
running install
running bdist_egg
running egg_info
writing warpctc_pytorch.egg-info/PKG-INFO
writing dependency_links to warpctc_pytorch.egg-info/dependency_links.txt
writing top-level names to warpctc_pytorch.egg-info/top_level.txt
reading manifest file 'warpctc_pytorch.egg-info/SOURCES.txt'
writing manifest file 'warpctc_pytorch.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/warpctc_pytorch
copying warpctc_pytorch/__init__.py -> build/lib.linux-x86_64-3.6/warpctc_pytorch
running build_ext
building 'warpctc_pytorch._warp_ctc.__warp_ctc' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/build
creating build/temp.linux-x86_64-3.6/build/warpctc_pytorch
creating build/temp.linux-x86_64-3.6/build/warpctc_pytorch/_warp_ctc
creating build/temp.linux-x86_64-3.6/home
creating build/temp.linux-x86_64-3.6/home/ryuichi
creating build/temp.linux-x86_64-3.6/home/ryuichi/Dropbox
creating build/temp.linux-x86_64-3.6/home/ryuichi/Dropbox/numeric
creating build/temp.linux-x86_64-3.6/home/ryuichi/Dropbox/numeric/warp-ctc
creating build/temp.linux-x86_64-3.6/home/ryuichi/Dropbox/numeric/warp-ctc/pytorch_binding
creating build/temp.linux-x86_64-3.6/home/ryuichi/Dropbox/numeric/warp-ctc/pytorch_binding/src
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/ryuichi/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include -I/home/ryuichi/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/ryuichi/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC -I/usr/local/cuda/include -I/home/ryuichi/Dropbox/numeric/warp-ctc/include -I/home/ryuichi/anaconda3/include/python3.6m -c build/warpctc_pytorch/_warp_ctc/__warp_ctc.c -o build/temp.linux-x86_64-3.6/build/warpctc_pytorch/_warp_ctc/__warp_ctc.o -std=c++11 -fPIC -DWARPCTC_ENABLE_GPU
cc1: warning: command line option ‘-std=c++11’ is valid for C++/ObjC++ but not for C
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/ryuichi/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include -I/home/ryuichi/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/ryuichi/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC -I/usr/local/cuda/include -I/home/ryuichi/Dropbox/numeric/warp-ctc/include -I/home/ryuichi/anaconda3/include/python3.6m -c /home/ryuichi/Dropbox/numeric/warp-ctc/pytorch_binding/src/binding.cpp -o build/temp.linux-x86_64-3.6/home/ryuichi/Dropbox/numeric/warp-ctc/pytorch_binding/src/binding.o -std=c++11 -fPIC -DWARPCTC_ENABLE_GPU
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
creating build/lib.linux-x86_64-3.6/warpctc_pytorch/_warp_ctc
g++ -pthread -shared -L/home/ryuichi/anaconda3/lib -Wl,-rpath=/home/ryuichi/anaconda3/lib,--no-as-needed build/temp.linux-x86_64-3.6/build/warpctc_pytorch/_warp_ctc/__warp_ctc.o build/temp.linux-x86_64-3.6/home/ryuichi/Dropbox/numeric/warp-ctc/pytorch_binding/src/binding.o -L/home/ryuichi/Dropbox/numeric/warp-ctc/build -L/home/ryuichi/anaconda3/lib -lwarpctc -lpython3.6m -o build/lib.linux-x86_64-3.6/warpctc_pytorch/_warp_ctc/__warp_ctc.cpython-36m-x86_64-linux-gnu.so -Wl,-rpath,/home/ryuichi/Dropbox/numeric/warp-ctc/build
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/warpctc_pytorch
copying build/lib.linux-x86_64-3.6/warpctc_pytorch/__init__.py -> build/bdist.linux-x86_64/egg/warpctc_pytorch
creating build/bdist.linux-x86_64/egg/warpctc_pytorch/_warp_ctc
copying build/lib.linux-x86_64-3.6/warpctc_pytorch/_warp_ctc/__warp_ctc.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/warpctc_pytorch/_warp_ctc
byte-compiling build/bdist.linux-x86_64/egg/warpctc_pytorch/__init__.py to __init__.cpython-36.pyc
creating stub loader for warpctc_pytorch/_warp_ctc/__warp_ctc.cpython-36m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/warpctc_pytorch/_warp_ctc/__warp_ctc.py to __warp_ctc.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying warpctc_pytorch.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying warpctc_pytorch.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying warpctc_pytorch.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying warpctc_pytorch.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
warpctc_pytorch._warp_ctc.__pycache__.__warp_ctc.cpython-36: module references __file__
creating dist
creating 'dist/warpctc_pytorch-0.1-py3.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing warpctc_pytorch-0.1-py3.6-linux-x86_64.egg
removing '/home/ryuichi/anaconda3/lib/python3.6/site-packages/warpctc_pytorch-0.1-py3.6-linux-x86_64.egg' (and everything under it)
creating /home/ryuichi/anaconda3/lib/python3.6/site-packages/warpctc_pytorch-0.1-py3.6-linux-x86_64.egg
Extracting warpctc_pytorch-0.1-py3.6-linux-x86_64.egg to /home/ryuichi/anaconda3/lib/python3.6/site-packages
warpctc-pytorch 0.1 is already the active version in easy-install.pth

Installed /home/ryuichi/anaconda3/lib/python3.6/site-packages/warpctc_pytorch-0.1-py3.6-linux-x86_64.egg
Processing dependencies for warpctc-pytorch==0.1
Finished processing dependencies for warpctc-pytorch==0.1

python tests/test.py

Variable containing:
 0.1770 -0.7081  0.1770  0.1770  0.1770
-0.8230  0.1770  0.2919  0.1770  0.1770
 0.2919  0.1770 -0.8230  0.1770  0.1770
-0.8213  0.1787  0.2665  0.1975  0.1787
[torch.cuda.FloatTensor of size 4x5 (GPU 0)]

.Variable containing:
 0.1770 -0.7081  0.1770  0.1770  0.1770
 0.1770 -0.8230  0.2919  0.1770  0.1770
 0.2919  0.1770 -0.8230  0.1770  0.1770
 0.1787  0.1787 -0.7335  0.1975  0.1787
[torch.cuda.FloatTensor of size 4x5 (GPU 0)]

.Variable containing:
 0.0000  0.0000  0.0000  0.0000  0.0000
 0.0000 -1.0000  1.0000  0.0000  0.0000
 1.0000  0.0000 -1.0000  0.0000  0.0000
 0.0000  0.0000  0.0000  0.0000  0.0000
[torch.cuda.FloatTensor of size 4x5 (GPU 0)]

.Variable containing:
 0.1770 -0.7081  0.1770  0.1770  0.1770
 0.1770  0.1770 -0.7081  0.1770  0.1770
[torch.cuda.FloatTensor of size 2x5 (GPU 0)]

.
----------------------------------------------------------------------
Ran 4 tests in 3.011s

OK

python -c "import torch; print(torch.__version__)" 
0.3.1b0+94ba828

SeanNaren · 2018-02-24T11:11:32Z

Hmm I wonder if it was just my install, I'll retry again! Sorry about this!

r9y9 · 2018-03-02T17:49:14Z

Do you find any problem? or can we just merge #17?

SeanNaren · 2018-03-05T10:39:28Z

On a fresh install of anaconda, running the build commands:

cd warp-ctc
mkdir build
cd build
cmake ..
make

The installing the bindings:

cd pytorch_binding
CUDA_HOME=/usr/local/cuda python setup.py install

Run into this notorious issue when running the tests:

python test.py
Traceback (most recent call last):
  File "test.py", line 8, in <module>
    from warpctc_pytorch import CTCLoss
  File "/home/sean.narenthiran/anaconda3/lib/python3.6/site-packages/warpctc_pytorch-0.1-py3.6-linux-x86_64.egg/warpctc_pytorch/__init__.py", line 7, in <module>
    from ._warp_ctc import lib as _lib, ffi as _ffi
ImportError: cannot import name 'lib'

You sure you don't run into this issue as well?

This is causing error on python2. fixes SeanNaren#12 (comment)

r9y9 · 2018-03-08T16:08:11Z

I can reproduce on pyhon2.7. Fix comming.

r9y9 · 2018-03-08T16:12:18Z

@SeanNaren I think I fixed the problem. Could you try if it works?

bstriner · 2018-03-14T09:30:13Z

I think this pull breaks things, at least for me on python 3. The correct code on my machine ends up as:

from ._warp_ctc.__warp_ctc import lib as _lib, ffi as _ffi

Not sure what is causing the extra level there.

r9y9 · 2018-03-17T14:50:30Z

@bstriner I am wondering if you had old generated code as well? Could you try it again after rm -rf warpctc_pytorch && git checkout . and python setup.py install?

SeanNaren · 2018-03-19T14:49:02Z

what @r9y9 just suggested is what gave me weird install issues as well. Make sure to uninstall/remove the site package from the conda environment, and re-clone the repo.

bstriner · 2018-03-19T15:39:30Z

Actually, this was a Dockerfile, so no previous anything. Fresh git clone, install, etc.

Current version is here (using a fork of the repo). If you switch the url back to the master repo it should reproduce the issue.

https://github.com/cmudeeplearning11785/machine_learning_gpu/blob/master/Dockerfile

On that note, @SeanNaren any thoughts on setting up Travis or something similar?

bstriner · 2018-03-19T15:41:26Z

I tried a few different ways to install but all got the same result (you can see some in the comments in the Dockerfile).

bstriner · 2018-03-19T15:52:49Z

If you enable travis I can start putting a config file together.

r9y9 added 3 commits January 28, 2018 15:45

pytorch_binding: Use setuptools instead of distutils

56d21a5

Use idiomatic way to install FFI

b809242

this avoids creating an unnecessary directory `warp_ctc` in the top of the pytorch_binding Quote from: http://pytorch.org/docs/master/ffi.html package : set to True to build in package mode (for modules meant to be installed as pip packages

Add gitignore for python package

f535520

SeanNaren merged commit 745281f into SeanNaren:pytorch_bindings Feb 16, 2018

r9y9 deleted the py-fixes branch February 16, 2018 11:29

SeanNaren mentioned this pull request Feb 16, 2018

Revert "A few minor fixes" #15

Merged

r9y9 added a commit to r9y9/warp-ctc that referenced this pull request Mar 8, 2018

Remove auto-generated parts from __init__.py

88db461

This is causing error on python2. fixes SeanNaren#12 (comment)

r9y9 mentioned this pull request Mar 8, 2018

Remove auto-generated parts from __init__.py #27

Merged

bstriner mentioned this pull request Apr 2, 2018

Please enable travis #36

Closed

ronybanerjee93 mentioned this pull request Apr 6, 2019

Enormous CUDA memory allocation #118

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A few minor fixes #12

A few minor fixes #12

r9y9 commented Jan 28, 2018

SeanNaren commented Feb 16, 2018

SeanNaren commented Feb 16, 2018 •

edited

Loading

r9y9 commented Feb 16, 2018

SeanNaren commented Feb 16, 2018 •

edited

Loading

r9y9 commented Feb 19, 2018

r9y9 commented Feb 24, 2018 •

edited

Loading

SeanNaren commented Feb 24, 2018

r9y9 commented Mar 2, 2018

SeanNaren commented Mar 5, 2018 •

edited

Loading

r9y9 commented Mar 8, 2018

r9y9 commented Mar 8, 2018

bstriner commented Mar 14, 2018

r9y9 commented Mar 17, 2018

SeanNaren commented Mar 19, 2018

bstriner commented Mar 19, 2018

bstriner commented Mar 19, 2018

bstriner commented Mar 19, 2018

A few minor fixes #12

A few minor fixes #12

Conversation

r9y9 commented Jan 28, 2018

SeanNaren commented Feb 16, 2018

SeanNaren commented Feb 16, 2018 • edited Loading

r9y9 commented Feb 16, 2018

SeanNaren commented Feb 16, 2018 • edited Loading

r9y9 commented Feb 19, 2018

r9y9 commented Feb 24, 2018 • edited Loading

SeanNaren commented Feb 24, 2018

r9y9 commented Mar 2, 2018

SeanNaren commented Mar 5, 2018 • edited Loading

r9y9 commented Mar 8, 2018

r9y9 commented Mar 8, 2018

bstriner commented Mar 14, 2018

r9y9 commented Mar 17, 2018

SeanNaren commented Mar 19, 2018

bstriner commented Mar 19, 2018

bstriner commented Mar 19, 2018

bstriner commented Mar 19, 2018

SeanNaren commented Feb 16, 2018 •

edited

Loading

SeanNaren commented Feb 16, 2018 •

edited

Loading

r9y9 commented Feb 24, 2018 •

edited

Loading

SeanNaren commented Mar 5, 2018 •

edited

Loading