Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simple Infer pickle.load EOF error #2

Open
jaskarannagi19 opened this issue Aug 2, 2018 · 12 comments
Open

Simple Infer pickle.load EOF error #2

jaskarannagi19 opened this issue Aug 2, 2018 · 12 comments

Comments

@jaskarannagi19
Copy link

jaskarannagi19 commented Aug 2, 2018

(Pose) jaskaran@jaskaran-pc:~/densepose$ python tools/infer_simple.py --cfg configs/DensePose_ResNet101_FPN_s1x-e2e.yaml --output-dir DensePoseData/infer_out/ --image-ext jpg --wts DensePose_ResNet101_FPN_s1x-e2e.pkl DensePoseData/demo_data/demo_im2.jpg
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/h5py/init.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._conv import register_converters as _register_converters
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/h5py/init.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from . import h5a, h5d, h5ds, h5f, h5fd, h5g, h5r, h5s, h5t, h5p, h5z
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/h5py/_hl/group.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .. import h5g, h5i, h5o, h5r, h5t, h5l, h5p
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/linalg/basic.py:17: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._solve_toeplitz import levinson
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/linalg/init.py:207: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._decomp_update import *
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/special/init.py:640: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._ufuncs import *
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/special/_ellip_harm.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._ellip_harm_2 import _ellipsoid, _ellipsoid_norm
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/interpolate/_bsplines.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from . import _bspl
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/sparse/lil.py:19: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from . import _csparsetools
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/sparse/csgraph/init.py:165: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._shortest_path import shortest_path, floyd_warshall, dijkstra,
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/sparse/csgraph/_validation.py:5: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._tools import csgraph_to_dense, csgraph_from_dense,
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/sparse/csgraph/init.py:167: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._traversal import breadth_first_order, depth_first_order,
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/sparse/csgraph/init.py:169: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._min_spanning_tree import minimum_spanning_tree
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/sparse/csgraph/init.py:170: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._reordering import reverse_cuthill_mckee, maximum_bipartite_matching,
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/spatial/init.py:95: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .ckdtree import *
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/spatial/init.py:96: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .qhull import *
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/spatial/_spherical_voronoi.py:18: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from . import _voronoi
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/spatial/distance.py:122: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from . import _hausdorff
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/io/matlab/mio4.py:18: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .mio_utils import squeeze_element, chars_to_strings
/home/jaskaran/.conda/envs/Pose/lib/python2.7/site-packages/scipy/io/matlab/mio5.py:98: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .mio5_utils import VarReader5
Found Detectron ops lib: /home/jaskaran/.conda/envs/Pose/lib/libcaffe2_detectron_ops_gpu.so
E0803 01:04:59.011703 10154 init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
E0803 01:04:59.011721 10154 init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
E0803 01:04:59.011726 10154 init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
WARNING cnn.py: 25: [====DEPRECATE WARNING====]: you are creating an object from CNNModelHelper class which will be deprecated soon. Please use ModelHelper object with brew module. For more information, please refer to caffe2.ai and python/brew.py, python/brew_test.py for more information.
INFO net.py: 51: Loading weights from: DensePose_ResNet101_FPN_s1x-e2e.pkl
Traceback (most recent call last):
File "tools/infer_simple.py", line 140, in
main(args)
File "tools/infer_simple.py", line 91, in main
model = infer_engine.initialize_model_from_cfg(args.weights)
File "/home/jaskaran/densepose/detectron/core/test_engine.py", line 336, in initialize_model_from_cfg
model, weights_file, gpu_id=gpu_id,
File "/home/jaskaran/densepose/detectron/utils/net.py", line 54, in initialize_gpu_from_weights_file
src_blobs = pickle.load(f)
EOFError

Please see if you can help out in this

@siamaksalman
Copy link
Owner

What is your setup ? OS ? GPU ? CUDA version ? python version ?

@jaskarannagi19
Copy link
Author

jaskarannagi19 commented Aug 3, 2018

@siamaksalman
OS: Ubuntu 16.04
GPU: Nvidia GTX 1050
CUDA: 8.0
Python: 2.7 from anaconda

@siamaksalman
Copy link
Owner

siamaksalman commented Aug 4, 2018

@jaskarannagi19
As I mentions in the guide, the official docker file of DensePose uses CUDA 9.0 with cudnn7, I think installing cuda 9 will solve your problem.

Use this command to make sure your caffe2 can access your CUDA:

python2 -c 'from caffe2.python import     workspace; print    (workspace.NumCudaDevices())'

@jaskarannagi19
Copy link
Author

@siamaksalman
Will I be need to reinstall caffe 2 as well ? If I update my cuda version?

@jaskarannagi19
Copy link
Author

Yes caffe 2 is able to access my cuda and shows 1 for GPU command

@siamaksalman
Copy link
Owner

I don't thing reinstalling caffe2 is needed. did you upgrade your CUDA version?

@jaskarannagi19
Copy link
Author

jaskarannagi19 commented Aug 4, 2018 via email

@jaskarannagi19
Copy link
Author

jaskarannagi19 commented Aug 5, 2018

@siamaksalman
updating cuda didnt help either now cua is 9.0 cudnn is 7 and nccl is 2 i did reinstalled caffe2 like this

conda install -c caffe2 caffe2-cuda9.0-cudnn7

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

python2 -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure"
this says success

python2 -c 'from caffe2.python import workspace; print(workspace.NumCudaDevices())
says 1

@siamaksalman
Copy link
Owner

oh my bad, yes also you needed to install caffe2-cuda9.0-cudnn7. do you still get the same error while running the script ?

@jaskarannagi19
Copy link
Author

jaskarannagi19 commented Aug 5, 2018 via email

@siamaksalman
Copy link
Owner

siamaksalman commented Aug 8, 2018

I think your setup is okey, maybe downloading DensePose latest commit will do the job.
Latest commit e118a4c 6 days ago

@jaskarannagi19
Copy link
Author

@siamaksalman some one suggested its a memory issue I'll download llighter weight files to confirm but I am 80% sure that when its memory issue pickle says EOFerror ran out of input memory. Running very high fever need to take break before I redo my Ubuntu and follow all steps you listed here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants