Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault #35

Closed
JZArray opened this issue Jun 17, 2024 · 8 comments
Closed

Segmentation fault #35

JZArray opened this issue Jun 17, 2024 · 8 comments

Comments

@JZArray
Copy link

JZArray commented Jun 17, 2024

Hello guys, nice work! But I have a problem while running the demo.

loaded weight from ./pretrained_models/hallo/net.pth
Segmentation fault (core dumped)

How do I solve this issue?

Thanks in advance!

@xumingw
Copy link
Contributor

xumingw commented Jun 17, 2024

Could you please provide more details about the error?

@JZArray
Copy link
Author

JZArray commented Jun 17, 2024

Could you please provide more details about the error?

Some weights of the model checkpoint were not used when initializing UNet2DConditionModel:
['conv_norm_out.bias, conv_norm_out.weight, conv_out.bias, conv_out.weight']
INFO:hallo.models.unet_3d:loaded temporal unet's pretrained weights from pretrained_models/stable-diffusion-v1-5/unet ...
The config attributes {'center_input_sample': False} were passed to UNet3DConditionModel, but are not expected and will be ignored. Please verify your config.json configuration file.
Load motion module params from pretrained_models/motion_module/mm_sd_v15_v2.ckpt
INFO:hallo.models.unet_3d:Loaded 453.20928M-parameter motion module
loaded weight from ./pretrained_models/hallo/net.pth
Fatal Python error: Segmentation fault

Thread 0x00007f0ec00ff640 (most recent call first):
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/threading.py", line 324 in wait
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/threading.py", line 600 in wait
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/tqdm/_monitor.py", line 60 in run
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/threading.py", line 1009 in _bootstrap_inner
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/threading.py", line 966 in _bootstrap

Current thread 0x00007f126a8fa740 (most recent call first):
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456 in _conv_forward
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460 in forward
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/hallo/models/resnet.py", line 63 in forward
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520 in _call_impl
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511 in _wrapped_call_impl
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/hallo/models/face_locator.py", line 108 in forward
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520 in _call_impl
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511 in _wrapped_call_impl
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/hallo/animate/face_animate.py", line 342 in call
File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115 in decorate_context
File "/xxxx/hallo-main/scripts/inference.py", line 327 in inference_process
File "/xxxx/hallo-main/scripts/inference.py", line 379 in

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, yaml._yaml, PIL._imaging, google._upb._message, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._cdflib, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, regex._regex, av._core, av.logging, av.bytesource, av.buffer, av.audio.format, av.enum, av.error, av.utils, av.option, av.descriptor, av.container.pyio, av.dictionary, av.format, av.stream, av.container.streams, av.sidedata.motionvectors, av.sidedata.sidedata, av.packet, av.container.input, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video.frame, av.video.stream, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.audio.fifo, av.filter.pad, av.filter.link, av.filter.context, av.filter.graph, av.filter.filter, av.audio.resampler, av.bitstream, kiwisolver._cext, _cffi_backend, PIL._imagingft, skimage._shared.geometry, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, sklearn.__check_build._check_build, psutil._psutil_linux, psutil._psutil_posix, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.utils._random, sklearn.utils._seq_dataset, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, sklearn.decomposition._online_lda_fast, sklearn.decomposition._cdnmf_fast, skimage.measure._ccomp, insightface.thirdparty.face3d.mesh.cython.mesh_core_cython, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, PIL._imagingmath, scipy.signal._sigtools, scipy.signal._max_len_seq_inner, scipy.signal._upfirdn_apply, scipy.signal._spline, scipy.signal._sosfilt, scipy.signal._spectral, scipy.signal._peak_finding_utils, soxr.cysoxr, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, msgpack._cmsgpack (total: 236)
Segmentation fault (core dumped)

@subazinga
Copy link
Contributor

Which GPU do you use? And can you also list the version of all packages?

@JZArray
Copy link
Author

JZArray commented Jun 18, 2024

I am using A100.
absl-py 2.1.0
accelerate 0.28.0
albucore 0.0.11
albumentations 1.4.8
alembic 1.13.1
aniso8601 9.0.1
anna 0.1.0
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
astroid 3.2.2
attrs 23.2.0
audio-separator 0.17.2
audioread 3.0.1
av 12.1.0
beartype 0.18.5
bitsandbytes 0.43.1
blinker 1.8.2
bzip2 1.0.8
ca-certificates 2024.3.11
cachetools 5.3.3
certifi 2024.6.2
cffi 1.16.0
cfgv 3.4.0
charset-normalizer 3.3.2
click 8.1.7
cloudpickle 3.0.0
coloredlogs 15.0.1
contextlib2 21.6.0
contourpy 1.2.1
cycler 0.12.1
Cython 3.0.10
decorator 4.4.2
decord 0.6.0
Deprecated 1.2.14
diffq 0.2.4
diffusers 0.27.2
dill 0.3.8
distlib 0.3.8
docker 7.1.0
easydict 1.13
einops 0.8.0
entrypoints 0.4
filelock 3.15.1
Flask 3.0.3
flatbuffers 24.3.25
fonttools 4.53.0
fsspec 2024.6.0
gitdb 4.0.11
GitPython 3.1.43
graphene 3.3
graphql-core 3.2.3
graphql-relay 3.2.0
greenlet 3.0.3
gunicorn 22.0.0
huggingface-hub 0.23.4
humanfriendly 10.0
identify 2.5.36
idna 3.7
imageio 2.34.1
imageio-ffmpeg 0.5.1
importlib_metadata 7.1.0
insightface 0.7.3
isort 5.13.2
itsdangerous 2.2.0
jax 0.4.29
jaxlib 0.4.29
Jinja2 3.1.4
joblib 1.4.2
julius 0.2.7
kiwisolver 1.4.5
lazy_loader 0.4
ld_impl_linux-64 2.38
libffi 3.3
libgcc-ng 9.1.0
librosa 0.10.2.post1
libstdcxx-ng 9.1.0
libuuid 1.0.3
llvmlite 0.43.0
Mako 1.3.5
Markdown 3.6
MarkupSafe 2.1.5
matplotlib 3.9.0
mccabe 0.7.0
mediapipe 0.10.14
ml-dtypes 0.4.0
ml_collections 0.1.1
mlflow 2.13.1
moviepy 1.0.3
mpmath 1.3.0
msgpack 1.0.8
ncurses 6.3
networkx 3.3
nodeenv 1.9.1
numba 0.60.0
numpy 1.26.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.19.3
nvidia-nvjitlink-cu12 12.5.40
nvidia-nvtx-cu12 12.1.105
omegaconf 2.3.0
onnx 1.16.1
onnx2torch 1.5.14
onnxruntime 1.18.0
opencv-contrib-python 4.9.0.80
opencv-python 4.9.0.80
opencv-python-headless 4.9.0.80
openssl 1.1.1w
opentelemetry-api 1.25.0
opentelemetry-sdk 1.25.0
opentelemetry-semantic-conventions 0.46b0
opt-einsum 3.3.0
packaging 24.1
pandas 2.2.2
pillow 10.3.0
pip 24.0
pip 24.0
platformdirs 4.2.2
pooch 1.8.2
pre-commit 3.7.1
prettytable 3.10.0
proglog 0.1.10
protobuf 4.25.3
psutil 5.9.8
pyarrow 15.0.2
pycparser 2.22
pydantic 2.7.4
pydantic_core 2.18.4
pydub 0.25.1
pylint 3.2.2
pyparsing 3.1.2
python 3.10.4
python-dateutil 2.9.0.post0
pytz 2024.1
PyYAML 6.0.1
querystring-parser 1.2.4
readline 8.1.2
regex 2024.5.15
requests 2.32.3
resampy 0.4.3
rotary-embedding-torch 0.6.2
safetensors 0.4.3
samplerate 0.1.0
scikit-image 0.23.2
scikit-learn 1.5.0
scipy 1.13.1
setuptools 69.5.1
setuptools 70.0.0
six 1.16.0
smmap 5.0.1
sounddevice 0.4.7
soundfile 0.12.1
soxr 0.3.7
SQLAlchemy 2.0.30
sqlite 3.38.5
sqlparse 0.5.0
sympy 1.12.1
threadpoolctl 3.5.0
tifffile 2024.5.22
tk 8.6.12
tokenizers 0.15.2
tomli 2.0.1
tomlkit 0.12.5
torch 2.2.2
torchvision 0.17.2
tqdm 4.66.4
transformers 4.39.2
triton 2.2.0
typing_extensions 4.12.2
tzdata 2024.1
tzdata 2024a
urllib3 2.2.1
virtualenv 20.26.2
wcwidth 0.2.13
Werkzeug 3.0.3
wheel 0.43.0
wheel 0.43.0
wrapt 1.16.0
xformers 0.0.25.post1
xz 5.2.5
zipp 3.19.2
zlib 1.2.12

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0

CUDA Version: 12.2

@subazinga @xumingw

@JZArray
Copy link
Author

JZArray commented Jun 18, 2024

solved

@JZArray JZArray closed this as completed Jun 18, 2024
@gtfaiwxm
Copy link

gtfaiwxm commented Jul 2, 2024

how to solve the problem?

@JZArray
Copy link
Author

JZArray commented Jul 2, 2024

how to solve the problem?

@gtfaiwxm I use torch-1.12, then it can work. I think it is due to the conflict between my CUDA(11.x) and torch 2.0.x in my case.

@gtfaiwxm
Copy link

gtfaiwxm commented Jul 2, 2024

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants