Segmentation fault when importing nemo.collections.asr (protobuf?) #5405
sanchit-gandhi
started this conversation in
General
Replies: 8 comments
-
|
Yes probably from protobuf - make sure you're using a relatively modern version - https://github.com/protocolbuffers/protobuf/issues?q=is%3Aissue+seg |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
The protobuf version installed when building NeMo from source is 3.19.4: Let's try and upgrade protobuf to the latest version! If we first uninstall protobuf: And then install the latest version (4.21.5 https://pypi.org/project/protobuf/#history): We get a dependency clash: Right! So we need protobuf<3.20. Thus, the latest version we can install is 3.19.4 (https://pypi.org/project/protobuf/#history), but as we know this version throws the segmentation fault! protobuf is required for tensorboard. We have tensorboard version 2.10.0 installed with NeMo: Uninstalling tensorboard & protobuf and re-installing both throws the numba/numpy version issue: Full Traceback[NeMo W 2022-08-24 14:35:46 optimizers:55] Apex was not found. Using the lamb or fused_adam optimizer will error out.
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 import nemo.collections.asr.models as asr
File ~/NeMo/nemo/collections/asr/__init__.py:15, in <module>
1 # Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
---> 15 from nemo.collections.asr import data, losses, models, modules
16 from nemo.package_info import __version__
18 # Set collection version equal to NeMo version.
File ~/NeMo/nemo/collections/asr/models/__init__.py:16, in <module>
1 # Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
15 from nemo.collections.asr.models.asr_model import ASRModel
---> 16 from nemo.collections.asr.models.classification_models import EncDecClassificationModel
17 from nemo.collections.asr.models.clustering_diarizer import ClusteringDiarizer
18 from nemo.collections.asr.models.ctc_bpe_models import EncDecCTCModelBPE
File ~/NeMo/nemo/collections/asr/models/classification_models.py:28, in <module>
25 from pytorch_lightning import Trainer
26 from torchmetrics.regression import MeanAbsoluteError, MeanSquaredError
---> 28 from nemo.collections.asr.data import audio_to_label_dataset
29 from nemo.collections.asr.models.asr_model import ASRModel, ExportableEncDecModel
30 from nemo.collections.asr.parts.preprocessing.features import WaveformFeaturizer
File ~/NeMo/nemo/collections/asr/data/audio_to_label_dataset.py:15, in <module>
1 # Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
---> 15 from nemo.collections.asr.data import audio_to_label
16 from nemo.collections.asr.data.audio_to_text_dataset import convert_to_config_list, get_chain_dataset
19 def get_classification_label_dataset(featurizer, config: dict) -> audio_to_label.AudioToClassificationLabelDataset:
File ~/NeMo/nemo/collections/asr/data/audio_to_label.py:22, in <module>
19 import torch
20 import webdataset as wd
---> 22 from nemo.collections.asr.data.audio_to_text import expand_audio_filepaths
23 from nemo.collections.asr.parts.preprocessing.segment import available_formats as valid_sf_formats
24 from nemo.collections.common.parts.preprocessing import collections
File ~/NeMo/nemo/collections/asr/data/audio_to_text.py:25, in <module>
22 import webdataset as wd
23 from torch.utils.data import ChainDataset
---> 25 from nemo.collections.asr.parts.preprocessing.features import WaveformFeaturizer
26 from nemo.collections.common import tokenizers
27 from nemo.collections.common.parts.preprocessing import collections, parsers
File ~/NeMo/nemo/collections/asr/parts/preprocessing/__init__.py:16, in <module>
1 # Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
15 from nemo.collections.asr.parts.preprocessing.feature_loader import ExternalFeatureLoader
---> 16 from nemo.collections.asr.parts.preprocessing.features import FeaturizerFactory, FilterbankFeatures, WaveformFeaturizer
17 from nemo.collections.asr.parts.preprocessing.perturb import (
18 AudioAugmentor,
19 AugmentationDataset,
(...)
32 register_perturbation,
33 )
34 from nemo.collections.asr.parts.preprocessing.segment import AudioSegment
File ~/NeMo/nemo/collections/asr/parts/preprocessing/features.py:38, in <module>
35 import math
36 import random
---> 38 import librosa
39 import numpy as np
40 import torch
File /opt/conda/envs/hf/lib/python3.8/site-packages/librosa/__init__.py:209, in <module>
207 # And all the librosa sub-modules
208 from ._cache import cache
--> 209 from . import core
210 from . import beat
211 from . import decompose
File /opt/conda/envs/hf/lib/python3.8/site-packages/librosa/core/__init__.py:5, in <module>
1 #!/usr/bin/env python
2 # -*- coding: utf-8 -*-
3 """ Core IO and DSP functions"""
----> 5 from .convert import * # pylint: disable=wildcard-import
6 from .audio import * # pylint: disable=wildcard-import
7 from .spectrum import * # pylint: disable=wildcard-import
File /opt/conda/envs/hf/lib/python3.8/site-packages/librosa/core/convert.py:7, in <module>
5 import re
6 import numpy as np
----> 7 from . import notation
8 from ..util.exceptions import ParameterError
9 from ..util.decorators import deprecate_positional_args
File /opt/conda/envs/hf/lib/python3.8/site-packages/librosa/core/notation.py:8, in <module>
6 import numpy as np
7 from .._cache import cache
----> 8 from ..util.exceptions import ParameterError
9 from ..util.decorators import deprecate_positional_args
11 __all__ = [
12 "key_to_degrees",
13 "key_to_notes",
(...)
18 "list_thaat",
19 ]
File /opt/conda/envs/hf/lib/python3.8/site-packages/librosa/util/__init__.py:77, in <module>
1 #!/usr/bin/env python
2 # -*- coding: utf-8 -*-
3 """
4 Utilities
5 =========
(...)
74
75 """
---> 77 from .utils import * # pylint: disable=wildcard-import
78 from .files import * # pylint: disable=wildcard-import
79 from .matching import * # pylint: disable=wildcard-import
File /opt/conda/envs/hf/lib/python3.8/site-packages/librosa/util/utils.py:9, in <module>
6 import scipy.sparse
8 import numpy as np
----> 9 import numba
10 from numpy.lib.stride_tricks import as_strided
12 from .._cache import cache
File /opt/conda/envs/hf/lib/python3.8/site-packages/numba/__init__.py:206, in <module>
203 return False
205 _ensure_llvm()
--> 206 _ensure_critical_deps()
208 # we know llvmlite is working as the above tests passed, import it now as SVML
209 # needs to mutate runtime options (sets the `-vector-library`).
210 import llvmlite
File /opt/conda/envs/hf/lib/python3.8/site-packages/numba/__init__.py:146, in _ensure_critical_deps()
144 raise ImportError("Numba needs NumPy 1.18 or greater")
145 elif numpy_version > (1, 22):
--> 146 raise ImportError("Numba needs NumPy 1.22 or less")
148 try:
149 import scipy
ImportError: Numba needs NumPy 1.22 or lessLet's try uninstalling numba and reinstalling with conda to get the correct numpy version: Full Traceback---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 import nemo.collections.asr.models as asr
File ~/NeMo/nemo/collections/asr/__init__.py:15, in <module>
1 # Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
---> 15 from nemo.collections.asr import data, losses, models, modules
16 from nemo.package_info import __version__
18 # Set collection version equal to NeMo version.
File ~/NeMo/nemo/collections/asr/losses/__init__.py:15, in <module>
1 # Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
---> 15 from nemo.collections.asr.losses.angularloss import AngularSoftmaxLoss
16 from nemo.collections.asr.losses.ctc import CTCLoss
17 from nemo.collections.asr.losses.lattice_losses import LatticeLoss
File ~/NeMo/nemo/collections/asr/losses/angularloss.py:18, in <module>
1 # ! /usr/bin/python
2 # Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
3 #
(...)
13 # See the License for the specific language governing permissions and
14 # limitations under the License.
16 import torch
---> 18 from nemo.core.classes import Loss, Typing, typecheck
19 from nemo.core.neural_types import LabelsType, LogitsType, LossType, NeuralType
21 __all__ = ['AngularSoftmaxLoss']
File ~/NeMo/nemo/core/__init__.py:16, in <module>
1 # Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
15 import nemo.core.neural_types
---> 16 from nemo.core.classes import *
File ~/NeMo/nemo/core/classes/__init__.py:18, in <module>
16 import hydra
17 import omegaconf
---> 18 import pytorch_lightning
20 from nemo.core.classes.common import (
21 FileIO,
22 Model,
(...)
27 typecheck,
28 )
29 from nemo.core.classes.dataset import Dataset, IterableDataset
File /opt/conda/envs/hf/lib/python3.8/site-packages/pytorch_lightning/__init__.py:34, in <module>
31 _logger.addHandler(logging.StreamHandler())
32 _logger.propagate = False
---> 34 from pytorch_lightning.callbacks import Callback # noqa: E402
35 from pytorch_lightning.core import LightningDataModule, LightningModule # noqa: E402
36 from pytorch_lightning.trainer import Trainer # noqa: E402
File /opt/conda/envs/hf/lib/python3.8/site-packages/pytorch_lightning/callbacks/__init__.py:14, in <module>
1 # Copyright The PyTorch Lightning team.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
---> 14 from pytorch_lightning.callbacks.callback import Callback
15 from pytorch_lightning.callbacks.checkpoint import Checkpoint
16 from pytorch_lightning.callbacks.device_stats_monitor import DeviceStatsMonitor
File /opt/conda/envs/hf/lib/python3.8/site-packages/pytorch_lightning/callbacks/callback.py:25, in <module>
22 from torch.optim import Optimizer
24 import pytorch_lightning as pl
---> 25 from pytorch_lightning.utilities.types import STEP_OUTPUT
28 class Callback:
29 r"""
30 Abstract base class used to build new callbacks.
31
32 Subclass this class and override any of the relevant hooks
33 """
File /opt/conda/envs/hf/lib/python3.8/site-packages/pytorch_lightning/utilities/types.py:29, in <module>
27 from torch.optim import Optimizer
28 from torch.utils.data import DataLoader
---> 29 from torchmetrics import Metric
30 from typing_extensions import Protocol, runtime_checkable
32 _NUMBER = Union[int, float]
File /opt/conda/envs/hf/lib/python3.8/site-packages/torchmetrics/__init__.py:14, in <module>
11 _PACKAGE_ROOT = os.path.dirname(__file__)
12 _PROJECT_ROOT = os.path.dirname(_PACKAGE_ROOT)
---> 14 from torchmetrics import functional # noqa: E402
15 from torchmetrics.aggregation import CatMetric, MaxMetric, MeanMetric, MinMetric, SumMetric # noqa: E402
16 from torchmetrics.audio import ( # noqa: E402
17 PermutationInvariantTraining,
18 ScaleInvariantSignalDistortionRatio,
(...)
21 SignalNoiseRatio,
22 )
File /opt/conda/envs/hf/lib/python3.8/site-packages/torchmetrics/functional/__init__.py:14, in <module>
1 # Copyright The PyTorch Lightning team.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
---> 14 from torchmetrics.functional.audio.pit import permutation_invariant_training, pit_permutate
15 from torchmetrics.functional.audio.sdr import scale_invariant_signal_distortion_ratio, signal_distortion_ratio
16 from torchmetrics.functional.audio.snr import scale_invariant_signal_noise_ratio, signal_noise_ratio
File /opt/conda/envs/hf/lib/python3.8/site-packages/torchmetrics/functional/audio/__init__.py:23, in <module>
20 from torchmetrics.utilities.imports import _PESQ_AVAILABLE, _PYSTOI_AVAILABLE
22 if _PESQ_AVAILABLE:
---> 23 from torchmetrics.functional.audio.pesq import perceptual_evaluation_speech_quality # noqa: F401
25 if _PYSTOI_AVAILABLE:
26 from torchmetrics.functional.audio.stoi import short_time_objective_intelligibility
File /opt/conda/envs/hf/lib/python3.8/site-packages/torchmetrics/functional/audio/pesq.py:22, in <module>
19 from torchmetrics.utilities.imports import _PESQ_AVAILABLE
21 if _PESQ_AVAILABLE:
---> 22 import pesq as pesq_backend
23 else:
24 pesq_backend = None
File /opt/conda/envs/hf/lib/python3.8/site-packages/pesq/__init__.py:5, in <module>
1 # 2019-May
2 # github.com/ludlows
3 # Python Wrapper for PESQ Score (narrowband and wideband)
----> 5 from ._pesq import pesq, pesq_batch
6 from ._pesq import PesqError, InvalidSampleRateError, OutOfMemoryError, BufferTooShortError, NoUtterancesError
7 __all__ = ['pesq', 'pesq_batch',
8 'PesqError', 'InvalidSampleRateError', 'OutOfMemoryError', 'BufferTooShortError', 'NoUtterancesError']
File /opt/conda/envs/hf/lib/python3.8/site-packages/pesq/_pesq.py:8, in <module>
6 from multiprocessing import Pool, Queue, Process, cpu_count
7 from functools import partial
----> 8 from .cypesq import cypesq, cypesq_retvals, cypesq_error_message as pesq_error_message
9 from .cypesq import PesqError, InvalidSampleRateError, OutOfMemoryError
10 from .cypesq import BufferTooShortError, NoUtterancesError
File /opt/conda/envs/hf/lib/python3.8/site-packages/pesq/cypesq.pyx:1, in init cypesq()
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObjectYikes... uninstalling numba and installing with pip: Cool! This seems to work! Appears as though it was an issue with tensorboard / protobuf. Hopefully anyone who encounters this issue can follow these steps as a workaround until the aforementioned modules are fixed! |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Yeah that is one complicated workaround. What Nemo version were you using, I assume main branch or 1.10? |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Condensed code snippet for workaround: Main branch! (265f7b1) |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Wow, that's painful. Thanks for finding this. Fyi @ericharper @fayejf @nithinraok |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
A few other ways to solve this issue. Simplest way
After this follow basic nemo install steps - install pytorch from conda first, install Cython, then install NeMo toolkit and everything works out of the box. This is because the pre-compiled protobuf for python 3.9 works out of the box. More convoluted way if you absolutely must use python=3.8conda uninstall tensorboard protobuf Numba does not interfere with protobuf directly so no need to uninstall and reinstall |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Amazing! Thanks for the help @titu1994 :) python=3.9 seems like the way to go! |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
What's sad is that this is even necessary. Protobuf and tensorboard aren't even majorly used in Nemo and are not even direct dependencies of Nemo. Just underlying libraries install it and cause this mess. Well, at least there's a fix |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the bug
Importing
nemo.collections.asrthrows a segmentation fault.Steps/Code to reproduce bug
Install NeMo in a fresh conda environment as per the instructions on the README.md: https://github.com/NVIDIA/NeMo#installation
Packages
Now try and import
nemo.collections.asr:Debugging with gdb:
Traceback
Is this possibly related to protobuf? Note: uninstalling and re-installing protobuf gives the same error:
Expected behavior
Import of the module!
Environment overview (please complete the following information)
Environment details
Beta Was this translation helpful? Give feedback.
All reactions