Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ubuntu 22.04 Python 3.11 [asr]: multiple errors dataclasses ValueError: mutable default * for field * is not allowed: use default_factory #7166

Closed
alllexx88 opened this issue Aug 4, 2023 · 12 comments · Fixed by #7616
Assignees
Labels
bug Something isn't working

Comments

@alllexx88
Copy link

alllexx88 commented Aug 4, 2023

Describe the bug

After installing latest stable 1.19.1 from pipI, or the latest current commit with [asr] extras, I'm getting this error from hydra-core==1.2.0 when trying to import nemo.collections.asr:

Traceback (most recent call last):
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/test.py", line 5, in <module>
    import nemo.collections.asr as nemo_asr
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/__init__.py", line 15, in <module>
    from nemo.collections.asr import data, losses, models, modules
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/losses/__init__.py", line 15, in <module>
    from nemo.collections.asr.losses.angularloss import AngularSoftmaxLoss
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/losses/angularloss.py", line 18, in <module>
    from nemo.core.classes import Loss, Typing, typecheck
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/core/__init__.py", line 16, in <module>
    from nemo.core.classes import *
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/core/classes/__init__.py", line 16, in <module>
    import hydra
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module>
    from hydra import utils
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module>
    import hydra._internal.instantiate._instantiate2
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module>
    from hydra._internal.utils import _locate
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module>
    from hydra.core.utils import get_valid_filename, validate_config_path
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module>
    from hydra.core.hydra_config import HydraConfig
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module>
    from hydra.conf import HydraConf
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module>
    class JobConf:
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf
    @dataclass
     ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1230, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1220, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 958, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 815, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory

If I then manually upgrade hydra-core to the current latest (1.3.2), I get similar errors from nemo-toolkit:

Traceback (most recent call last):
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/test.py", line 5, in <module>
    import nemo.collections.asr as nemo_asr
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/__init__.py", line 15, in <module>
    from nemo.collections.asr import data, losses, models, modules
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/losses/__init__.py", line 16, in <module>
    from nemo.collections.asr.losses.audio_losses import SDRLoss
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/losses/audio_losses.py", line 21, in <module>
    from nemo.collections.asr.parts.preprocessing.features import make_seq_mask_like
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/parts/preprocessing/__init__.py", line 16, in <module>
    from nemo.collections.asr.parts.preprocessing.features import FeaturizerFactory, FilterbankFeatures, WaveformFeaturizer
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/parts/preprocessing/features.py", line 44, in <module>
    from nemo.collections.asr.parts.preprocessing.perturb import AudioAugmentor
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/parts/preprocessing/perturb.py", line 50, in <module>
    from nemo.collections.common.parts.preprocessing import collections, parsers
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/common/__init__.py", line 16, in <module>
    from nemo.collections.common import data, losses, parts, tokenizers
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/common/parts/__init__.py", line 15, in <module>
    from nemo.collections.common.parts.adapter_modules import LinearAdapter, LinearAdapterConfig
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/common/parts/adapter_modules.py", line 147, in <module>
    @dataclass
     ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1230, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1220, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 958, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 815, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'nemo.core.classes.mixins.adapter_mixin_strategies.ResidualAddAdapterStrategyConfig'> for field adapter_strategy is not allowed: use default_factory

It's easy to fix with a patch like this:

--- nemo/collections/common/parts/adapter_modules.py.orig       2023-08-04 13:55:53.464534800 +0200
+++ nemo/collections/common/parts/adapter_modules.py    2023-08-04 14:05:45.579537700 +0200
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from dataclasses import dataclass, is_dataclass
+from dataclasses import dataclass, is_dataclass, field
 from typing import Any, Optional
 
 from hydra.utils import instantiate
@@ -151,5 +151,5 @@ class LinearAdapterConfig:
     activation: str = 'swish'
     norm_position: str = 'pre'
     dropout: float = 0.0
-    adapter_strategy: Optional[Any] = adapter_mixin_strategies.ResidualAddAdapterStrategyConfig()
+    adapter_strategy: Optional[Any] = field(default_factory=adapter_mixin_strategies.ResidualAddAdapterStrategyConfig)
     _target_: str = "{0}.{1}".format(LinearAdapter.__module__, LinearAdapter.__name__)

However then another error of the kind comes up:

Traceback (most recent call last):
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/test.py", line 5, in <module>
    import nemo.collections.asr as nemo_asr
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/__init__.py", line 15, in <module>
    from nemo.collections.asr import data, losses, models, modules
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/models/__init__.py", line 18, in <module>
    from nemo.collections.asr.models.clustering_diarizer import ClusteringDiarizer
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/models/clustering_diarizer.py", line 29, in <module>
    from nemo.collections.asr.metrics.der import score_labels
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/metrics/der.py", line 24, in <module>
    from nemo.collections.asr.metrics.wer import word_error_rate
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/metrics/wer.py", line 27, in <module>
    from nemo.collections.asr.parts.submodules import ctc_beam_decoding, ctc_greedy_decoding
  File "/home/alex/T7/src/speaker_verification/nemo_speaker/venv11/lib/python3.11/site-packages/nemo/collections/asr/parts/submodules/ctc_beam_decoding.py", line 593, in <module>
    @dataclass
     ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1230, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1220, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 958, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 815, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'nemo.collections.asr.parts.submodules.ctc_beam_decoding.FlashlightConfig'> for field flashlight_cfg is not allowed: use default_factory

This can also be fixed accordingly, but all in all, it appears these issues are pretty common in the code base.

Looks like NeMo isn't Python 3.11 ready, at least the asr collection.

Steps/Code to reproduce bug

With Python 3.11

  1. Install nemo-toolkit[asr] either 1.19.1, or current HEAD from git (b498d43)
  2. Run import nemo.collections.asr

Expected behavior

Import nemo.collections.asr without errors

Environment overview (please complete the following information)

  • Environment location: Bare-metal
  • Method of NeMo install: pip install nemo-toolkit==1.19.1 or from source b498d43

Environment details

  • Ubuntu 22.04
  • PyTorch version 2.0.1
  • Python version 3.11.4
@alllexx88 alllexx88 added the bug Something isn't working label Aug 4, 2023
@alllexx88 alllexx88 changed the title Ubuntu 22.04 Python 3.11 [asr]: multiple errors dataclasses ValueError: mutable default * for field * is not allowed: use default_factory Ubuntu 22.04 Python 3.11 [asr]: multiple errors dataclasses ValueError: mutable default * for field * is not allowed: use default_factory Aug 4, 2023
@alllexx88
Copy link
Author

alllexx88 commented Aug 4, 2023

Seems to be a similar to #7002

@titu1994
Copy link
Collaborator

titu1994 commented Aug 4, 2023

Interesting. The fix is easy but needs to be applied to basically every single place that has this constructor for our adapter configs. Let me see if I can update it. But no guarantees on how soon fixes will come in main.

@alllexx88
Copy link
Author

Looking forward to it @titu1994 ! Thanks 😃

@shaper
Copy link

shaper commented Aug 26, 2023

@titu1994 I was looking to use NeMo speaker diarization with Python 3.11 and hit this dataclass issue. I patched everything involved in the specific code paths I needed: lmnt-com@d89acf9

I was using the neural diarizer as described in https://github.com/NVIDIA/NeMo/tree/main/examples/speaker_tasks/diarization

I'd be happy to upstream this if it's helpful.

I haven't checked whether this is backwards compatible for earlier python/dataclass versions, do you know?

For reference, what led me to this issue, though it's duplicative to the above discussion:

Similar error
StackOverflow solution

@alllexx88
Copy link
Author

@shaper Thanks for sharing. For brevity, you don't really need a lambda when you don't pass any init parameters, like this:

field(default_factory=lambda: ConfidenceConfig())

You can just do

field(default_factory=ConfidenceConfig)

It's only needed when you do pass parameter(s), like

field(default_factory=lambda: beam_decode.BeamRNNTInferConfig(beam_size=4))

@github-actions
Copy link
Contributor

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

@github-actions github-actions bot added the stale label Sep 26, 2023
@boeddeker
Copy link

I have the same issue. @tango4j suggested using one of the models from https://huggingface.co/spaces/hf-audio/open_asr_leaderboard, but I cannot import nemo.collections.asr:

    Traceback (most recent call last):
      File "/opt/pycharm-2022.3.3/plugins/python/helpers/pycharm/docrunner.py", line 138, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest NeMoASR[2]>", line 1, in <module>
        NeMoASR().apply_asr(file)
        ^^^^^^^^^
      File "/home/cbj/python/cbj/cbj/transcribe/pretrained.py", line 504, in __init__
        import nemo.collections.asr as nemo_asr
      File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/__init__.py", line 15, in <module>
        from nemo.collections.asr import data, losses, models, modules
      File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/losses/__init__.py", line 16, in <module>
        from nemo.collections.asr.losses.audio_losses import SDRLoss
      File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/losses/audio_losses.py", line 21, in <module>
        from nemo.collections.asr.parts.preprocessing.features import make_seq_mask_like
      File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/parts/preprocessing/__init__.py", line 16, in <module>
        from nemo.collections.asr.parts.preprocessing.features import FeaturizerFactory, FilterbankFeatures, WaveformFeaturizer
      File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/parts/preprocessing/features.py", line 44, in <module>
        from nemo.collections.asr.parts.preprocessing.perturb import AudioAugmentor
      File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/parts/preprocessing/perturb.py", line 50, in <module>
        from nemo.collections.common.parts.preprocessing import collections, parsers
      File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/common/__init__.py", line 16, in <module>
        from nemo.collections.common import data, losses, parts, tokenizers
      File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/common/parts/__init__.py", line 15, in <module>
        from nemo.collections.common.parts.adapter_modules import LinearAdapter, LinearAdapterConfig
      File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/common/parts/adapter_modules.py", line 147, in <module>
        @dataclass
         ^^^^^^^^^
      File "/opt/py/2023/lib/python3.11/dataclasses.py", line 1230, in dataclass
        return wrap(cls)
               ^^^^^^^^^
      File "/opt/py/2023/lib/python3.11/dataclasses.py", line 1220, in wrap
        return _process_class(cls, init, repr, eq, order, unsafe_hash,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/py/2023/lib/python3.11/dataclasses.py", line 958, in _process_class
        cls_fields.append(_get_field(cls, name, type, kw_only))
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/py/2023/lib/python3.11/dataclasses.py", line 815, in _get_field
        raise ValueError(f'mutable default {type(f.default)} for field '
    ValueError: mutable default <class 'nemo.core.classes.mixins.adapter_mixin_strategies.ResidualAddAdapterStrategyConfig'> for field adapter_strategy is not allowed: use default_factory

For documentation (I had to search in the provided links):
Mutable defaults were never allowed in dataclasses (by convention), but in python 3.11 they improved the check: Instead of checking some types (dict, list, set) they now use hashable as indicator for mutable.

An alternative to default_factory would be to use frozen dataclasses, but I don't know whether in this code base the configs are used as mutable objects or not.

@titu1994
Copy link
Collaborator

You need to update to NeMo 1.20, omegaconf did a fix that should resolve this

@github-actions github-actions bot removed the stale label Sep 28, 2023
@boeddeker
Copy link

I have NeMo 1.20.0.
With pip install nemo_toolkit and pip install pytorch_lightning I installed yesterday nemo.
So it should be the newest PyPI version.

$ pip show nemo_toolkit
Name: nemo-toolkit
Version: 1.20.0
Summary: NeMo - a toolkit for Conversational AI
Home-page: https://github.com/nvidia/nemo
Author: NVIDIA
Author-email: nemo-toolkit@nvidia.com
License: Apache2
Location: /opt/py/2023/lib/python3.11/site-packages
Requires: huggingface-hub, numba, numpy, onnx, python-dateutil, ruamel.yaml, scikit-learn, setuptools, tensorboard, text-unidecode, torch, tqdm, wget, wrapt
Required-by: 

$ pip show omegaconf
Name: omegaconf
Version: 2.3.0
Summary: A flexible configuration library
Home-page: https://github.com/omry/omegaconf
Author: Omry Yadan
Author-email: omry@yadan.net
License: 
Location: /home/cbj/.local/lib/python3.11/site-packages
Requires: antlr4-python3-runtime, PyYAML
Required-by: hydra-core

$ python -c "import nemo.collections.asr as nemo_asr"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/__init__.py", line 15, in <module>
    from nemo.collections.asr import data, losses, models, modules
  File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/losses/__init__.py", line 16, in <module>
    from nemo.collections.asr.losses.audio_losses import SDRLoss
  File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/losses/audio_losses.py", line 21, in <module>
    from nemo.collections.asr.parts.preprocessing.features import make_seq_mask_like
  File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/parts/preprocessing/__init__.py", line 16, in <module>
    from nemo.collections.asr.parts.preprocessing.features import FeaturizerFactory, FilterbankFeatures, WaveformFeaturizer
  File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/parts/preprocessing/features.py", line 44, in <module>
    from nemo.collections.asr.parts.preprocessing.perturb import AudioAugmentor
  File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/asr/parts/preprocessing/perturb.py", line 50, in <module>
    from nemo.collections.common.parts.preprocessing import collections, parsers
  File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/common/__init__.py", line 16, in <module>
    from nemo.collections.common import data, losses, parts, tokenizers
  File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/common/parts/__init__.py", line 15, in <module>
    from nemo.collections.common.parts.adapter_modules import LinearAdapter, LinearAdapterConfig
  File "/opt/py/2023/lib/python3.11/site-packages/nemo/collections/common/parts/adapter_modules.py", line 147, in <module>
    @dataclass
     ^^^^^^^^^
  File "/opt/py/2023/lib/python3.11/dataclasses.py", line 1230, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/opt/py/2023/lib/python3.11/dataclasses.py", line 1220, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/py/2023/lib/python3.11/dataclasses.py", line 958, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/py/2023/lib/python3.11/dataclasses.py", line 815, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'nemo.core.classes.mixins.adapter_mixin_strategies.ResidualAddAdapterStrategyConfig'> for field adapter_strategy is not allowed: use default_factory

@titu1994
Copy link
Collaborator

Hmm ok I'll take a look

@titu1994
Copy link
Collaborator

titu1994 commented Oct 5, 2023

Sorry for the long delay, this should be fixed now. Let me know if there are remaining data classes that I need to fix

@MedAymenF
Copy link
Contributor

@titu1994 Running examples/asr/transcribe_speech.py still gives me the following error:

transcribe_speech.py 104 <module>
@dataclass

dataclasses.py 1232 dataclass
return wrap(cls)

dataclasses.py 1222 wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash,

dataclasses.py 958 _process_class
cls_fields.append(_get_field(cls, name, type, kw_only))

dataclasses.py 815 _get_field
raise ValueError(f'mutable default {type(f.default)} for field '

ValueError:
mutable default <class 'nemo.collections.asr.modules.conformer_encoder.ConformerChangeConfig'> for field conformer is not allowed: use default_factory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants