Server not compatible with marshmallow-dataclass>8.5.3

[nixjdm]

marshmallow-dataclass 8.5.3 works.

With marshmallow-dataclass 8.5.4 through 8.5.8, we get

$ lektor server
/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py:353: UserWarning: ****** WARNING ****** marshmallow_dataclass was called on the class <function NewType.<locals>.new_type at 0x7f0af2e076d0>, which is not a dataclass. It is going to try and convert the class into a dataclass, which may have undesirable side effects. To avoid this message, make sure all your classes and all the classes of their fields are either explicitly supported by marshmallow_dataclass, or define the schema explicitly using field(metadata=dict(marshmallow_field=...)). For more information, see https://github.com/lovasoa/marshmallow_dataclass/issues/51 ****** WARNING ******
  warnings.warn(
Traceback (most recent call last):
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/dataclasses.py", line 1197, in fields
    fields = getattr(class_or_instance, _FIELDS)
AttributeError: 'function' object has no attribute '__dataclass_fields__'. Did you mean: '__dataclass_params__'?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 350, in _internal_class_schema
    fields: Tuple[dataclasses.Field, ...] = dataclasses.fields(clazz)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/dataclasses.py", line 1199, in fields
    raise TypeError('must be called with a dataclass type or instance')
TypeError: must be called with a dataclass type or instance

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 364, in _internal_class_schema
    created_dataclass: type = dataclasses.dataclass(clazz)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/dataclasses.py", line 1185, in dataclass
    return wrap(cls)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/dataclasses.py", line 1176, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/dataclasses.py", line 909, in _process_class
    for b in cls.__mro__[-1:0:-1]:
AttributeError: 'function' object has no attribute '__mro__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/joe/miniconda3/envs/lektor/bin/lektor", line 8, in <module>
    sys.exit(main())
  File "/home/joe/wrkspc/lktr/lektor/lektor/cli.py", line 621, in main
    cli.main(args=args, prog_name=name)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/joe/wrkspc/lktr/lektor/lektor/cli.py", line 336, in server_cmd
    from lektor.devserver import run_server
  File "/home/joe/wrkspc/lktr/lektor/lektor/devserver.py", line 9, in <module>
    from lektor.admin import WebAdmin
  File "/home/joe/wrkspc/lktr/lektor/lektor/admin/__init__.py", line 1, in <module>
    from .webui import WebAdmin
  File "/home/joe/wrkspc/lktr/lektor/lektor/admin/webui.py", line 12, in <module>
    from lektor.admin.modules import api
  File "/home/joe/wrkspc/lktr/lektor/lektor/admin/modules/api.py", line 137, in <module>
    @_with_validated(_PathAndAlt)
  File "/home/joe/wrkspc/lktr/lektor/lektor/admin/modules/api.py", line 97, in _with_validated
    schema_class = mdcls.class_schema(param_type, base_schema=_SchemaBase)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 339, in class_schema
    return _internal_class_schema(clazz, base_schema, clazz_frame)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 382, in _internal_class_schema
    attributes.update(
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 385, in <genexpr>
    field_for_schema(
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 707, in field_for_schema
    or _internal_class_schema(typ, base_schema, typ_frame)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 367, in _internal_class_schema
    raise TypeError(
TypeError: _PathType is not a dataclass and cannot be turned into one.

[dairiki]

I fixed this in 9e7a3ce by pinning (temporarily, hopefully) typing-inspect<0.8.0.

Reference: lovasoa/marshmallow_dataclass#207.

[nixjdm]

@dairiki It doesn't look like that works, possibly because of some other issue with pkg_resources. Constraining marshmallow-dataclass directly might be the better option.

$ lektor s
Traceback (most recent call last):
  File "/home/joe/miniconda3/envs/lektor/bin/lektor", line 8, in <module>
    sys.exit(main())
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/lektor/cli.py", line 621, in main
    cli.main(args=args, prog_name=name)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/lektor/cli.py", line 341, in server_cmd
    ctx.load_plugins(extra_flags=extra_flags)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/lektor/cli_utils.py", line 140, in load_plugins
    initialize_plugins(self.get_env())
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/lektor/pluginsystem.py", line 127, in initialize_plugins
    plugins = load_plugins()
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/lektor/pluginsystem.py", line 121, in load_plugins
    rv[ep.name] = ep.load()
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2470, in load
    self.require(*args, **kwargs)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2493, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/home/joe/miniconda3/envs/lektor/lib/python3.10/site-packages/pkg_resources/__init__.py", line 795, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'typing.inspect<0.8.0' distribution was not found and is required by Lektor

[dairiki]

@dairiki It doesn't look like that works, possibly because of some other issue with pkg_resources. Constraining marshmallow-dataclass directly might be the better option.
pkg_resources.DistributionNotFound: The 'typing.inspect<0.8.0' distribution was not found and is required by Lektor

Hmph. Typing-inspect<0.8.0 is certainly available from PyPI.

It looks like the quoted exception is being raised when Lektor tries to load a plugin entry point. Perhaps you need to rebuild Lektor's plugin cache (e.g. lektor plugins reinstall or similar)?

Maybe something doesn't like the spelling of typing.inspect. The canonical spelling is typing-inspect, though I'm pretty sure it shouldn't matter.

(It would be nice to understand what is causing the problem, rather than just changing the pin.)

[nixjdm]

Alright, here's what I've got so far. This seems to happen iff there's a plugin in use that lists Lektor as an install requirement, as lektor-s3 does here. Removing that line is an effective fix, as a test, but we can't fix this plugin-side. It isn't uncommon for plugins to do this, though they probably shouldn't.

[dairiki]

Thank you @nixjdm. I was able to reproduce here.

As unlikely as it seems, changing the spelling the typing-inspect requirement seems to fix the problem. I.e. change typing.inspect < 0.8.0 to typing-inspect < 0.8.0 in setup.cfg, then reinstall Lektor (e.g. pip install -e .).
Can you verify that that fixes it for you?

I don't think changing the pin to marshmallow_dataclass fundamentally changes the issue (other than that marshmallow_dataclass is already in the canonical spelling, I think.) Part of what's going on is that when lektor-s3 gets installed in the package cache, because of its dependency on "lektor", pip installs the latest stable (3.3.6) version of Lektor into the package cache. That version of Lektor happens to have no dependency on either marshmallow_dataclass or typing-inspect, so neither of those get installed in the package cache. I'm still not totally sure where the problem arises — apparently somewhere deep in pkg_resources — but I'm pretty sure the fact that we're now dealing with two different versions of Lektor has something to do with it.

(I wonder if dropping the use of pkg_resources in favor of importlib.metadata (or importlib-metadata for older pythons) would help?)

[nixjdm]

Yeah, I'm not sure where exactly this is a problem either, but I can at least confirm your fix!

# a git diff
-    typing.inspect < 0.8.0
+    typing-inspect < 0.8.0

works.

[dairiki]

Okay, let's go with that for now.

I've started playing with disusing pkg_resources in favor of importlib.metadata and I think that "fixes" things — in that the plugin entry points can be loaded without an exception being thrown — as well.