Work around new(er) python 3 import semantics

[RomanHargrave]

This is not a bug within beets itself, per-se. I first began
observing this around Python 3.7 or 3.8, and I am currently on 3.8.

This is related to, but not exactly a duplicate of, #3730, where a
user experienced this exact problem.

---

In configurations where beets, and therefore the core
beetsplug-namespaced components, are installed in the system
packages directory, installing any beets plugin in the user
directory will result in the entire contents of the beetsplug
namespace being ignored in favor of that in the user packages
directory. This is ostensibly because python's understanding of
namespaces appears to have changed, and is now incompatible with beets
use of the beetsplug namespace as a de-facto search path.

Suppose I have the following structure in
/usr/lib/python3.X/site-packages/beetsplug:

beetsplug
 \- absubmit.py

and then, as a result of running pip3 install --user beets-extrafiles, I had
$HOME/.local/lib/python3.X/site-packages/beetsplug containing the
following

beetsplug
 \- extrafiles.py

In older python distributions, python would consider the contents of
both directories when searching for a module, while newer
distributions will search only the last matching directory. This is
largely desirable because it ensures that some version of a package
installed by the system will not interfere with a different version
installed by the user, should it contain different modules. Instead,
python will simply ignore the system package (assuming that
/usr/lib/python3.X/site-packages is searched before
$HOME/.local/lib/python3.X/site-packages). In other words, this
means that once a beetsplug directory is created in
$HOME/.local/lib/python3.X/site-packages, any beets plugin not
installed in that directory effectively does not exist until the
aforementioned directory is removed. In terms of the previously
described directory listings, beetsplug.absubmit effectively does
not exist, but beetsplug.extrafiles now does.

I had, at some point, found either a mailing list discussion, RFC, or
changelog describing this new behavior; however, because of python's
popularity, and the fact that I do not know what exact choice of words
were used in that article, I currently do not know where to find it
and cannot locate it as any search results remotely relating to python
are all horrid blogs detailing precisely how not to write software
under the premise of teaching one how to code. I will try to find that
article and post it here, if possible.

For users that may install beets via their distribution's package
manager, this means that they are forced to polute the system search
path with files installed using pip instead of their distributions
package manager. This can be more problematic for certain
distributions than others, and can lead to frustrating or unintuitive
behavior. For instance, if beetsplug.extrafiles is installed via pip
into the system packages directory, and then a system upgrade results
in a newer python distribution, beets will have been installed in the
new search path along with any plugins installed via the package
manager, but the user may then have to update any packages installed
via pip.

The short term solution, of course, is to install all of beets and its
pugins into either the user or system path, but not both ¹;
however, this is both inelegant and potentially frustrating for users
that do not realize what is happening, nor why, and assume that either
beets or the plugin that they have just installed is broken. Python is
not my primary language, and I do not like it very much for several
reasons that are not relevant or worth discussing here, so I can't
offer the most pythonic solution; however, I think one option would
be to expect that all out-of-tree plugins be distributed under another
namespace, and that users then specify the plugin by its fully
qualified name. For instance, instead of extrafiles being
beetsplug.extrafiles, it might be holzhaus.extrafiles and
specified in config.yml not as extrafiles but rather
holzhaus.extrafiles. The plugin search logic would then be updated
to search first for beetsplug.{name} and then fall back to {name}
(perhaps only if it contains namespace separators, to avoid confusion
with the vast number of modules that could exist within the root
namespace).

---

¹ Some distributions will prevent the user from installing packages via pip outside of the user directory, such as Gentoo:

$ sudo pip3 install beets-extrafiles                                                                                                                                                                                                     
ERROR: (Gentoo) Please run pip with the --user option to avoid breaking python-exec

p.s. Sorry about the paragraph width. Github appears to have deviated from normal markdown behavior by not treating contiguous lines of text as a paragraph. I edited most of this with emacs, where I have autofill enabled for markdown.

[sampsyo]

Thanks for pointing this out! I've converted this thread to a discussion so we can talk in a more free-form way about what approach to take.

Basically, I think that the beetsplug namespace package has probably outlived its usefulness. I originally modeled it after Flask's flaskext namespacing for plugins, but that was abandoned long, long ago (cf pallets/flask#1135) for similar reasons about namespace packages being flaky and annoying. A new scheme like the "fully qualified name" proposal you mentioned above seems like it would be a good idea to look more deeply into.

[RomanHargrave]

I think the FQN concept would be fairly simple to implement. Ideally, it would not have any special logic for the in-built plugin names, and those could then be moved into the beets namespace, leading to plugin names such as beets.replaygain and holzhaus.extrafiles. This means that the existing plugin import logic could probably be adapted to fit the new scheme with very little work.

Of course, there is the consideration that this will break existing configurations. This could be approached in a few ways -

• A transitional period exists where beets will try to "do the right thing" and guess whether or not to look for a plugin name in the "core" namespace or to treat it as a fully qualified name. If possible, beets might re-write the configuration file to reflect the new scheme, though it would be desirable to retain any comments and particular entry order, otherwise beets could nag the user after it's done running. After some time, the old behavior could be removed entirely.
• Versioning is added to the configuration file, in the style of docker-compose. This doesn't seem to have many positives, considering that many beets users are probably more interested in "setting and forgetting", and this places an onus on them to maintain their configuration file throughout versions, or on the beets maintainers to maintain support for older configuration behaviors indefinitely.
• This is approached as a breaking change, and it is expected that folks who upgrade to the new version will update their configuration file to accommodate it. If beets runs and can't find some plugins, it will list the plugins it didn't find and then print a helpful message informing users of the changes to plugin import mechanics and how to update their configuration file.

[sampsyo]

Yeah, the "attempt to guess" approach actually seems quite feasible. We could try importing both names and use whichever succeeds.