Move Mopidy-Local extension out of Mopidy

[adamcik]

To further slim down mopidy we will be moving mopidy-local out to an external extension. The main hurdle for doing this is that much of our core tests are implemented via the local tests. Once this has been cleaned up it would be trivial to create a mopidy-local-base extension externally which other packages such as mopidy-local-sqlite could build on.

Alternatively we get rid of the "base" helpers and leave it fully to extension authors to figure out with the building blocks we provide.

[bernd-wechner]

A brilliant idea, but hasn't moved since Feb 2015, so in over 2 years.

Which raises two questions (given I'd like to see it done):

1. What can be done to prioritise this?
2. How much work roughly is involved?

It seems the main hurdle is clearly laid out and sounds expensive (refactoring a pile of core tests?) but how much work exactly would you estimate?

From a naive newb's perspective this:

(mopidy) bernd@bigfoot ~/workspace/Mopidy/mopidy/tests $ grep -rlE "(local|media_dir)" * | grep -v \.pyc
audio/test_scan.py
core/test_tracklist.py
internal/test_encoding.py
internal/test_xdg.py
local/test_search.py
local/test_library.py
local/test_json.py
local/test_translator.py
local/__init__.py
local/test_tracklist.py
local/test_playback.py
mpd/test_translator.py

that 12 test files reference local or media_dir and would be affected by such a move. Is that a fair estimate?

Just kinda curious. Local file management is one of my focal interests I guess and where I'm looking closely.

[adamcik]

Someone just has to start doing it. For the tests we can probably work around this short term by installing mopidy-local next to mopidy and running the tests for both in Travis etc.

[bernd-wechner]

Who is someone ;-). Just curious how dev is organised I guess. Certainly great to hear!

[kingosticks]

It is literally anyone in the world!

[adamcik]

There is no real strict policy for who works on what, people mostly just pick whatever interests them. In the case of local we haven't had anyone focus on improving it for some time so if you want to be the one to give this a go we can support you getting this done as best we can :-)

[bernd-wechner]

Someone just has to start doing it.

My bad, I misread this as "Someone has just started doing it." ;-). I must have been half asleep.

It's certainly a possibility, though I'm majorly time poor alas. But to compensate for that eager for a lasting music player solution and mopidy ticks a lot of boxes. Once settled on one solution I have some some extensions in mind ;-), but first a stable solution with a large local library is my goal and mopidy ticks some big boxes so far (Python, has a database, looks alive - vibrant current commit history and folk (adamcik anyhow ;-) responsive to issues, with a lively discussion forum and pretty good HTTP, MPD support, so far pretty appealing).

Once settled and committed I'd indeed invest some modicum time on development as I have some goals (other issues) but step 1 is a stable working solution and so the local-sqlite part is my focus right now. I'm also liking the great documentation and tox and virtualenv implementations (stuff I'm weak on and need for another project).

That said, I'm majorly time poor (a day job, three kids, sit on a couple of commitees, and at least three major projects running on the side, and a sideline hosting travellers and maintaining a property ... aaargh. All of which means I have fits of activity on one thing and then get distracted and drawn away.

But if there's one thing I love and want it's a living product with a future and vibrant team that is extensible, hackable and proactively wants support - offers mentoring. What I dislike about the community driven and FOSS world rather intensely is fracturing, forking and the seemingly endless diversity - I mean just finding a music player one is bogged down with a major project in analysing the many options and trialing and casting ones lot ...). Even in server space, aaargh, XMMS, MPD, mopidy ... maybe I missed one ...

So, I'll work a little on trialing with my library and see how I go. I may well take an interest in splitting this out some time as local is indeed my focus! ;-). Not much into streaming (yet).

[jodal]

I'm interested in pushing for the split between Mopidy and Mopidy-Local to happen before Mopidy 3.0.

Moving the local backend out to its own extension would reduce the scope of porting Mopidy core to Python 3. Of course, the local extension has to be ported as well before we have everything running as before, just on Python 3, but then it becomes two smaller projects instead of one huge.

In #1774, @adamcik just claimed that the bundled Mopidy-Local-JSON library has played out its role. The obvious replacement is the external extension Mopidy-Local-SQLite, by @tkem, maybe together with Mopidy-Local-Images, also by @tkem.

The only other implementation of the Mopidy-Local library interface that has been published to PyPI, aside from -JSON and -SQLite, is -Whoosh, which is an experiment by @adamcik from 2014.

All projects mentioned here are part of the Mopidy organization on GitHub.

I imagine two alternative ways of organizing the Mopidy-Local ecosystem going forwards:

Alternative A

• Mopidy-Local as an external extension instead of being bundled with Mopidy
  1. Mopidy-Local-JSON still shipped with Mopidy-Local, or
  2. Mopidy-Local-JSON removed, and Mopidy-Local shipped without a default library implementation
• Mopidy-Local-SQLite as is
• Mopidy-Local-images as is

This is almost as is, only with a smaller Mopidy core. If one removes Mopidy-Local-JSON, one needs to install at least two packages, Mopidy-Local and Mopidy-Local-SQLite, in addition to Mopidy to get a working local extension.

Alternative B

• Mopidy-Local as an external extension instead of being bundled with Mopidy
• Mopidy-Local-JSON removed
• Mopidy-Local-SQLite merged into Mopidy-Local
• Mopidy-Local-Images merged into Mopidy-Local

In this alternative, it is up to the maintainers of Mopidy-Local if they want to keep the library extension interface, or if they want to remove the support for alternative libraries to reduce the maintenance burden.

Either way, this alternative provides a user friendly setup, where everything we currently recommend for local setups are installed by a single package, neatly integrated.

What's next?

The timeline for any of this is until the end of the year, when Python 2.7 reaches end of life. By that time, we should have shipped Mopidy 3.0, and preferably a Mopidy-Local 3.0 running on Python 3 as well.

@tkem @adamcik @kingosticks What are your thoughts?

I can take on the task of moving the existing Mopidy-Local source code and tests to a new repo.

If we go for alternative B, are anyone interested in taking on the task of merging the three extensions, and getting them up to speed for Mopidy 3.0 and Python 3 before years end?

[adamcik]

I'm leaning towards just having -sqlite and maybe -images collapse into a singular mopidy-local. Upgrade path would just be to rescan everything.

Other major advantage of this, and partially reason I would like to consider moving -mpd out is that it gives us smaller pieces that might be easier to find new owners for. And as for a owner of a new -local, this should be up for grabs to anyone, not just the people mentioned in the last comment.

[jodal]

@tkem What's your take on this? I don't want to decide on anything here without hearing your thoughts first, as the creator of two of the three packages in question.

[tkem]

I tend to agree with @adamcik, i.e. Alternative B: The current state of affairs is somewhat confusing for users, and due to the lack of alternative Mopidy-Local implementations, there is no real advantage to it, IMHO.

So I'd recommend merging these and including mopidy-local-images, since I think a local backend definitely should also provide album artwork, even if it's usable only by Web clients.

Even though I haven't used any local backend in a long time, I guess I'll find some time for merging and porting this to Python 3 later this year if a new owner hasn't been found till then.

[jodal]

New Mopidy-Local extension

I updated the extension cookiecutter a bit and created a new Mopidy-Local extension using it. The new extension is up at https://github.com/mopidy/mopidy-local. Then, I copied the Mopidy-Local extension from the mopidy repo over to the new mopidy-local repo.

Because the extension claims the "local" extension name, it conflicts with the old bundled Mopidy-Local extension in Mopidy 2.2. Thus, the new extension is set to depend on Mopidy >= 3.0.0a1 (aka the develop branch), where there will soon be no bundled Mopidy-Local extension.

Finally, I released the extracted extension to PyPI as 3.0.0a1 mainly to reserve the package name. I've added @adamcik and @tkem as project owners there, so you can also upload releases.

Changes to Mopidy

PR #1784 removes the local backend from the Mopidy repo.

Once the PR lands, I'd like to release a 3.0.0a1 pre-release of Mopidy to PyPI. That shouldn't affect any users not passing the --pre argument to pip, but makes it possible to have tox and CI, etc. working properly in the Mopidy-Local project.

Going forward, I intend to release new pre-releases whenever we are at a level that is helpful to another project, like we're now with regards to Mopidy-Local. To keep the maintenance overhead down, I do not intend to maintain separate changelogs for each pre-release.

Next up

After the PR lands, I'd like to review all issues labeled with A-local and move most of them to the Mopidy-Local repo.

@tkem Feel free to take the ball on from here with regards to merging Mopidy-Local-SQLite and Mopidy-Local-Images into the Mopidy-Local extension.

[tkem]

@jodal: Sounds good! I should be able to find some time for this after Aug 15.

[jodal]

I've merged #1784 and released Mopidy 3.0.0a1 so that Mopidy-Local's is installable with pip install --pre mopidy-local and it should be possible to get its CI to run green.

I haven't touched the issues labeled A-local. Feel free to review those and move any issues that you think should/could be tackled in the extension to the extension repo.

[tkem]

Great, this should give me a good point to start - there's still some other stuff that demands my attention, but I guess I'll be able to start with this within the next couple of days...