Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port Mopidy to Python 3 #779

Closed
18 of 19 tasks
jodal opened this issue Jul 12, 2014 · 32 comments
Closed
18 of 19 tasks

Port Mopidy to Python 3 #779

jodal opened this issue Jul 12, 2014 · 32 comments
Assignees
Milestone

Comments

@jodal
Copy link
Member

@jodal jodal commented Jul 12, 2014

Python 3 is the future of Python, it is widely available on the environments Mopidy is deployed on, and I'm quite interested in being able to use asyncio, which requires Python 3.3+. There's nothing outside the Mopidy project blocking us from moving to Python 3.

This is the master bug to track our way towards running Mopidy on Python 3.

  • Dependencies:
    • Pykka supports Python 2.6+, 3.2+.
    • Tornado supports Python 2.6+, 3.2+.
    • GStreamer:
      • #225: Switch GStreamer Python bindings from gst-python (Python 2 only) to PyGI (Python 2+3). To get everything working with PyGI, a port of GStreamer from 0.10 to 1.x at the same time is required.
      • #665: Make mixers independent of GStreamer, as mixers are removed in GStreamer 1.x.
      • #671: Extract/reimplement our playlist format support to not depend on GStreamer. There's some issues with reimplementing this on top of PyGI/GStreamer 1.x.
  • Move code style towards being Python 2+3 compatible:
    • flake8 warning clean code.
    • Drop support for all Python versions older than 2.7.
    • Use from __future__ import unicode_literals everywhere and mark binary strings with b'foo'.
    • Selectively fix portability issues identified by the 2to3 tool.
  • Development support:
    • Use tox to run tests, so we can easily add Python 3 testing in parallel with Python 2.7.
  • Mopidy-Spotify:
    • Implement new libspotify binding, pyspotify 2, which supports Python 2.7, 3.2+.
    • Reimplement Mopidy-Spotify on top of pyspotify 2.
  • Mopidy-MPRIS:
    • python-indicate bindings to libindicate can be replaced with PyGI.
    • python-dbus can maybe be replaced by GDBus through PyGI.
  • Extension ecosystem:
    • Use tox as test runner in cookiecutter-mopidy-ext to be ready to start running tests on Python 3 too.
    • Send pull requests/create issues for Python 2+3 support in all extensions when Mopidy itself works on Python 2+3.
  • Once ~all extensions supports Python 2+3, drop Python 2 support in Mopidy.
  • Remove Python 2isms.
  • Enjoy new Python 3 features, like asyncio.
@jodal jodal self-assigned this Jul 12, 2014
@jodal jodal added this to the v0.22 - PyGI and GStreamer 1.x milestone Feb 5, 2015
@DanielFallon
Copy link

@DanielFallon DanielFallon commented Feb 19, 2016

Hey, I'd like to help with Python 3 migration, anything in particular that's under way? don't want to step on anyone's toes

@jodal
Copy link
Member Author

@jodal jodal commented Feb 19, 2016

I've started a branch at jodal/feature/py3-compat where I've:

  • updated the Python version check that stops you from starting Mopidy under Python 3
  • updated the tox.ini setup
  • starting fixing Py3 issues module-by-module, running the tests under both Python 2 and Python 3

I haven't touched this since November, except that I rebased the branch on top of Mopidy 2.0.0 a couple of days ago. The current status is that I've fixed around 1000 tests, with about 240 remaining.

The plan ahead is to:

  1. Fix the rest of the issues module-by-module.
  2. Take the experience from fixing all the issues and try to land as much of it as possible in Mopidy 2.x. My current module-by-module approach changes things (e.g. it starts to threat file paths as Unicode instead of bytes) that I don't want to land in Mopidy until the 3.0 release, where we're allowed to break things. The can't-land-until-3.0 diff should be as small as possible.
  3. Use Mopidy with Python 3, finding all the issues not covered by tests.

@DanielFallon
Copy link

@DanielFallon DanielFallon commented Feb 19, 2016

ok, I will fork off of that branch and try to fix whatever tests I can

@aleprovencio
Copy link

@aleprovencio aleprovencio commented Dec 10, 2016

Hello guys, any news on this issue?

@flying-sheep
Copy link

@flying-sheep flying-sheep commented Dec 10, 2016

3½ years left for this 😉

@Prior99
Copy link

@Prior99 Prior99 commented Mar 22, 2017

Are there any updates on this?

@jodal
Copy link
Member Author

@jodal jodal commented Mar 22, 2017

Porting to Python 3 is my personal highest priority task when working on Mopidy, problem is I have been quite busy with work for the last year.

My current plan, in approximate order, is to:

  • Fix the Mopidy-MPRIS test suite, so that it can survive the upcoming changes. This is a bit boring and my doorstep mile.
  • Remove all the deprecated stuff from Mopidy and make a major release. See this milestone for the detailed issues. This I look forward to.
  • Ensure that all extensions in the Mopidy org still works after the removal, and make necessary changes and releases.
  • Port Mopidy to Python 2 + 3 and make another major release.
  • Port all extensions in the Mopidy org to Python 3 only and release.
  • Port/help port all other extensions to Python 3 only and urge for release.
  • Remove the Python 2 support from Mopidy itself.

As you can see, there's a couple of tasks that should be done first before starting the porting itself. That said, the largest body of work is already complete: Rewriting pyspotify to workon Python 3 (two years of my spare time was spent here) and porting Mopidy to GStreamer 1.x (one paternity leave of spare time spent here).

I'll get there, but it takes time. If people want to help out, I think the most helpful would be helping port extensions to Python 3 as soon as Mopidy supports 2+3.

@Prior99
Copy link

@Prior99 Prior99 commented Mar 22, 2017

Thanks for the update, @jodal.

@eamanu
Copy link

@eamanu eamanu commented Mar 23, 2017

@jodal
Copy link
Member Author

@jodal jodal commented Mar 23, 2017

Any help with the project that frees some of the maintenance burden helps with the big hairy goals like Python 3, as it frees time from the maintainers to either work on the larger issues themselves or to do proper code review on larger contributes, which can be quite exhausting and time consuming. Two prime examples are helping other users with debugging their problems, and to reproduce possible bugs, minimize the steps needed to reproduce, and document the results in the issues.

@eamanu
Copy link

@eamanu eamanu commented Mar 24, 2017

@jodal Where we can see your work with python3?

regards

@Equidamoid
Copy link

@Equidamoid Equidamoid commented Dec 30, 2017

@jodal the py3-compat branch you mentioned looks a bit... abandoned. Is there a new one somewhere?
I'm considering writing an extension and I'd prefer digging into some py3 bugs in unstable mopidy branch rather than remembering nasty py2 workarounds.

@hrehfeld
Copy link

@hrehfeld hrehfeld commented Aug 20, 2018

What's the status on this? is there a version of mopidy that can be used with python3?

@hugovk
Copy link
Contributor

@hugovk hugovk commented Aug 20, 2018

Dependency pylast will be dropping support for legacy Python 2 soon: pylast/pylast#265

@jodal jodal removed this from the v4.0 milestone Sep 29, 2018
@jodal jodal added this to the v3.0 milestone Sep 29, 2018
@noahfx
Copy link

@noahfx noahfx commented Jan 3, 2019

Hi,
Currently I maintain pylast in Debian. I intend to upload the latest version (1) which no longer supports Python 2.7
I plan to publish this version at the end of February, if mopidy still uses Python 2 will not be included
in the next Debian release which could be a shame since mopidy is an amazing piece of software 😃

Is there any official branch where this is going on? I am wiling to help this to happen

Regards

@jjok
Copy link
Contributor

@jjok jjok commented Jan 7, 2019

@jodal Are the extensions the blocker here? If so, I would suggest opening an issue against each extension and referencing them here to keep track. If you can give some instructions in the issue on what changes are likely to be required, then people how aren't that familiar with Python (like me) could hopefully still help.

@jjok
Copy link
Contributor

@jjok jjok commented Jan 7, 2019

PS I'm happy to help with the extentions that I use (GMusic, Iris, Podcast, Scrobbler, Spotify), I just don't really know what to do at the moment.

@jodal
Copy link
Member Author

@jodal jodal commented Feb 4, 2019

I'm still committed to porting Mopidy to Python 3. I won't let Mopidy die with Python 2.

History

Most of the work of moving us to Python 3 was done years ago in assuring that all our dependencies works on Python 3:

  • Pykka has been Python 3 compatible since it was extracted from Mopidy in 2011.
  • pyspotify 2.x was a full rewrite to get Python 3 support and complete libspotify bindings. This alone took almost two years from start to finish, work that is now mostly void thanks to Spotify breaking large parts of libspotify.
  • The port of Mopidy from GStreamer 0.10/PyGObject to GStreamer 1.x/PyGI, released in Mopidy 2.0 three years ago, was the primary project of my second paternity leave.

As you can probably read between the lines from the long history of this issue and the state of the issue tracker, this project and open source in general, hasn't been my top priority since I moved from working in a large enterprise to a startup three years ago.

However, I'm slowly starting to pick up my open source projects again. I'm trying to focus on one project at the time and ship it before taking on the next. So far Mopidy.js and Mopidy-MPRIS has received some tender love and care. Currently, I'm working on a new release of Pykka, which among other small improvements will reduce the startup time off many Mopidy setups by a huge factor.

Next up

Once Pykka 2 has shipped, my primary goal is moving Mopidy to Python 3.

I won't be giving any time estimate for when that work will be complete, as that would simply be setting myself up for failure. It all depends a lot on external factors and motivation.

As for helping out, I don't see that there's much new contributors can do to significantly help the effort directly. I'm still not back to working on open source in a predictable and regular cadence, so PR reviews would quickly become demotivating for both parties. As previously mentioned in this issue, any help to reduce the general maintenance burden always helps indirectly.

As for other core devs, I'm sure they would be able to help out, but we all have our priorities and projects. I won't wait for our time and motivation to all align up. If they join in once I get properly started on Mopidy 3, that would be lovely, but I have no expectations.

Putting all my rambling about life and everything aside, this is neither a gigantic task nor a task I'm unfamiliar with. It's mostly a matter of completing the other ongoing stuff so that this can become the primary focus for a while.

Mopidy 3

Since I wrote the last plan here in March 2017, Python 3 adoption has gotten so far that I no longer see the point in taking gradual steps from Python 2 only via 2+3 to Python 3 only. Mopidy 2.x has served us quite well for three years, and will be a good and stable place to leave those stuck on Python 2 for whatever reason.

Thus, revising the plan from March 2017, I imagine a process like this:

  • Remove all the deprecated stuff from Mopidy. See the v3.0 milestone for the detailed issues.
  • Port Mopidy to Python 3.
  • Make a pre-release to PyPI, e.g. 3.0.0-rc1. This won't affect those running pip install mopidy, but makes it possible to run pip install --pre mopidy to install the pre-release for testing of ported extensions.
  • Port all extensions in the Mopidy org to Python 3, and optionally make pre-releases to PyPI.
  • Port/help port other extensions to Python 3. Help wanted once we get here.
  • Make a coordinated major release of Mopidy core and all extensions in the Mopidy org.

@jodal
Copy link
Member Author

@jodal jodal commented Jun 22, 2019

Since there has been five months, I guess it is time to update this issue. The Pykka release I spoke of last time was shipped as Pykka 2.0 six weeks ago. Since then, I've started working on Mopidy again:

  • Bugfix release 2.2.3 is out, so we have currently nothing unreleased in the release-2.2 branch.
  • In the develop branch which will become Mopidy 3.0 most of the deprecated stuff has been removed.

Next up for me is probably:

  • Simplifying logging (#1452)
  • Decide and implement how to handle file system paths when moving from byte-oriented Python 2 APIs to text-oriented Python 3 file system APIs. (No issue tracking this yet.)
  • Start making the test suite pass on Python 3.

@tmccombs
Copy link

@tmccombs tmccombs commented Aug 17, 2019

Is there anything I could do to help with making either mopidy core, or the mopidy-soundcloud plugin work with python3?

@PureTryOut
Copy link

@PureTryOut PureTryOut commented Aug 19, 2019

Thanks for your work so far @jodal!

I'm wondering, how are we supposed to install the docs when using the Python 3 version of Sphinx to build them (asking for packaging purposes)?

I used to do make -C docs SPHINXBUILD=sphinx-build-2 man but changing that for sphinx-build-3 just errors out:

make: Entering directory '/home/builder/aports/community/mopidy/src/Mopidy-3.0.0a1/docs'
sphinx-build-3 -b man -d _build/doctrees   . _build/man
Running Sphinx v1.8.4

Configuration error:
The configuration file (or one of the modules it imports) called sys.exit()

This happens on both 2.2.3 and 3.0.0a1. Or is this part just not ported yet?

@tmccombs
Copy link

@tmccombs tmccombs commented Aug 21, 2019

Given that v3 will break backwards compatibility anyway, and python2 will be end-of-lifed in january, is there a reason to keep python2 compatibility in v3?

@jodal
Copy link
Member Author

@jodal jodal commented Aug 23, 2019

@tmccombs said:

Given that v3 will break backwards compatibility anyway, and python2 will be end-of-lifed in january, is there a reason to keep python2 compatibility in v3?

No, we're not planning on keeping Python 2 compatability in Mopidy 3.

Quoting myself from February:

Since I wrote the last plan here in March 2017, Python 3 adoption has gotten so far that I no longer see the point in taking gradual steps from Python 2 only via 2+3 to Python 3 only. Mopidy 2.x has served us quite well for three years, and will be a good and stable place to leave those stuck on Python 2 for whatever reason.

@jodal
Copy link
Member Author

@jodal jodal commented Aug 23, 2019

@PureTryOut said:

This happens on both 2.2.3 and 3.0.0a1. Or is this part just not ported yet?

None of Mopidy has been ported to Python 3 yet. We've just been doing other simplifications and removed deprecated stuff to make the transition easier. Once we've ported, we'll build the docs with Sphinx running on Python 3.

@PureTryOut
Copy link

@PureTryOut PureTryOut commented Oct 3, 2019

Could I advise making sure the next release (2.4.0?) is compatible with Python 3? Most distributions are working on getting rid of their Python 2 packages. For my case, Alpine Linux is dropping Python 2 by the next release (3.11, end of January next year) which means Mopidy will be dropped from the repositories if it's not Python 3 compatible by then.

@jodal
Copy link
Member Author

@jodal jodal commented Oct 3, 2019

Mopidy 3.0 will be compatible with Python 3. The plan is to have it out before the end of the year.

@zubieta
Copy link

@zubieta zubieta commented Oct 15, 2019

Hi @jodal, I might have some spare cycles in the coming weeks to help with some Python 3 functionality. Either in mopidy directly or in the extensions. Do you have any particular issues to tackle?

@jodal
Copy link
Member Author

@jodal jodal commented Oct 15, 2019

Hi @zubieta!

We've recently merged a test setup which successfully runs ~10% of the test suite on Python 3. There are a couple of PRs soon to be merged that increases that to ~20%. Please check what PRs are already open, so you don't duplicate any work, and see the description of #1809 for a step-by-step guide on how to port a few more modules and their tests.

Extensions will follow as soon as Mopidy core is running on Python 3.

@jodal
Copy link
Member Author

@jodal jodal commented Nov 6, 2019

I think it's time for a status update on Mopidy and Python 3...

Mopidy 3.0.0a2 runs on Python 3 🎉

Every single one of the 2016 tests in Mopidy's test suite now runs on both Python 2.7 and Python 3.7. All of this work is merged in the develop branch. A huge thanks to @kingosticks for helping out with the porting effort!

I've just released the develop branch as-is to PyPI as Mopidy 3.0.0a2. It can be installed with:

python3.7 -m pip install --pre mopidy

This release has not been subject to any extensive testing, other than having a passing test suite, answering a few requests across MPD and HTTP, and playing a handful of MP3 files.

Road ahead ➡️

Mopidy 3.0.0a2 will probably be the only Mopidy release ever to support both Python 2 and 3. We'll promptly start removing Python 2.7 support and make Mopidy a cleaner and more modern Python code base.

The plan towards a final release is approximately as follows:

  • Remove Python 2.7 support from Mopidy and clean up after the porting effort.
  • Make sure Python 3.8 is supported.
  • Format the source code with Black.
  • Make another alpha release.
  • Fix the issues in the v3.0 milestone.
  • Make a beta release.
  • Fix any bugs discovered through use and porting of extensions.
  • Make a release candidate.
  • Once a large enough set of extensions have been ported...
  • Release Mopidy 3.0 final.

Help needed 😍

In parallel with the march towards 3.0 final, we need help with the rest of the ecosystem of 122 packages on PyPI found by searching for "mopidy".

For each extension you care about:

  • Port to Python >= 3.7. Drop Python 2.7 support.
  • Consider including project setup modernizations from the extension cookiecutter. I'll update the cookiecutter shortly, once I get around to porting some extensions myself.
  • Make pre-releases of the ports to PyPI.
  • Once Mopidy 3.0 final is released, make final releases to PyPI.

@jodal
Copy link
Member Author

@jodal jodal commented Nov 9, 2019

76 commits, 204 files changed, 9832 insertions(+), 9612 deletions(-) later, we have another pre-release: Mopidy 3.0.0a3 is now on PyPI. It can be installed with:

python3 -m pip install --pre mopidy

New since 3.0.0a2:

  • Python 2.7 support is gone, including lots of legacy/compatibility code:
    • The mopidy.compat module is gone.
    • # encoding: utf-8 comments are gone.
    • from __future__ ... imports are gone.
    • All subclassing of object are gone.
    • .encode() and .decode() no longer include an explicit "utf-8" argument.
  • Source code is formatted with black. 🖤
  • isort are now configured, and can be used to clean up imports when needed.
  • mock is replaced with unittest.mock.
  • unittest assertion methods are replaced with pytest assert statements, which means better readability and better error messages.
  • Most instances of % and .format() are replaced with f-strings.
  • All linters run on Python 3.
  • Docs are built on Python 3.
  • Tests in CI runs on Python 3.8 in addition to Python 3.7.
  • setup.py has been minimized and replaced by a declarative setup.cfg.
  • Dependencies in tox.ini, dev-requirements.txt, and docs/requirements.txt are all replaced by "extras" in setup.cfg. This means that dev dependencies are now installed with python3 -m pip install -e ".[dev]".

The road ahead still looks like I drafted in the previous comment.

@jodal
Copy link
Member Author

@jodal jodal commented Nov 9, 2019

With regards to Mopidy core, I think we're done here.

The rest of the way to Mopidy 3.0 final is tracked in the v3.0 milestone:
https://github.com/mopidy/mopidy/milestone/55

The porting of key extensions to Python 3 is tracked on this project board:
https://github.com/orgs/mopidy/projects/2

If you test out Mopidy on Python 3 in the near future, please open issues for any problems you run into!

@jodal jodal closed this Nov 9, 2019
Mopidy 3 on Python 3 automation moved this from In progress to Done Nov 9, 2019
@jodal jodal moved this from Done to Pre-release out in Mopidy 3 on Python 3 Nov 10, 2019
@jodal jodal changed the title Move to Python 3 Port Mopidy to Python 3 Nov 10, 2019
@jodal jodal unpinned this issue Dec 14, 2019
@jodal jodal moved this from Released in pre-release to Released in final release in Mopidy 3 on Python 3 Dec 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Mopidy 3 on Python 3
Released in final release
Linked pull requests

Successfully merging a pull request may close this issue.

None yet