Uploading to PyPi and other sources

[marioortizmanero]

I'm creating this issue to keep track of this API progress. I know it's too early to release it but this is helpful for people who want to use it via PyPi in their project. It'd also need a new name, as I told you in the original spotipy thread.

Also, I wanted to let you know that once it's done I can take care of uploading and maintaining it in the AUR repos.

[felix-hilden]

Thank you for the offer! Let's discuss about that when the package is a bit closer to something constituting a release.

To comment on the progress, I'd say in addition to returning dataclass models from the client, most of the work is in documentation and tests. Functionality is already beyond the original Spotipy.

[felix-hilden]

An update: we are a lot closer to a usable package now. More or less every line of the package is now executed by the test suite. And the client is tested against the live Spotify server. This means the model-based client is at least a lot more reliable than it was. The test cases are not anything special though. They would need to be expanded to try different combinations of arguments.

With regards to the package name considerations, I have made a request to PyPI to transfer the spotipy name to this package. In my understanding it will result in one of two things. Either plamere responds, in which case we actually get a long-overdue dialog going, or after six weeks or so the name is transferred making informing the users of the old package essentially effortless.

I haven't figured out what to do with the old version and version numbers yet if the transfer is successful. My current plan is to update the package to the plamere repository version, which has been the source of many complaints in the issues. And along with that issue some warnings (deprecations or such) about the next major release. That would give this package a version number of 3 and development of would continue on from there.

[marioortizmanero]

Good to know! I didn't know PyPi let you transfer names. That plan sounds good to me. As long as it's not an abrupt change that would break other packages it should be fine.

Also, a small thing I noticed that I don't consider big enough to open an issue for, is that maybe all would be a better name for what is currently every in the scopes module. Really great job! :)

[felix-hilden]

Thanks! No abrupt changes, agreed. It would benefit no one. Yeah, all was my first option too, but it shadows a builtin function, so I figured let's not go there again.

[felix-hilden]

@marioortizmanero In #42 you mentioned a name transfer in the Arch repositories too. Is "spotipy" currently reserved? And would you enlighten me a bit, it's completely separate from PyPI, right? So, do you have any information on what's the share between installs from AUR and PyPI/pip? But yes, I would appreciate you hosting the package there very much!

[marioortizmanero]

Yes, spotipy is currently the old API: https://aur.archlinux.org/packages/python-spotipy/

The AUR is just another different repository for Arch Linux users. It's intended to be separate from PyPi, meaning that the dependencies are other packages from the AUR too. Here's what we call the PKGBUILD, aka the installation script. You just list the dependencies from the AUR and run a python setup.py install (basically).

[felix-hilden]

Update plan

I think we'll have to go a bit bass-ackwards with the release. The current version on PyPI is 2.4.4, which reportedly does not reflect the state of the original repository. An old version of the package along with documentation should be available for users that don't want to migrate. To release this final version of the original package a legacy RTD site should be available.

• Transfer name on PyPI
• Transfer name on RTD
• Discuss the transfer on Plamere's repository

Then if a consensus favoring this plan is reached: Consensus did not favor this plan.

• Inform Plamere repository watchers about the transfers
• Publish original documentation on RTD under version number 2.4.5
• Publish version 2.4.5 containing the most recent plamere code and the legacy documentation URL
• Publish version 2.5.0 with identical code, a deprecation warning and links pointing to new resources, perhaps a guide on how to upgrade too

Then let's let things settle for a couple of weeks and...

• Prepare package for release
• Publish new documentation and version 3.0.0 as the rewritten version
• Transfer name on AUR and publish the package
• Let Spotify know of this new library

I think AUR can interleave with other steps with some restrictions. It could transfer whenever and publish after PyPI. Then onto the next release.

[marioortizmanero]

After thinking about it, the name change on all these platforms is probably going to be hard, I'm not sure if it's worth it. Many outdated modules and programs will stop working even with the warnings on new downloads. Specially because using this new API isn't possible if you're using Python <3.7, which is a majority. Maybe we should get more opinions from the original github repo before doing anything else.

But if you still want to do the rename, I'll submit a request on the AUR soon.

[felix-hilden]

I have been somewhat hesitant as well. It can break things. That being said I have two arguments:

The package is better off requiring 3.7

• Enumerations in 3.4
• Type hints in 3.5
• f-strings in 3.6
• dataclasses in 3.7 are huge for the model-based API (actually, the dataclasses package backports this to 3.6 but I'm not sure how complete the functionality is)
• According to PyPI stats, 3.7 alone is the majority of spotipy downloads, ca. 60 %, with 3.6 declining. They have switched during the summer. When looking at requests downloads, excluding 2.7, 3.6 is still in the lead but 3.7 is not far off and will of course continue to rise. BTW, interesting to see what happens when the plug is pulled from Python 2.

The package should take over the spotipy name

• It is safe. 3.7 is required in setup. So if a user is on anything below that, the upgrade should not take place. This can be further verified in the setup script by examining the Python version and raising an error but should already be enforced by pip.
• Name changes should not be hard. Transfer on PyPI is already taking place, RTD is verified to be possible and next in line. But I don't know about AUR. Maybe that's something else. But we at least should acquire the names. Then they would at least be in the hands of active people.
• Outdated modules that work on 3.7 will stop working, yes. Outdated modules. And with some documentation we can ease the pain by directing users back to the 2.x versions.
• Spotipy is popular but the package is outdated. The Web API has progressed a lot and there are no other versions that are completely up-to-date or in active development. The easiest way to inform users is deprecation messages, and in my opinion then continuing to publish versions beyond 2.x is natural. But the last point is subject to debate. spotipy3 and links to it in deprecations is a valid approach as well, but to me it seems unnecessary. With the safety nets above in place it would at least be easier for new users not having to look for some updated version and for older users it's one install away from the original.

Some questions:

• What do you think of the arguments?
• Given a transfer and a new release, should the new release also have some warnings initially?
• If this package is released under a different name, would the original package be worth developing?

[marioortizmanero]

With this explanation the idea sounds much better to me. Specially because if the PyPi guys are okay with it, it's for a reason. And they have much more experience with this. As long as the old API still has a place in PyPi so that it can still be used, it should be okay, I guess.

AFAIK the name change on the AUR shouldn't be too hard, I'll keep you updated.

I'm not sure if the new release should have warnings too. It depends on how long this process is. But it's a good idea.

[marioortizmanero]

Here's the thread: https://lists.archlinux.org/pipermail/aur-general/2019-October/035493.html

I'm not too experienced with this but let's see how it goes :^)

[marioortizmanero]

Here's what the PKGBUILD would look like:

# Maintainer: Mario O.M. <marioortizmanero@gmail.com>
pkgname=python-spotipy
pkgver=3.0.0
pkgrel=1
pkgdesc="Client for the Spotify Web API"
arch=("any")
url="https://github.com/felix-hilden/spotipy"
license=("custom:MIT")
depends=("python>=3.7" "python-requests")
makedepends=("python-setuptools")
source=("https://github.com/felix-hilden/spotipy/archive/${pkgver}.tar.gz")

build() {
    cd "spotipy-$pkgver"
    python setup.py build
}

package() {
    cd "spotipy-$pkgver"
    python setup.py install --root="$pkgdir" --optimize=1 --skip-build
    install -Dm644 LICENSE "$pkgdir/usr/share/licenses/$pkgname/LICENSE"
}

• Are you OK with the description?
• Is the first release going to be 1.0 or should I use 3.0?
• Is there a specific version needed for requests?
• How are the release packages going to be named?

Edit: Okay I updated the script with what you answered

[felix-hilden]

Looks good, but the description could be the same as will be in PyPI: "Client for the Spotify Web API". The first release will be 3.0 if nothing too surprising happens during the PyPI transfer. A requests version could be pinned down, I'll look into it. Releases will be named conventionally, v3.0, v3.0.1 and so on.

[marioortizmanero]

By the way, it'd also be a good idea to let the Spotify devs know in spotify/web-api
so that they update the libraries page with this new API.

[felix-hilden]

Good idea, I'll do that on first release.

[SHxKM]

Since there seem to be some stalling/many "ifs" regarding transferring of ownerships, and since this library is already kind of backward incompatible with the "original" spotipy, I'm all for a new name. These are just the immediate advantages I see:

1. No delay in PyPI launch due to external sources
2. Immediate visibility for this (much better) library
3. No need in the future to explain this change to other users who may be confused

[marioortizmanero]

I was a bit conflicted too on the name change, but here are some counterpoints:

1. The delay shouldn't be that much at this point. If PyPi doesn't delay it for too long it should come out at early 2020. This would actually help the backwards compatibility part, because Python 2 will be retired in 2020, and the only issue would be Python <3.7 (support for 3.5 ends in Oct 2020 and 3.6 ends in Jan 2022). At some point the old spotipy will have to be retired too (it's already been 2 years since the last commit and Python keeps growing), and this is a good chance to take care of that. But yeah 1-2 months can be long depending on your intentions.

2. Immediate visibility sounds good but not that much in comparison to taking over the old spotipy repo. It currently has ~1000 daily downloads.

3. You're right about that. The part that I dislike the most is the confusion for new users. But with time it should get better, I guess :/

Of course these are just options to consider and we're all up for discussion here. It's only my opinion after all, and I'm not even the owner of this repo :)

[felix-hilden]

Thank you both! The most important deal for me is addressing the problems in old Spotipy and taking care of its users who are often left confused. Many users have complained about the repo being out of sync with PyPI and to be frank, at least after the playlist API changes somewhere early 2019 the library is quite obsolete. And as I've pointed out in a number of issues and comments, and as Mario mentioned, the library is still popular!

This rewrite is still based on Plamere's code, even though lots of refactoring has been done. Starting a third Spotify library (Spotipy, pyfy and this) and waiting for users to discover it would be a waste in my opinion. Thousands of users would continue to download the obsolete version and discover out-of-sync documentation. So,

1. I'm willing to wait for the sake of old users
2. I think the take-off without an established user base would be quite slow
3. As with every major or backwards incompatible change in a library, deprecating and informing users is reality. I expect to have to perform these anyway when Spotify makes changes to their API.

Let me know what you think and if you have any further concerns.

[felix-hilden]

Spotipy has been transferred on PyPI!

[marioortizmanero]

Wait, already? That's awesome! :)

So now we have to wait for Read The Docs, right? Is there something else that can be done for now besides the deprecation warnings?

[felix-hilden]

To release the new old versions (2.4.5 and 2.5) we need a stable link to the old documentation. So any release to PyPI has to wait for RTD as well. That's 19.12. or later. And the Plamere repo should be informed at some point as well. I'll add that to the todo list.

[SHxKM]

@felix-hilden Any way to install this package using pipenv? it doesn't seem to support installation from git, and I really want to start with this package instead of the old spotipy :/

pipenv requires an #egg fragment for version controlled dependencies. Please install remote dependency in the form git+https://github.com/felix-hilden/spotipy.git#egg=.

[felix-hilden]

I think I may have missed your point, SHxKM, about the immediate visibility of a new name. Or at least there is one more argument to be made in favor of having another name. Users that have gotten used to the fact that spotipy is outdated or have heard about it may hesitate to try out the new version. In a sense it would be a much cleaner start for this library.

But this doesn't tip the scale for me.

[SHxKM]

@felix-hilden fair enough. I actually use the old spotipy in a fairly popular web app and would be bummed to upgrade and have everything break (in tests of course), then sometime after figure out the whole library has changed. But you’re the one currently doing all the hard work and it’s safe to say you have more dev experience than me, so take my advice with a grain of salt.

[stephanebruckert]

@felix-hilden you can't just replace a library that's being downloaded hundreds of time daily and make it fully backward-incompatible. That would mean that 100% of existing apps and examples on the web that suggest to dopip install spotipy would break.

People are free to create forks, one alternative fork for spotipy seems to be https://github.com/Harrison97/spotipy. You can also create your own fork. If your library is not a fork, it's a new library, not a replacement.

@marioortizmanero's idea to use spotipy3 would be OK in my opinion.

[marioortizmanero]

A good way of getting more opinions on this would be posting an issue in the original repo or in an already existing one. We should warn there before releasing it anyway.

[felix-hilden]

I second posting about this to the original repository. But first, let's cut the hostility and find some common ground. I'd like to find the correct way to proceed. No solution will fully satisfy everyone, we all have our own horse in the race some way or another. Exaggerated statements like "100 % of apps would break" are obviously incorrect merely by the fact that this will be Python 3.7+ only. One could hit back with another low blow by saying that Plamere's own examples of searching for items without authentication don't work. My worry is exactly that: hundreds of people downloading the package and being left confused, daily. So let's try finding that common ground.

1. The original package is out of sync with the current repository and documentation contents. Would you agree that they need to be synced by publishing another version with the final code of Plamere?
2. As this is still the only active version, should the Plamere version direct users to this new version regardless of the name we choose? Perhaps by releasing another minor version as planned.
3. Yes, this is technically not a fork, but initially it might as well have been. The library has just progressed a lot. Would you hold any "real" forks to the same standard of not breaking backwards-compatibility? Lot's of people have complained in Plamere's issues about problems in the code. Updating to match the current Web API could also be backwards-incompatible.

On the topic of choosing another potential name, let it be known that I really dislike naming things package2 and package3. That seems a lot like a version number, and version numbers are already a thing. I would prefer another clever name.

Side note, everything breaking would of course be avoided by pinning down the version: spotipy=2.4.4, which should be standard, but who actually does that anyway. We still need to figure out a suitable requests version to require.

SHxKM, did you manage to install the package with pipenv? I could look into it a bit more if there's something we can do to make it installable from GitHub. After releasing its value will of course diminish a lot but if pipenv is gaining traction, we could try to support it.

[SHxKM]

SHxKM, did you manage to install the package with pipenv? I could look into it a bit more if there's something we can do to make it installable from GitHub. After releasing its value will of course diminish a lot but if pipenv is gaining traction, we could try to support it.

@felix-hilden no, and honestly I just moved to using pip as pipenv was too opinionated/stubborn for me anyway. Every bit of criticism on that repository is met with resistance.

[stephanebruckert]

@felix-hilden I saw that plamere is active on Twitter https://twitter.com/plamere, did you try to reach out with your current concerns?

[felix-hilden]

Thats too bad, SHxKM, well perhaps it can be considered some other time.

When transferring ownership of the package, Plamere was contacted via Twitter and pinged on GitHub. He did not respond. I take it that he is not intetested in discussing the future of his own package.

What did you think about my points? And this is for everyone, not just @stephanebruckert.

[SHxKM]

@felix-hilden - I always tend to take the path with the least (potential) drama. How similar is this spotipy to the old one? From my initial usage, not overly, and in a good way. But again, maybe I’m looking at it too narrowly.

[felix-hilden]

I like that principle. If there is no other more important deciding factor I'd also like to minimise drama. But I don't think it is the highest value of all.

This package is similar in spirit and feel, but streamlined. Most notable changes in the client and authentication modules are renaming methods and classes, making them more consistent, implementing the remaining endpoints and splitting some methods that were hard to call right without a careful read of the argument descriptions, like playback_start. Additionally, the client returns dataclass models instead of bare dictionaries. Moving to this version should not require much else than renaming calls and converting dict access to dot notation: artist['name'] -> artist.name. Authentication flows, call signatures and response contents are the same, as they directly reflect the Web API structure. But I recognise that even though these changes can be small for someone just trying out spotipy, for a larger application they might be tricky.

Scope, sender, and serialise modules were not present in the old version and are not necessarily needed. The util module was expanded beyond just prompting for a user token.

[felix-hilden]

One approach to deprecating the package could be to let the two clients co-exist for a while, for example as spotipy.v2.client+OAuth2 and spotipy.client+auth. But this would still require changes, which can again be eliminated by requiring an earlier version. The only way the old applications would not need to change is moving the new modules to e.g. spotipy.v3. So maybe not the way to go, but another option to consider.

[felix-hilden]

Here is my estimate of what the average user would experience when publishing 3.0.

Not aware of the rewrite

New individual user or app developer

A working library and comprehensive documentation, outdated resources elsewhere on the web.

Old individual user

Scripts breaking after an update. Most likely they will discover that there is no OAuth2 module or something similar. After searching around and landing on

• the PyPI page
• RTD documentation
• this GitHub site
• Plamere's issues (where we will announce the update)

they will be informed of what took place and either downgrade or modify their code.

Old app developer

Tests or even app breaking if the tests are not comprehensive enough, confused users complaining after they manually update Spotipy. Like the individual users they will discover the rewrite and update their dependencies and code accordingly.

Aware of the rewrite

Things could be a lot better though. If they upgrade to the deprecation notice version, they will be able to prepare for the change. So maybe a few weeks of notice isn't enough. A month perhaps?

These steps required to react to the update seem reasonable to me.

In the absence of comments and opinions of others, particularly @stephanebruckert, I'll assume the concerns raised were properly addressed and that we agree to continue as planned.

[marioortizmanero]

This is a similar situation to the Python 2 deprecation. Python 2 was outdated and needed some serious changes and the only way to do this was by using a new not-backwards-compatible version. Of course, this had its ups and downs but I think overall the community ended up winning with it. It can be annoying to "rewrite" one of your projects from one version to another, but you have to do it in order to keep your app updated. You can always modify setup.py and use spotipy<3.0 anyway, and it won't break. The only important part is awareness.

But the deprecation has to be long enough for everyone to know. You should post this in the old repo as soon as possible IMO.

[stephanebruckert]

In the absence of comments and opinions of others, particularly @stephanebruckert, I'll assume the concerns raised were properly addressed

I can't follow everything being said and I'm not the one deciding anyway, though I have my own opinion.

This is why it's probably better to raise this with the wider community at https://github.com/plamere/spotipy than here in the background. It will open different subjects that haven't been discussed objectively, such as:

• is plamere really inactive and unable to merge existing PRs/re-publish?
• any chance we all coordinate and get plamere to onboard motivated maintainers?
• the package being very active [1] [2], is it acceptable for it to break?
• why not just continue in a forked repo?
• are there any other candidate repositories for the matter?

Please also let the community know that:

• you are already able to publish new versions since you managed to get the spotipy pip package transferred to you PEP 541: transfer ownership of unmaintained spotipy package pypi/support#28,
• you have a plan to take over the doc by the 19th of this month Take over abandoned spotipy readthedocs/readthedocs.org#6335 (comment)

[felix-hilden]

Thank you both for the comments! I'll move the notice/discussion earlier on the plan and post an issue to discuss this.

But to answer the some of your points Stephane, Plamere really has abandoned the package, which was disappointing to me as well, not to mention the people raising issues on the repo. There are many other rewrites and forks, but none of them are in active development. One of them however, Harrison's fork, is almost as popular though slowly falling behind.

[felix-hilden]

The issue has been posted here!

[felix-hilden]

I pinned Requests to >=2.19,<3. 2.19 had something to do with Python 3.7 compatibility, so that was a fine version in my opinion. And let's see on the next major release of Requests if it breaks anything and update the requirement accordingly.

[felix-hilden]

I have now handed Spotipy back to the original owner, as he suddenly became available again. This library will continue as another thing. Next up:

• Rename (Renaming Spotipy #121) and disassociate from Spotipy
• Set up Read The Docs, I think stable and latest are enough, no need for versioned documentation quite yet
• Release 0.1.0 on PyPI
• Make final changes to the library if something needs to be refactored still
• Setup a dev branch (Git flow and release pipeline #43)
• Release a stable 1.0
• Inform the folks over at Spotify of this library

@marioortizmanero This won't be python-spotipy, but are you still interested in maintaining it on AUR?

[marioortizmanero]

Sure! I'm still going to use this library for my personal proyect, and it's awesome enough to be in the AUR :)

[felix-hilden]

Fantastic, thank you! Actually, that made me think. We could have a showcase of some projects that use this library somewhere, that'd be really cool. Users could submit their applications and be inspired from others. A positive feedback loop for the package. Maybe in the documentation index, right after the feature list?

[marioortizmanero]

Sure, sounds like a great idea. I could add mine if you want, then. The positioning you suggested also seems adequate :)

[felix-hilden]

0.1.0 has been released on PyPI!

[marioortizmanero]

On the AUR too! Please make sure everything is correct :)

https://aur.archlinux.org/packages/python-tekore/

It was wonderfully easy to upload! AFAIK there isn't a python-dataclasses package because Arch Linux uses Python 3.8, so that part isn't needed.

[felix-hilden]

Fantastic! Fair enough, so dropping the Python requirement down to 3.6 isn't sensible either? What if someone purposefully installs 3.6 on their machine? I imagine other dependencies might affect the decision. Are pip dependencies an option there? But I anticipate requiring 3.7 there isn't a big issue.

[marioortizmanero]

If someone is still using 3.6, it's probably inside a virtual environment, which has easy access to pip. For regular users, it doesn't matter, as they will be using 3.8. Although if someone leaves a comment asking for support I'll investigate more (I'd probably have to upload the backport myself or something like that).

[felix-hilden]

Version 1.0.0 has been released and spotify informed in spotify/web-api#1429. It's been a long discussion, but I'm happy to finally close it.

I thought about the showcase for applications a bit more, and decided it was not such a good idea after all. I'd still love to show off them somewhere, but to have them in our source could be a pain eventually - having to update the links and remove broken ones and such. There could be some sort of community put up somewhere. That would be a good place for Q&A and these kinds of fun stuff.

[marioortizmanero]

Great! I've updated the AUR repo as well.