Playlist Corruption and Definition Loss

[Leviathan1753]

Just to give you an overall of my general usage:

• I've got 7 or so playlists, they just have normal conditions, and don't reference each other
• I open Google Music on two desktops /w the autoplaylist plugin and one phone. I can either have both desktops open and running, or just one, depending. It never caused a problem having both open in the month of usage so far.
• I close the Google Music tab and reopen on both desktops at least once a day.

I was rolling great with that setup for about a month or so, but two days ago I sat down at my computer and glanced over to see my playlists were gray. On clicking them, I got a "Couldn't open playlist, please try again" message. I then opened my autoplaylist extension, and found all my playlist definitions were wiped. I reinstalled the extension, and everything remained broken. One thing I did notice was that my phone still saw the playlists properly and was able to use them. In addition, while the extension was uninstalled, the playlists would be gray and unopenable, but whenever the extension was uninstalled, they'd pop back. It's almost as if the presence of the extension put them in their funk state. I recreated my autoplaylists by looking at the description text from my phone and remaking them. Everything repopulated alright and worked the rest of the day. Today it happened again, and upon uninstalling the extension, most of my playlists are openable again. However, my definitions are again wiped, and I'll have to remake them. Not sure what's going on, but it doesn't seem like a one time thing.

[simon-weber]

Ouch; sorry about that! Thanks for the detailed report.

Here's what I'd suggest:

• first try avoiding running the extension on two computers at once
• if you still get unavailable playlists, try disabling syncing by setting your interval to 0 (under settings)
• if you can bring them back that way, try turning syncing back on at a higher interval like 15 minutes

Here's all the details behind the suggestions:

The grey/unopenable/unavailable playlists symptom has happened before. It's usually caused by Google getting overloaded by the extension's requests. The switch to the new syncing apis seems to have largely fixed that, so I'm surprised to see you run into it (especially with relatively few playlists).

Running on two computers at once is known to cause issues (#78, though not unavailable playlists). My bet is this is the root cause: the extensions may have gotten into a loop where they add and remove eachother's updates, causing more requests to Google than expected for your limited playlists. It sounds like this is something I should prioritize fixing -- it shouldn't be too hard to add some locking.

Your playlist definitions are stored by Google and unfortunately get wiped automatically when you uninstall. This sounds like an implicit vote for backups (#40).

[Leviathan1753]

Alright, I'll pick one of my desktops and try to get it to be the only one with the extension. Might not be possible if chrome sync gets annoying? haha. Might just sync it as installed back and forth.

One thing of note is that today as I typed all this, only one of my computers was on, and I reinstalled the extension and it was still in a bad state. So, I don't think it was -currently- in a loop syncing back and forth. The autoplaylist definitons were just empty, and the currently defined playlists in google were locked until I uninstalled the extension.. so definitely just in an odd state overall, even when not currently looping.

I'll run one and report back if it continues.

And yeah, I think we should have a handle on our playlist definitions for sure. 8)

[simon-weber]

Ah, right, there's not a straightforward way to disable an extension on one computer (maybe I should add that?). The easiest way would be to go to chrome://extensions then uncheck/recheck the "enabled" checkbox. After that it'll be loaded, but not actually syncing until you reload a Google Music tab on that computer.

so definitely just in an odd state overall, even when not currently looping.

Yeah, I've never had a good guess for what's happening on Google's end. It usually takes them up to a day to recover once it happens.

[Leviathan1753]

All good so far. Definitely think it was due to a low sync period (less than 10m) + two computers. Haven't had the problem after raising it to 300, as any pingponging would only happen once every 5 hours instead of 5 minutes, haha.

Anyway, maybe just a warning if the user selects a low sync period? Other than that, I think this is settled.

[simon-weber]

Good to hear! I'll make a new ticket around low sync periods.

[Noggog]

Heya, despite the different account, I'm the original OP of the ticket.

Anyhoo, I had my sync time set to 3 hours, but apparently still got bit. I'm bringing it up, because I feel like I might be able to scrape some logs or something useful for you. I feel like it's the app in a bad state, rather than Google (even if perhaps Google caused it initially). I have the hunch that it would be possible for the app to recover, but that's just my gut.

Here's why I think so:

• It's been 12 hours or so where I've had google play off, and the plugin disabled. After that time, things still don't work despite Google having time to flush its system.
• However, if I were to reinstall the plugin, it would begin to work immediately. Thus, I think right at the current moment, it's the app in a funky state, not Google.
• One odd thing is I'm able to make new playlists, and they show up on google play, but they cannot be opened. Again, if I were to reinstall the extension, it would work.
• My phone is able to use the playlists perfectly. I think the app is actively blocking access to the playlists on my desktop. Uninstalling immediately allows usage of the playlists. The other thing is that the app is blocking the playlists on BOTH my computers, so whatever is funky is synced between both.

I do think there's something funky in the app internal systems. Maybe a button could be made that essentially factory resets the internals, and reattaches the playlist definitions to any existing playlists by name? That, or the ability to save/load definitions.. as then we could reinstall and load.

Either way, I'm fine with leaving my playlists broken for a short bit. Is there any logs you'd like me to grab before I clean the system out?

[simon-weber]

Huh, that's an odd set of symptoms. It definitely goes against my server load theory, which can't explain 1) a difference between disabling and uninstalling the extension and 2) why they'd be accessible on your phone. I don't have any other good theories, so we'll have to do a bit of digging.

Can you try this once in the broken state?

• create a new chrome profile/user and don't sync it with your account. Does switching to it make your playlists accessible, while they're still inaccessible from your usual profile?
• load Music, open the dev tools and switch to the network tab, filter for "loaduserplaylist", then hit clear. Then, try to open an inaccessible playlist. You should see a request in the network tab. Is there anything immediately off about the response (eg a non-200 status code)? You can also right click > save as har with content and then email to me. Also, switch back to the console tab and check for interesting messages (you can also right click > save as here if you find something interesting).

[Noggog]

Well, I left it yesterday in a broken state with the extension disabled. Just re-enabled the extension, opened the dev tools, and now things seem to be back to normal, of course, haha.

Will open dev tools and look for those messages next time it rolls around.

Thanks for the troubleshooting help!

[Noggog]

I've emailed you some logs that I just got.

Not sure how to do your first suggestion. Wouldn't creating a new user mean I wouldn't be logged into my Google Play anymore? How would my playlists be accessible while logged in as someone else?

[simon-weber]

Ah, I'm referring to a Chrome Profile, which are sometimes called Chrome Users. For our purposes the effect will be a bit like an incognito window: it'll rule out any state in chrome or the extension.

Instructions are at https://support.google.com/chrome/answer/2364824. Be sure not sync it with your account (that would install the extension, which we don't want). Instead, log into Music manually after creating and switching to it.

[Noggog]

Made a test person, did not do anything to sync, and didn't see any extensions installing or anything. Logged into my google music manually. The playlists were not accessible from that Test person setup.

[simon-weber]

Hm. That seems to contradict some of our earlier findings: if uninstalling it fixes the problem, I'd expect the profile without the extension to have no problems.

re: not seeing requests in the network tab, I think I missed a step: can you try my earlier instructions again, except manually refresh the page after you hit clear? The filter should stay applied after the refresh and show you a couple of requests. I'm expecting that the ones corresponding to inaccessible playlists have a 500 status.

[simon-weber]

Oh, and another question: can you summarize the conditions for the playlists you're using? I'm wondering if there's something about them that's generating lots of mutations (perhaps large playlists sorted by random?).

[Noggog]

I've got ~16k total songs in my library. The auto playlists are usually a combination of:

• Exclude 1-3 genres
• Last played 60+ days ago
• Some thumbs up/down state

Sorts happen by:

• Usually just random
• A few have sorting by playcount first, then random

I do not have any referencing of other playlists in my conditions, all just pull from my list of songs in my library.

[simon-weber]

(from a response over email) I recognize those [2,2,0,0] responses from when the extension used to use those apis: it's the same one I used to see all the time before switching to the optimized apis. It signals a serverside error of some sort. I never figured out exactly what the different fields mean (it's a protobuf message intended for javascript, and I haven't found a way to get at the schema).

So far it looks like everything is pointing at Google getting overloaded, which is a bummer since I thought the new apis had solved that for good. I'm not sure I can do much more around reducing the updates in each sync: the code already does some whiteboard-interview algos stuff to perform the absolute minimum amount.

One thing I can try is more aggressive backoff. Right now, when Google responds with an error or times out, we don't pay attention and try the next sync anyway. In the early days this was needed since even successful requests didn't apply fully, but I'm wondering now if multiple failed attempts end up building up into a worse problem than one on its own.

In the meantime, here's some load-reduction ideas which - quoting some Slack release notes - "I suggest to you now only with great regret and self loathing":

• use an infrequent sync interval (maybe 30+ minutes)
• add limits to your playlists to make them smaller (I'm guessing they're 1k)
• avoid random sorts on large playlists with large result sets (since syncs will usually replace the entire playlist)

[Noggog]

Alright, I'll try lowering my playlist number to ~1-200, since I probably wouldn't listen to more than that before they remade themselves anyway.

[Noggog]

I just uninstalled to extension to try to get things back up and running. Unlike what I said before, the playlists did not immediately begin to work. Perhaps that was just coincidence/placebo.

In fact, even reinstalling the plugin (settings were wiped), and remaking a playlist or two, it generated empty and unopenable... and these new playlists also were empty when seen on my phone.

Just some more info. I'm going to test and see if the playlists start working again if i leave the plugin uninstalled. Perhaps it is the retry/send anyway even on failure that's keeping it in this weird state. I'll keep you posted

[Noggog]

Playlists started to be accessible again. Not sure if that's because I went home to a different computer or time has passed.

[Noggog]

Just dumping more data points.
Got into work this morning (my home computer is off, so it's the first computer connected to the internet in ~10 hours). Started making my playlists again, but with a limit of 200. All seemed okay and was chugging along listening to one as I went. I had one playlist that was not yielding the expected results (maybe just paranoia or bad rules), but I decided to reload my google play to get a different random seed. Now the playlists are unopenable. Something seems weird, as sync is set to 240 minutes, and my playlists are now a meager 200 vs my old 1000. I don't know how I could've sync spammed it to being broken again. I guess I was resyncing playlists while creating the rules, but not excessively.

[simon-weber]

Hm, ok. Thanks for the info.

I'm also wondering if the overall size of requests is a problem. Since the new apis support batching I'm sending all changes at once for all playlists. This is intended to minimize the number of UI refreshes (there's one per api call), but if sending more smaller batches helps prevent problems then we can trade it off.

In your case, this would be something like 1.4k (==200*7) individual changes in one request, which does seem big given that they usually struggle to sync a maxed out 1k playlist. I'll take a look in GA and see if I can tease apart differences between total changes and changes per request, but I'm not sure I have the data for it.

[Noggog]

The fact that Google is even remotely having a problem here blows my mind haha. Even going back to the concept that we have 1k limits to begin with... seems so crazy for a data giant like Google. Can make self driving cars, and host my 12 years of custom music, but don't you try to show them a list of 1000 songs... that's gonna blow the tops off their servers.

But yeah, maybe a compromise of not a GUI refresh per update, but not all at once either might be a good route. This whole thing seems so non-deterministic, though. Just shoot blindly and hope the updates go through, lol.

[simon-weber]

Ok, backoff is looking promising (at least in my contrived test case).

I managed to start getting 500s by adding a bunch of 1k random playlists, then constantly reloading to force an ordering change. Once I got to ~10k+ changes per sync, Google's responses went above 10s and started returning 500s. I couldn't get it to the point where playlists were actually greyed out, but I'm guessing that happens if the problem gets even worse.

The backoff code kicked in and paused all syncs (even additions and deletions, which normally happen right away) for 15 minutes. After the time was up, all the queued syncs drained like normal and periodic syncs started succeeding again.

As a side note, I'm really thankful for the introduction of queuing from #114: without it, it'd be a lot messier to defer syncs properly.

[simon-weber]

5.1.5 is out with the change. I'll be watching the numbers over the new few days to see if it helps.

[Noggog]

Woke up this morning to unopenable playlists. I updated to 5.1.5 when you posted your comment, so I should be using the most up to date internals. The playlists had been accessible last night when I went to bed, but this morning I spun up my work desktop and things were bugged. Both desktops had been off for 10ish hours.

Another note: I've noticed that the playlists I'm following that are owned by other people still work, as well as Google's smart playlists like the "Thumbs Up" playlist. However, all of my playlists don't work, and it's not just my autoplaylists. Ones that were just normal drag and drop playlists don't open in this funk state either.

[simon-weber]

Yeah, I was afraid something like this would happen: the change should prevent this state if it's the result of multiple syncs (and should help recovery if we get into it). But, it won't help much if a single sync is enough to put Google in a bad state. I'm guessing that's what happened this morning.

So, it sounds like we'll need to look into batching. Ideally I think we want to keep batches <500 (going off of syncs that usually succeed on GA), but it gets complicated to split one playlist into two updates because of how their relative addressing works. So, I think I'll probably just start with doing one batch per playlist (and maybe combining small updates together). I should have time for it within the next week.

[Noggog]

Another thing I noticed just now... I have google music open with autoplaylists still installed, and am just listening to all my songs on shuffle, while my playlists are broken. I assumed everything was just freaked out, and I'm waiting for things to settle down.

However, my phone (which usually is able to read the playlists okay) shows that the playlist is still currently updating. One of them updated a few minutes ago, according to the playlist's description. The phone sees that and is able to read/play them okay, while the desktop is still borked.

I did another test, and reopened my google music to reset the random seed and see if that affected the results. It synced and the same songs as before were present, despite the playlist being stamped with a newer time after the reload, so I don't think the seed is being reset while things are freaked out.

Just some odd behavior that might help pinpoint the problem.

[simon-weber]

I think what probably happened there is that Google synced the playlist metadata correctly (the description, timestamp, etc), but not the playlist contents, which is the call that's been failing.

Also heads up that I'm hoping to push a release with batching by the end of the day.

[simon-weber]

Oh, also: I checked my syncing metrics to see the impact of backing off. GA and Google's api manager don't show meaningful decreases in errors, but interestingly sentry does: from ~800/day to ~150/day. I'm not exactly sure why the numbers differ so much between sources, but it does seem promising.

[simon-weber]

Ok, that change is going out now in 5.1.6. It should limit individual requests to 2k mutations max. My completely unscientific testing shows Google takes ~1 second per 1k mutations, so hopefully this prevents us from triggering the 500s (which happen at ~10s responses).

@Noggog when you have a chance, check the box in settings and then reload. You'll likely need to wait one break+recovery cycle to see the impact (since you'll trigger a breaking sync when opening the extension to enable the option).

[Noggog]

Ticked the setting on. It's currently in a broken state, so I'm going to disable until recovery, and then I'll spin it up and let you know how it goes

[Noggog]

No go. 8(
I actually was broken for like 4-5 days, just enabling/disabling the plugin. I only got it fixed after uninstalling, and redoing my playlists yesterday.

This morning, woke up, turned on my computer and after a song and a half of listening the playlists broke again. Not really sure what's causing all this to be honest, and it seems to be getting worse as time goes on. The pattern of when it happens is hard to follow now haha.

[simon-weber]

Hm. The strangest parts to me are 1) why doesn't this seem to be happening to more people, and 2) why does uninstalling seem to make a difference compared to disabling? I can always add more logging and roll these changes out to other users to see if there's an impact on the aggregate metrics, but maybe we're also missing something more specific to your case?

Reading back through the thread, syncing from multiple computers is an early suspicion that may still be relevant (#78). It could actually explain both points 1 and 2 if it's not commonly done and chrome syncs uninstalls differently than disabling.

An easy way to test this would be to add a local option to disable syncing per device. You could just leave one device on and then disable it elsewhere. If it ends up working, I could build a way to handle that behind the scenes.

[Noggog]

1. Yeah, definitely don't have any ideas on that. I don't think I'm doing anything over the top.

• I occasionally accidentally have two computers on. I would say that's the problem, except I've seen a few occasions where it choked when I knew I only had one computer on.. so I don't think it's only when two computers are on. I'll try to be extremely careful this next two weeks to doubly ensure I never have two on.
• I also have 17k songs, which maybe is more than other users? I wouldn't know without asking.
• Lastly I've got about 8 playlists. Maybe others just do 1-2? Again, don't know without asking.
• I've been thinking a lot on what I do different, and I can't imagine I do anything crazy. If things "worked", id have my 8 playlists pulling from 17k songs, and i'd just let them do their thing and re-sync every 4 hours or so. I wouldnt be force syncing, or changing rules all the time, id just be playing from one of them and letting them resync naturally whenever the timer went off.

2. Not sure. I do notice that disabling seems to sync across computers. If I disable on one, when I get to the other it's disabled there too

If you want to have a local sync disable option, that'd be good to have, for sure.

Also, so far, things came back up and are still okay right now. Maaaaybe it was just one last residual throw up of Google sync from the old code? One can hope, haha. I'll let you know how it goes the next few days.

[Noggog]

Been giving it a few days, since this whole thing has been unpredictable. So far, though things seem to have solidified. I do have the experimental checkmark on. It failed that once quickly after upgrading a while back as I explained in my last post, but I'm hoping was just leftover residue.

Since then it's been staying upright. I've been getting less careful the past few days, too, such as leaving two computers on again, and it's done fine so far.

It could just be a good stroke of luck, but it feels fixed in the last few patches.. or at least optimized enough not to choke on whatever the core problem is. Anyway, thanks for all the hard work and hopefully this is gone for good! I will pipe up if if every shows up again.

[simon-weber]

Great to hear! Keep me posted.

If all continues going well, I'll look into rolling out these changes by default.

[Noggog]

I think we can close this. I've been very unsafe with my usage, leaving computers on, upping my playlist size to 500, syncing manually at will, and I haven't seen any issues the past two weeks.

[simon-weber]

Excellent! I'll look into rolling these changes out to everyone.