NPO Streaming back-end changed

[wouters]

He guys,

I love the app! It worked for a while but yesterday it stopped working. When I click a livestream or program, it doesn't respond. I see a very faint spinner in the background for about a second with the text "even geduld" but nothing happens. Tried rebooting and reinstalling the app. I also have the "npo live app" from Mauricevb. That one also stopped working so my best guess would be something changed in the links on npo's side. Everything else works fine.

anyone else experiencing this?

[dizzzz]

yes, I was about to file an issue. My colleague confirmed it yesterday eve too. Tonight it still failed.

it looks there is actually an attempt to start the stream.... as @wouters describes.

Latest (nonBeta) tvOS.

[4np]

I just noticed as well... taking a look right now...

[4np]

It seems like the javascript containing the token needed to play the videos is gone...

[4np]

Found the token...

[4np]

Some more things have changed, but it seems to be relatively straightforward fix... I won't manage today anymore, but I will most likely fix this tomorrow... :)

[4np]

I have resolved the issue, but I have to wait until after 19:00 when the NPO3 and BEST24 video streams will start broadcasting to find out what the new stream identifiers are.

The good news is that all streams are now adaptive. The bad news is that the live and themed channels stream have changed for the worse and the maximum quality is now a measly 576p while it used to be 1080p for NPO1, NPO2 and NPO3. Hopefully they have just set it to 576p for evaluation purposes and they will increase it soon to higher qualities... When they do, it should just work transparently...

[dizzzz]

Maybe they have commercial reasons? like http://www.npoplus.nl/

[4np]

Actually, I already found higher quality streams :P But it seems like the web player does not list them...

[4np]

Everything is done, waiting for the NPO3 and NPO Best channels to start broadcasting to obtain the identifiers (the latter starts at 20:00) after which I will start preparing a new release...

[dizzzz]

@4np thnx for the fix! are the streams back on 1080p ?

[4np]

Yep :)

[technimad]

For me the live streams now show a black screen with a 'not allowed' sign:

[4np]

Did you pull the latest code? They work fine on my end?

[wouters]

here it seems to work perfectly! thanks again for the wonderfull app!!!!

[4np]

Good to know! And thanks for the compliment ;) Not though that I still need to create the final release and I just pushed the latest changes for the channels...

[apptvgreat]

Were can i find the ipa to download And try it.

[4np]

There is no ipa, you have to compile it yourself :)

[apptvgreat]

And help jou to make a better app

[apptvgreat]

With Xcode 7 progamm. To work and make ipa app

[technimad]

I did a new clone, but my issue with the black screen on the live streams persists.
On demand streams are working fine btw.

[4np]

I just created release 1.4.0 which fixes this issue and additionally adds a gradient to the tips images (thanks to @technimad - #20). Please upgrade to 1.4.0...

@technimad, that sounds strange... do you see any logs in the Xcode console? You could try cleaning the derived data (e.g. rm -rf ~/Library/Developer/Xcode/DerivedData/) and see if that helps.

[arnoldhoogerwerf]

Hi have the same issue as @technimad. Clicking once more gives me the following message: "this content requires HDCP for playback. Either your HDMI cable isn't properly connected or your HDMI connection doesn't support HDCP."

[4np]

Very strange, I just tried it on my Apple TV 4 and I am able to watch the episodes, live streams and themed channels without any problems (and apparently, so does @wouters ?). I will try my second Apple TV 4 as well.

Questions: Are you a paid Apple developer? Are you abroad? Are you able to watch the same content via the NPO website?

Regarding the HDCP issue, most posts suggest to reconnect the HDMI cable or try a different HDMI cable. Can you try these steps from Apple Support?

[doedje]

Just to share my experience as it might provide insights: 1.4.0 does play episodes without problems. But the live streams do also result in a black screen, like technimad posted above. I will try to clear my DerivedData..... Nope, does not change the results: HDCP issue persists... I changed cables and ports as suggested.

Now I am more than happy with the episodes working again, I never watch live TV. But just to give you some more information I am a paid Apple developer and I live in the Netherlands.

[TookiTheGreat]

This is exactly the kind of behaviour we have been experiencing for several weeks now, only with live streams working and the episodes not. Things seem to be started up OK with "Even Geduld", then a black screen with spinning wait symbol, and then it flashes - briefly - a small red 'no entry' symbol in the left bottom corner, before going back to black and ultimately throwing up the HDCP issue.

We have given up using the app for now. When I get back home this weekend I will try cleaning out everything and install the new version to see what happens. I will keep everyone posted.

[4np]

The app is forcing the video player to a high quality stream, basically ignoring the adaptive stream that the NPO returns. As this may be problematic on lower bandwidth connections (or if perhaps some other funky business on the NPO's end is going on) I have just pushed a change that disables it and makes the player just use the adaptive streams that are actually being returned by the NPO. I just tested them and it looks like it will show a high quality stream (eventually) if your internet connection allows you to. As they are adaptive in nature they might start playing in low quality but as the player is able to buffer the quality may go up to high quality (for npo1, 2 and 3).

Could you (who are experiencing problems with the live feeds) please download / pull the latest code and see if the Live Streams and Themed Channels now work as expected?

Note: you will also see log entries in the Xcode console showing the url of the video stream for debugging purposes... I will disable those later after we have this resolved...

On the Apple Developer forums there are only a handful of questions (without answers) about HDCP on tvOS and most of them are from 2015. A Google search says it might be related to the cable or the TV / HD Receiver itself but it's strange this popped up right when things in the streaming back-end changed. The only thing an Apple employee posted in relation to HDCP is: "HDCP is only enforced for content protected using FairPlay, and there are not currently any APIs available to detect the HDCP state of the HDMI connection".

ps. @TookiTheGreat : do you have that only for the live stream or also for the 'regular' episodes? Perhaps the NPO changed more than just their stream locations... 🤔

[TookiTheGreat]

The live streams worked sometimes, the regular episodes stopped working altogether. It sounds like it is the change from adaptive to fixed quality that may cause the failure. Our Internet is very slow: 5MBps is a good day!

[4np]

@TookiTheGreat : The app has always been forcing the high quality streams for the live channels since they were introduced in 0.2.0 (not for the regular episodes). Around the 1st of March the NPO changed some things in their streaming back-end and the latest changes I made to the App (so those that were pushed after the 1.4.0 release) are now using the adaptive streams (for the live channels; the regular episodes were already using the adaptive streams). So all videos should work better for slower internet connections. I don't know what else changed (region checks?) but if you have some spare time to try it out on your Apple TV I would love to hear if this also solves the issue you were having (#18)?

[technimad]

Pulled in the latest changes and the livestreams work now.

Before nothing was logged in the debug console, now I see a stream url there:
stream url: http://pedgewarer25b.video.kpnstreaming.nl:80/session/dc58ad38-0001-11e7-8242-9cb65498b840/fqimjj/b2b/live/npo/live/npo/tvlive/npo1/npo1.isml/npo1.m3u8?token=f9c63c0576bda5f2c61e7caaffdc9005_1488625764_1488625764

[4np]

Interesting! My streams are coming directly from the NPO so it seems like KPN has some special streaming proxies or custom CDN going on to reduce network bandwidth? 🤔 They seem to use their own Video On Demand implementation...

Your KPN URL also has a token ?token=f9c63c0576bda5f2c61e7caaffdc9005_1488625764_1488625764 added to the URL which is the root cause of why the videos were failing to play on your end. The logic that was forcing the high bitrate videos modified the URL by removing the trailing .m3u8 and appending the high bitrate stream info at the end (so in your case adding it after that token) resulting in an invalid url...

For example I am seeing these stream urls (without trailing token):

stream url: http://l2cmef48e928910058b9568c000000.7801a002692638d1.smoote2f.npostreaming.nl/d/live/npo/tvlive/npo1/npo1.isml/npo1.m3u8

Hopefully it works for the rest now as well...

[TookiTheGreat]

I have just finished testing the latest version and here are my findings:

• live streams work well, a bit blocky at times but overall satisfactory. Don't forget I am in Australia, in a rural area with ADSL and an average download speed (on good days) of 5Mbps, so we're used to blocky video :-(.
• episodes still fail. The "Even Geduld" screen comes up, with the spinning wait pointer, then the screen turns black, and after a short while a "do not go here" symbol briefly flashes in the bottom left corner, before the black screen comes back with the spinning wait pointer. This screen doesn't change , unless I touch the remote, after which I often get the "HDCP" message described above - but not always.

Now, we can still watch episodes through the NPO website, so it is not that we are blocked from accessing anything (with the exception of some geo-blocked content) there. I do want to point out that on the website, the player offers a setting for playing from outside the Netherlands (buitenland aan/uit). We always set this to "aan" since it does seem to stabilize the streaming. So maybe there IS a difference between playing from a foreign location and playing from inside the Netherlands that can explain why the app chokes on the episodes.

Another weird fact is that the Apple TV simulator of my Xcode does manage to play episodes, though with very long wait times. Since both the Apple TV and the simulator go through the same router, I have to rule out firewall issues I reported earlier - those were resolved by whitelisting the URLs given by Jeroen. The only differences I can think of are memory size (my Apple TV has 16Mb if I remember correctly, and I don't know how much memory the simulator allocates) and the OS version. My Apple TV runs the latest version of the OS, whereas the simulator seems to be on an older version. Since the problems started just around the same time my Apple TV updated its OS, I do still suspect something may actually have changed on the box itself that is now causing this problem. Maybe there is a subtle change in the way URLs are handled, or streams are buffered, or even how the 'FairPlay' status (whatever that looks like) is being determined.

I am just speculating here. If I had more time and were at home more often I would try to do some debugging myself, but I am on the road almost full-time till at least mid-April, so all I can do is pass on these observations and speculations and hope they can contribute to a solution in future versions. I have told my wife to go back to using the web-version - which she is OK with for when she is home alone - but we would love to go back to using the excellent app again: that is so much more user-friendly than the website, and usually played in much better quality than the website would.

I still want to give my thanks to Jeroen for doing all the hard work and hope (but expect nothing) a solution can be found.

[TookiTheGreat]

Correction to the above: I was testing the simulator again to see if I could get a trace on the URLs being passed back and forth, and it now throws up the same "do not go there" symbol as the Apple TV itself. So, it seems there is a bit of randomness about whether the simulator gets to play an episode or not. Mmmmm. Maybe it is a timing issue more than anything else? Something timing out due to our Internet's slow speed and very variable latency and bandwidth? Again, just guessing.

Jeroen, are there any programmable time-outs I could turn to infinity to see if that makes a difference?

[4np]

That is indeed strange behaviour and it doe make me think Apple changed something in their video player when they updated tvOS. Unfortunately you do not have any control on the buffering or timeout in AVPlayer... :S I am not sure how we can solve this issue on your end... What URLs are you seeing in your console? The same as I was seeing?

[TookiTheGreat]

I am on the road again so I won't be able to check anything until I am back home in the weekend. Where would I look for the URLs, by the way? So far, all I did was check my firewall logs, which gives me IP-addresses, not URLs.

[4np]

When you run the latest code in Xcode (either in the simulator or on the Apple TV when it's connected via USB-c) you should see some URLs (and / or errors when playing episodes) in your Xcode console. I added some more debug information that should help determining if the experience you are having might be caused by connectivity issues: 'Player is now waiting to play at the specified rate' or 'Playback has stalled'.

[TookiTheGreat]

I will take a look next weekend when I am back home. Thanks.

[TookiTheGreat]

Hi Jeroen,

Just back from China I decided to run some more tests. This is what I get from the debugging window:

uncaught exception in notifier thread: N5realm10LogicErrorE: Bad version number
2017-03-11 14:04:49:192 UitzendingGemist[160:2875] episode stream (begin at 0s) url: https://content10c2d.omroep.nl/urishieldv2/l27m3def9b0972dbca2c0058c37760000000.46044df03e8523e73e80ca64abdada5c/ceresodi/h264/p/0e/10/10/3e/std_VARA_101382875.m4v?odiredirecturl=%252Fvideo%252Fida%252Fh264_std%252Fd203c080b2fcdd9598a9e78522ffdb11%252F58c3775d%252FVARA_101382875%252F1
2017-03-11 14:04:56:526 UitzendingGemist[160:2875] Player is now waiting to play at the specified rate
2017-03-11 14:05:36:146 UitzendingGemist[160:2875] Could not fetch still image (Optional(NPOKit.NPOError.networkError("The request timed out.")))
2017-03-11 14:05:38:309 UitzendingGemist[160:2875] Could not fetch still image (Optional(NPOKit.NPOError.networkError("The request timed out.")))
2017-03-11 14:05:38:310 UitzendingGemist[160:2875] Could not fetch still image (Optional(NPOKit.NPOError.networkError("The request timed out.")))
2017-03-11 14:05:38:312 UitzendingGemist[160:2875] Could not get image for episode (Optional(NPOKit.NPOError.networkError("The request timed out.")))
2017-03-11 14:05:38:314 UitzendingGemist[160:2875] Could not fetch still image (Optional(NPOKit.NPOError.networkError("The request timed out.")))

After those time out messages all I have is a black screen with the little "no entry" symbol in the bottom right corner, and no further signs of activity. My first guess would be that when the system tries to fetch a still image it tends to time out - either because of our slow Internet, or possibly because of some other error, and then fails when it can't get the image.

I can't see where it is trying to get the image from, so I cannot check for URL errors. My firewall is not showing anything flagged as blocked or suspicious. So that makes a simple time out due to slow Internet connection the most likely cause of the failure.

I did check my access to the actual episode, using the URL shown by the debugger, and I can download it in a normal browser (I use Safari on a MacBook Pro) where it will run fine - after a considerable waiting period, though. It can take more than 1 minute before the first still image shows up, and double or triple that time before it starts playing. So, I do have access to the file, it's just very slow, which confirms my suspicion that a simple lack of speed causes the process to time out and fail.

Which begs the question: why is this happening now, and not before? Could it be that previously lower resolution versions were retrieved? Or that the system was either more patient, or more forgiving if it couldn't get the opening still image straight away?

To solve this, is there a way to aim for a lower resolution version? Or is there a way to not let the process fail when it times out?

I have just done a second test where I wait for the fail sign to appear, press 'menu' to go the previous screen, and press 'play' straight away, hoping that some of the file would still be in a buffer somewhere. And it worked! First the opening image showed up, then a fairly long waiting cursor on top of that, and then the whole things plays, in high resolution.

So, a third solution could be to add a retry loop when the first attempt fails (maybe up to 3 or even 5 times) until we get over the initial speed bump.

If I knew where to begin I might try to do so myself, but unfortunately this is a completely unknown environment for me, and I have very little free time between all my traveling at the moment.

[TookiTheGreat]

And now I'm getting crashes with:

libc++abi.dylib: terminating with uncaught exception of type NSException

This is not looking very promising, is it?

[4np]

I am not certain, but you seem to be hitting a URL containing some sort of resource validation urishieldv2 that appears to be doing a redirect to /video/ida/h264_std/d203c080b2fcdd9598a9e78522ffdb11/58c3775d/VARA_101382875/1. I don't know if the video player is able to handle the redirect by itself but it might be that the redirect is causing the issue... However, as you say you are able to see some video after a retry my guess is that it is able to follow the redirect and start buffering the stream. I am not sure how to reproduce this as I don't get these shielded URLs, perhaps only when you are outside of The Netherlands?

This log entry most likely points to the source of the issue you are having:
2017-03-11 14:04:56:526 UitzendingGemist[160:2875] Player is now waiting to play at the specified rate

This basically means that the player is pausing to wait for the buffer to catch up in order to play (at normal playback speed). If that never goes away then the conclusion is that the internet speed cannot keep up with the video player basically making you buffer indefinitely. As the latest update is already using the NPO's adaptive streams the video player will already use the video stream with the highest (or lowest) bitrate that best matches your internet connection speed.

The image failures you are seeing are not the cause of the video issue (these are the still images you see on the episode screen) although the failure cause ("The request timed out") is symptomatic for a slow internet connection speed.

The fact that these image requests time out, the video player appears to be stuck waiting to play at the specified rate and that your do see some video when you retry to play the episode are clear indicators that your internet connection is too slow.

So why is this happening now? There can be many causes as there are many points of failure between here and Australia :) First of all the most likely suspect is that the NPO changed part of their video-streaming back-end so (this issue: #19) perhaps the streams have changed to a higher minimal bitrate making it just a tad bit too heavy for your internet connection. Perhaps your ISP has changed something on your end or somehow your ADSL cable has gained some extra interference from your house to the telco station. Perhaps the route from the NPO to Australia has changed and has negatively impacted your latency? But agreed, this does not look very promising... :/

Perhaps you could try plugging in your Apple TV directly to your internet connection and see if that helped; if so it is clear that your home setup might in part be responsible. If you're using WiFi for your Apple TV perhaps check to see if it works better on an ethernet connection (there might be WiFi interference going on). If you notice that playback does get better, try to look into QoS (Quality of Service) settings on your router to prioritise video streams, etc...

[4np]

I suppose no news is good news (other than Tooki's issue), so I am going to close this issue as it has been resolved for a while.

@TookiTheGreat: You have a separate issue, let's continue in #18 for your particular issue... Note that the geo validation of release 1.4.1 might help show you what episodes can be watched abroad as well... I am seeing more geo restrictions the last week or so, so perhaps are setting them more strictly now.