Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Live, multi-period DASH, Shaka makes thousands of requests for the same segments #1464

Closed
chrisfillmore opened this issue Jun 13, 2018 · 11 comments
Closed

Live, multi-period DASH, Shaka makes thousands of requests for the same segments #1464

chrisfillmore opened this issue Jun 13, 2018 · 11 comments
Assignees
@TheModMaker
Labels
status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Milestone
v2.5

Comments

@chrisfillmore
Copy link
Contributor

chrisfillmore commented Jun 13, 2018 •
edited

(Updated 6/15 with results from some additional testing)
Have you read the FAQ and checked for duplicate open issues?:
Yes

What version of Shaka Player are you using?:
This is reproducible in 2.3.9 and 2.4.1, and master

Can you reproduce the issue with our latest release version?:
This is not reproducible in 2.3.8 or 2.4.0.

Can you reproduce the issue with the latest code from master?:
Yes, I can reliably reproduce in master.

Are you using the demo app or your own custom app?:
Custom app

If custom app, can you reproduce the issue using our demo app?:
No

What browser and OS are you using?:
Chrome 67 macOS

What are the manifest and license server URIs?:
Sent a curl by email to obtain manifest url. Unfortunately it's not practical to share a license since there is a custom authorization flow.

What did you do?
Load multi-period live DASH stream with Shaka

What did you expect to happen?
If a request is successful, Shaka should not re-request the same segment.

What actually happened?
Shaka made thousands of requests for the same segments, in quick succession. Chrome network panel indicated HTTP status 200 (from disk cache) on subsequent requests for the same segment. See attachment below, which shows requests made after a few minutes of playback. It's hard to troubleshoot because DevTools becomes virtually unusable. Also note that I forced Shaka to use http_xhr_plugin to see if it was a problem with Fetch. I was able to reproduce using both plugins.

During this time, the player appears to go in and out of buffering states. It seems to recover at some point (it stops requesting the same segment over and over). Then playback proceeds. Then there is a period of buffering, and then playback proceeds again. Then it reaches a point where it makes several thousand more requests for the same segment, and goes through the whole process again.

This only appears to happen with live, multi-period content. This manifest comes from our ad provider, YoSpace. I was not able to reproduce with VOD or our regular single-period content. (For comparison, this manifest works as expected: https://dfwlive-sponsored.akamaized.net/Content/DASH.abre/Live/channel(CNNHD-8249.dfw.1080)/manifest_mobile.mpd)

For what it's worth I've only seen this happen with audio segments so far. That might be a coincidence. The behaviour looks like Shaka got stuck in a loop with a poorly defined exit condition. I will continue troubleshooting to see if I can learn more. Any ideas in the meantime?

Edit: appears to be caused by b212ef9

shaka_thousands_requests
@chrisfillmore
Copy link
Contributor Author

I tested in 2.4.1 today and can reproduce with that version.

@chrisfillmore
Copy link
Contributor Author

This appears to be caused by b212ef9. If I revert this change, the problem goes away.

Still investigating...

@chrisfillmore
Copy link
Contributor Author

Just to confirm the above, I've tested and can reproduce in 2.3.9, as well as 2.4.1.
2.3.8 and 2.4.0 do not have this problem.

@chrisfillmore
Copy link
Contributor Author

I've sent an email to shaka-player-issues with a curl you can use to get a manifest url.

@chrisfillmore
Copy link
Contributor Author

chrisfillmore commented Jun 15, 2018 •
edited

I have only ever seen this happen so far with audio segments. A sample manifest is attached. One thing I notice is that we have three AdaptationSet for audio in each Period, each AdaptationSet has one Representation. The languages for all are eng and the segments appear to be the same, only the bitrate is different. On further testing, it seems to be no different if I remove 2 of the audio AdaptationSets and leave just one.

sample_multi_period.txt

@chrisfillmore
Copy link
Contributor Author

chrisfillmore commented Jun 15, 2018 •
edited

StreamingEngine gets into a loop of scheduling updates every 0s. This is the V2 logging output, which is repeated over and over:

(audio:10484) updating in 0 seconds
streaming_engine.js:1533 (audio:10484) finished fetch and append
streaming_engine.js:1160 (audio:10484) timeNeeded=338513.9650917
streaming_engine.js:1171 (audio:10484) update_: playheadTime=338509.416251 bufferedAhead=4.5498399999924
streaming_engine.js:1322 (audio:10484) next position known: position=10
streaming_engine.js:1467 (audio:10484) fetchAndAppend_: playheadTime=338509.416251 currentPeriod.startTime=338459.755 reference.position=9 reference.startTime=48.1940917 reference.endTime=54.2100917
streaming_engine.js:1503 (audio:10484) fetching segment
streaming_engine.js:2151 fetching: reference=[object Object]
simple_abr_manager.js:181 Segment downloaded: deltaTimeMs=5 numBytes=73869 lastTimeChosenMs=1529086691745 enabled=true
simple_abr_manager.js:229 Suggesting Streams...
simple_abr_manager.js:245 Still within switch interval...
mp4_parser.js:184 Parsing MP4 box styp
mp4_parser.js:184 Parsing MP4 box sidx
mp4_parser.js:184 Parsing MP4 box prft
mp4_parser.js:184 Parsing MP4 box moof
mp4_parser.js:184 Parsing MP4 box mdat
streaming_engine.js:1822 (audio:10484) checking buffer length
streaming_engine.js:1852 (audio:10484) buffer behind too large: playheadTime=338509.416251 bufferedBehind=30.01016000000527 bufferBehind=30 overflow=0.010160000005271286
streaming_engine.js:1862 (audio:10484) evicted 0.010160000005271286 seconds
streaming_engine.js:1745 (audio:10484) appending media segment
streaming_engine.js:1751 (audio:10484) appended media segment

@chrisfillmore
Copy link
Contributor Author

I think I will put together a simple demo app that I can send over to Shaka team to repro this issue. But that will not happen this week, and I'm on vacation next week, so I likely won't return to this issue until ~Jul 4 at the earliest.

@TheModMaker
Copy link
Contributor

I think it would be much easier with a self-contained repro. Not having a license server makes repro-ing and fixing this hard. So we'll probably be waiting until you send us a repro. One thing you could try is removing this bit in the SegmentIndex. I think it is causing duplicate index entries and confusing streaming.

@TheModMaker TheModMaker added type: bug Something isn't working correctly status: waiting on response Waiting on a response from the reporter(s) of the issue flag: bot ignore Asks CI bots that maintain issues to ignore this issue and removed needs triage labels Jun 19, 2018
@TheModMaker TheModMaker added this to the Backlog milestone Jun 19, 2018
@chrisfillmore chrisfillmore mentioned this issue Jul 9, 2018
Closed
shaka-bot pushed a commit that referenced this issue Jul 10, 2018
@TheModMaker
Don't change start time in SegmentIndex.fit().
59355d3 
Since we have gap jumping, we don't need to worry about a gap at the
beginning of the segment index.  By changing the start time of the
first segment, it makes merging complicated since the pre-fit and
post-fit times are different.  This difference can cause duplicate
entries in the index.

Issue #1464
Closes #1486

Change-Id: Ib3521e186cd8e256366a3c3e922e0764bd58bbd9
@TheModMaker
Copy link
Contributor

I just pushed a change that will probably fix your issue. Could you retry from master or with the latest nightly site? https://nightly-dot-shaka-player-demo.appspot.com/demo/

@TheModMaker TheModMaker added status: waiting on response Waiting on a response from the reporter(s) of the issue and removed flag: bot ignore Asks CI bots that maintain issues to ignore this issue status: waiting on response Waiting on a response from the reporter(s) of the issue labels Jul 10, 2018
@chrisfillmore
Copy link
Contributor Author

The fix seems to be good. So far I haven't seen it get into making thousands of requests like it was before. However, I do still see it periodically request a segment a second time. Here I've sorted the requests in the Network tab by name:

screen shot 2018-07-10 at 9 08 38 pm

screen shot 2018-07-10 at 8 36 40 pm

The behaviour does not disrupt playback, but is there any reason why Shaka should request a segment twice?

@shaka-bot shaka-bot removed the status: waiting on response Waiting on a response from the reporter(s) of the issue label Jul 11, 2018
joeyparrish pushed a commit that referenced this issue Aug 6, 2018
@TheModMaker @joeyparrish
Don't change start time in SegmentIndex.fit().
282ea49 
Since we have gap jumping, we don't need to worry about a gap at the
beginning of the segment index.  By changing the start time of the
first segment, it makes merging complicated since the pre-fit and
post-fit times are different.  This difference can cause duplicate
entries in the index.

Issue #1464
Closes #1486

Change-Id: Ib3521e186cd8e256366a3c3e922e0764bd58bbd9
@chrisfillmore
Copy link
Contributor Author

I consider this issue resolved. I will file a new issue for the duplicate requests if I have an issue with it. Thanks.

@joeyparrish joeyparrish modified the milestones: Backlog, v2.5 Aug 13, 2018
@shaka-project shaka-project locked and limited conversation to collaborators Oct 9, 2018
@shaka-bot shaka-bot added the status: archived Archived and locked; will not be updated label Apr 15, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@TheModMaker TheModMaker

Labels
status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Projects
None yet
Milestone
v2.5
Development

No branches or pull requests
4 participants
@TheModMaker @joeyparrish @chrisfillmore @shaka-bot