Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Since ISA v21.1.1, some high bitrate content has abnormal memory usage that can cause Kodi to crash #1295

Closed
1 of 7 tasks
Liqianyu opened this issue Jun 16, 2023 · 62 comments · Fixed by #1323
Closed
1 of 7 tasks

Comments

@Liqianyu
Copy link

Bug report

Describe the bug

Here is a clear and concise description of what the problem is:

Stream some new episodes(?) such as THE DAY, Black Mirror Season 6 using the Netflix add-on. Playback suffers severe frame drops for 1-2 minutes and eventually freezes.
It's worth noting that the problem seems to only occur at the highest bitrate. Maybe it is necessary to reach a certain bit rate. So we need to manually select the highest bit rate video stream when playing.
ISA v21.1.2 didn't change anything, even worse I think.

Expected Behavior

Here is a clear and concise description of what was expected to happen:

Smooth video playback

Actual Behavior

Severely dropped frames that don't recover over time and eventually freeze.

Possible Fix

Apparently this is a problem introduced since v21.1.1, because if I use v21.1.0, this problem doesn't happen.

To Reproduce

Steps to reproduce the behavior:

1.Open Netflix add-on
2.Play specific content, such as Black Mirror Season 6
3.Choose 4K, best bitrate to play.

Debuglog

The debuglog can be found here:
https://paste.kodi.tv/nufepuyara.kodi

MPD/M3U8s/ISM

An example or copy of a manifest (or manifests for HLS - master and variants) can be found here:
https://paste.kodi.tv/uruwirupiy

Screenshots

Here are some links or screenshots to help explain the problem:

Additional context or screenshots (if appropriate)

Here is some additional context or explanation that might help:

Your Environment

Used Operating system:Shield TV Pro 2019

  • Android

  • iOS

  • tvOS

  • Linux

  • OSX

  • Windows

  • Windows UWP

  • Operating system version/name:9.1.1(Android TV 11)

  • Kodi version:Kodi 21 20230613-2307e4d6

  • inputstream.adaptive version:v21.1.1

note: Once the issue is made we require you to update it with new information should that be required.
Team Kodi will consider your problem report however, we will not make any promises the problem will be solved.

@Kampfader
Copy link

I can confirm the error. I wanted to open an issue about it a few days ago, but couldn't get a useful debug log with the stupid firetv stick 4k. Hardware to slow for debuging.

I tested Disney (StarWars movies) play 5min
And netflix movie (the mother) play 10min

@CastagnaIT
Copy link
Collaborator

CastagnaIT commented Jun 16, 2023

please test with this test build (i deleted my previous message, different link):
https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1292/7/artifacts

@Liqianyu
Copy link
Author

@CastagnaIT
I tested this build, but it didn't work.
About 1 minute and 30 seconds into playback, frames start dropping. Eventually after several severe frame drops Kodi freezes.

@Ronny-nerd
Copy link

Ronny-nerd commented Jun 16, 2023

Hardware: Nvidia Shield TV 2015 and Nvidia Shield TV Pro 2019

Nightly Kodi 21 Omega

inputstream.adaptive Version: 21.1.2 = Not OK
inputstream.adaptive Version: 21.1.0 = OK

Netflix Addon = 4K Movies For example

A Beautiful Life
Blood & Gold
The Mother

Disney+ Addon = 4K Movies For example

Star Wars: Das Erwachen der Macht ( Episode VII )
Star Wars: Die letzten Jedi ( Episode VIII )
Star Wars: Der Aufstieg Skywalkers ( Episode IX )

Have the same problem and can confirm it.

@Ronny-nerd
Copy link

Hardware: Nvidia Shield TV 2015

Nightly Kodi 21 Omega Alpha
inputstream.adaptive version: 21.1.2

Netflix Addon = 4K movies = Not OK
Disney+ Addon = 4K movies = Not OK
Amazon VOD Addon = 4K movies/series = OK

4K movie after a few minutes the picture and sound freezes and then you are out of Kodi.

Here is a Kodi Debug LOG with Netflix Addon:
https://paste.kodi.tv/eyuvohebul.kodi

Here is a Kodi Debug LOG with Disney+ Addon:
https://paste.kodi.tv/okubagelaz.kodi

Translated with www.DeepL.com/Translator (free version)

@Liqianyu Liqianyu changed the title Since ISA v21.1.1 Some Netflix content (4K high bitrate) drops frames and freezes Since ISA v21.1.1 Some 4K high bitrate content drops frames and freezes Jun 17, 2023
@CastagnaIT
Copy link
Collaborator

i dont have any 4k account and no 4k tv on my hand so i am going blind
on some log show some kind of problem with ae sync for audio
but its weird that the problem happens only for some streams

my first suspect for now is for PR 1238
i tried change some code by using a vector buffer no idea if could improve it
you can test here:
inputstream.adaptive-android-armv7-omega-078b8e1.zip
inputstream.adaptive-android-aarch64-omega-078b8e1.zip

note test will not works with D+ because the code change is incomplete, you can test only with N€tflix

@Ronny-nerd
Copy link

Thanks for your help, just tested it but unfortunately it did not solve the problem.

@CastagnaIT
Copy link
Collaborator

there is no problems with Kodi 20?

@Ronny-nerd
Copy link

With this InputStream Adaptive version, there are no more problems under Kodi 20 Nexus on the Nvidia Shield TV 2015 and the Nvidia Shield TV Pro 2019.

Netflix Addon = 4K films/series = OK
Disney+ Addon = 4K movies/series = OK
Amazon VOD Addon = 4K movies/series = OK

https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/Nexus/275/artifacts

@Kampfader
Copy link

there is no problems with Kodi 20?

with the crashes, there are no problems.

I've now watched two films of 20-30 minutes each. only N€tflix

N€tflix
The Mother and A Beautiful Life, Audio Passthrough-OFF == OK
Audio Passthrough-ON (AudioTrack(RAW), Android IEC packer) / Dolby Atmos was recognized == Not OK (after jumping forward 10min) logcat= https://paste.kodi.tv/ovepojirum.kodi

Amazon
The Ground Tour, Audio Passthrough-OFF == OK
Audio Passthrough-ON (AudioTrack(RAW), Android IEC packer) == Not OK (after jumping forward 10min)
logcat = https://paste.kodi.tv/alicuqewek.kodi

Disney
Star Wars IX, Audio Passthrough-OFF == Not OK (play max 5min) file to big https://www.file-upload.net/download-15159229/logcat.txt.html

@Ronny-nerd
Copy link

Ronny-nerd commented Jun 21, 2023

But maybe @glennguy has another idea to solve the problem in this direction:

Kodi 21 Omega: inputstream.adaptive Version: 21.1.2

@glennguy
Copy link
Contributor

Ok so might need some more info.

@Ronny-nerd could you please also supply the manifest file? In inputstream.adaptive settings you can turn on the option to save the manifest file, then please attach here.

Also can you try switching the audio track between 6 channel and 2 channel and see what the behaviour is?

I'm looking at the manifest in the OP and seeing that the NF video played there only has EC3 tracks, yet...
CDVDAudioCodecFFmpeg::Open() Successful opened audio decoder aac
Maybe parsing or setting the wrong codec info @CastagnaIT ? Have to go to bed but can look more later.

@CastagnaIT
Copy link
Collaborator

Maybe parsing or setting the wrong codec info @CastagnaIT ? Have to go to bed but can look more later.

mmh i will try check tonight or tomorrow

@Ronny-nerd
Copy link

Hope that worked out with the manifest file.

https://www.dropbox.com/s/32qsyibxk7cod1z/manifests.zip?dl=0

@Ronny-nerd
Copy link

Können Sie auch versuchen, die Audiospur zwischen 6-Kanal und 2-Kanal umzuschalten und zu sehen, wie sich das verhält?

Had changed it in the meantime, but the same problem is still there.

Nvidia Shield TV 2015 is connected directly to the HDR monitor.... no AVR or soundbar is used.

manifests.zip

@Kampfader
Copy link

A question for the developers, I often see the manifest data being distributed via .zip.

When size allows, I always use paste.kodi.tv
The question is, will the manifest text be edited again in paste.kodi? And therefore not presented in a way that is easy to understand for development?

@CastagnaIT
Copy link
Collaborator

so by reading n€tflix log at comment #1295 (comment)
UpdateSampleDescription: codec 1836069985 == mp4a
more likely has been selected the stereo audio track in that log

but in the log attached in the Issue description:
UpdateSampleDescription: codec 1700998451 == ec-3
so same to my shield despite i cannot get atmos but only dolby 5.1

i think its not a problem of wrong format type

i noticed there are tons of prints:
CAEStreamParser::TrySyncAC3 - Unexpected stream type: 0 (wantEAC3dependent: true)
that is not printed if you use an old kodi 21 nightly e.g. (Git:20230423-6e84065c25)

however maybe it could means some data problem? maybe slightly corrupted?
consider this affect both HLS (D+) and DASH (n€tflix) there should be some common code used that has been changed that cause this problem

@Kampfader depends from use cases,
for me pastebin its not safe to use with different text encoding than utf8 or text that contains particular unicode chars like some subtitles, e.g. SmoothStreaming manifest its not safe use pastebin due to different encoding but for HLS/DASH ok, pastebin avoid us to download many times a file that often need only to be read temporary, but also have an expire date so over time it will no longer be possible to make comparisons, at the end its also your choice

@CastagnaIT
Copy link
Collaborator

CastagnaIT commented Jun 27, 2023

a question you are using "android IEC packer"?
if so go to kodi audio settings and set "Kodi IEC packer" and try again

i have no way to investigate on this
the way to try find the problem is that someone test each code change
starting from last commit of v21.1.0 then build each PR until the problem appears

@Liqianyu
Copy link
Author

No, I always use Kodi IEC Packers. at this time I am unable to test further and hope someone else can. I agree that the best way to troubleshoot the issue is to try all commit changes for v21.1.0-v21.1.1 until we find the problem. It shouldn't take long to use the dichotomy.

@Kampfader
Copy link

Does it really run without crashing with 21.1.0? I haven't tested the version myself, I'll do it today after work. I use the last kodi nightly for

@Liqianyu
Copy link
Author

Someone on the Kodi forums reported the failure as a memory leak issue.

@CastagnaIT
Copy link
Collaborator

i made some incremental builds
if someone can test and report what works or not

i suggest to start test from the "test 5" and when fail try olders
no need to test all use cases only one e.g. only n€tflix

inputstream.adaptive-android-aarch64-omega-d660799_test5.zip
inputstream.adaptive-android-aarch64-omega-f5adcc9_test4.zip
inputstream.adaptive-android-aarch64-omega-65dbb6f_test3.zip
inputstream.adaptive-android-aarch64-omega-4b6ca27_test2.zip
inputstream.adaptive-android-aarch64-omega-454eaf0_test1.zip
inputstream.adaptive-android-aarch64-omega-77a58c0_test0.zip (this is same of 21.1.0)

@Ronny-nerd
Copy link

Hardware: Nvidia Shield TV 2015
My test:

_test0.zip (this is same of 21.1.0) = OK
_test1.zip = OK
_test2.zip = Not OK
_test3.zip = Not OK
_test4.zip = Not OK
_test5.zip = Not OK

LOG _test2.zip = Not OK

Problem InputStream Adaptive.zip

@Ronny-nerd
Copy link

_test1.zip = OK

Netflix Addon = OK
Disney+ Addon = OK
Amazon VOD = Not OK (HDR)

@CastagnaIT
Copy link
Collaborator

please test also this same as before with n€tflix
inputstream.adaptive-android-aarch64-omega-b4b069f_test6.zip

@Ronny-nerd
Copy link

_test6.zip = Not OK

After one minute
System
System memory utilisation: over 90%

After two or three minutes = Kodi Crash

@Ronny-nerd
Copy link

LOG _test6.zip = Not OK

Problem 2 InputStream Adaptive.zip

@CastagnaIT
Copy link
Collaborator

CastagnaIT commented Jun 28, 2023

luckily is not mutex problem, apparently is due to parser rework of PR 1067
but that pr dont touch segment data management difficult to understand

with test6 if you set 1080p what happens? do you still see a progressive increase on occupied memory (during playback)?

@Ronny-nerd
Copy link

What happens to test6 if you set it to 1080p? Still seeing a progressive increase in used memory (during playback)?

After 8 - 10 minute
System
System memory utilisation: = 67%

@Liqianyu
Copy link
Author

Liqianyu commented Jul 1, 2023

Apparently there is a memory leak in ISA and Kodi crashes when it runs out of memory, and Android terminates background programs to reclaim memory.
What's strange is that the problem only occurs with some content, not all 4K content encounters this problem.
Memory leaks are not easy to locate, so we need to keep this issue open.

I think we can start by listing the affected platforms and the content for now.

For example
Android Netflix 'Black Mirror' Season 6 4K Bitrate(Optional) Time(Crash time Optional)

@CastagnaIT
Copy link
Collaborator

Android Netflix 'Black Mirror' Season 6 4K Bitrate(Optional) Time(Crash time Optional)

can you explain better what means "Optional"?

@Liqianyu
Copy link
Author

Liqianyu commented Jul 1, 2023

Android Netflix 'Black Mirror' Season 6 4K Bitrate(Optional) Time(Crash time Optional)

can you explain better what means "Optional"?

Bitrate: Does it happen when a certain bitrate is required?
Time: Approximately how long it will take for the crash to occur
Optional means that it can not be filled in, because the highest bitrate is considered by default and a crash will occur soon.

@Ronny-nerd
Copy link

070123141328
070123142827
070123142841
070123142955

@CastagnaIT
Copy link
Collaborator

when you have time please try to see if there is some improvement with this test build:
https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1310/1/artifacts

@Kampfader
Copy link

Im testing the 239525e

N€tflix - 4k The Mother 30min viewed = no crash
System memory usage: max 90%
There is only 3% left to crash, it could be possible that it will crash in an hour run time.

Amazon Vod - 4k StarWars IX with and without HDR10 = crash after 2min
System memory usage: max 93%

Disney - 4k Avatar = crash after 2min
System memory usage: max 93%

I tested the v21.1.0 to see how far going the system memory usage. Max was 65%

@Ronny-nerd
Copy link

Wenn Sie Zeit haben, versuchen Sie bitte, mit diesem Test-Build zu sehen, ob sich eine Verbesserung ergibt: https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1310/1/artifacts

Tested it and still have the same problem with this version.

@Ronny-nerd
Copy link

But the hope remains that you will solve the problem.

@MauriceW67
Copy link

MauriceW67 commented Jul 2, 2023

I did some testing of my own with the Disney+ addon and Kodi Omega Alpha 2.

I don't think the problem has something to do with the bitrate of the video, but rather the length of the video.

With the movie Avatar, the problem also occurs if I choose the 2560x1440 stream with a bitrate of 15 or 18 Mbit/s. Same thing happens with the movie Black Panther: Wakanda Forever. With these movies the memory usage starts at over 80% and quickly rises to 93-95% after which Kodi crashes to the Shield's homescreen.

But if I play a TV show episode of about 40 mins in 4K with a bitrate of 23-24 Mbit/s, the problem does not occur and the memory usage stays at about 73-74%.

Also did some tests with Kodi Nexus and IS:A 20.3.9 and no issues there.

EDIT:

Did some more tests with the Netflix addon and that seems to behave a bit different. I played two episodes of Altered Carbon Season 1 (episode 6 with a length of about 46 mins and episode 7 with a length of about 1h07.).

In both cases the 18 Mbit stream made the memory usage go up to 94-95% and Kodi crashed. The 10 Mbit stream kept playing, although the memory usage did grow to about 83-85%.

@Ronny-nerd
Copy link

In this series "Welcome to Eden" Kodi does not crash.

070223151327
070223151344
070223151359

@Liqianyu Liqianyu changed the title Since ISA v21.1.1 Some 4K high bitrate content drops frames and freezes Since ISA v21.1.1, some high bitrate content has abnormal memory usage that can cause Kodi to crash Jul 3, 2023
@Liqianyu
Copy link
Author

Liqianyu commented Jul 3, 2023

It doesn't look like this should be a memory leak failure because certain content also takes up higher memory but the memory footprint doesn't keep growing. If the memory is large enough (looks like it needs to be 4GB?) then memory exhaustion will not occur and it will not cause Kodi to crash.

However mainstream Android TV set-top boxes only have 2-3G of RAM, which is why the issue only seems to affect devices like Android at the moment, as they have less RAM.

@Ronny-nerd
Copy link

For comparison, the Amazon VOD addon with the movie "Dune (2021)", Kodi does not crash.

Installed InputStream Adaptive version now

https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1311/4/artifacts

070323134049
070323134109
070323134124

@CastagnaIT
Copy link
Collaborator

CastagnaIT commented Jul 3, 2023

@glennguy i think i have found a way to investigate also under Windows os

by using following code snippet you can get HLS manifest from youtube (manifest url expire)

from yt_dlp import YoutubeDL # yt-dlp - https://github.com/yt-dlp/yt-dlp
url = "https://www.youtube.com/watch?v=Hf5enZVznC4"
ydl = YoutubeDL({'format':'bestvideo*+bestaudio/best'})
result = ydl.extract_info(url, download=False)
for format in result['formats']:
    if "manifest_url" in format.keys():
        print(f"found a manifest_url {format['manifest_url']}\n")

specified yt link on the code above, is a video of ~1h with 8K resolution,
(i have taken last url of the manifest list generated)
now playing the 8k resolution with ISA (ask quality) on Window 11 i have these results:

  • Omega master branch:
    3 minuts played, Kodi filled 1700Mb RAM (1,7gb)
  • Omega at version v21.1.0 + recent VP9 commit patch:
    3 minuts played, Kodi filled 860Mb RAM

i can see the gradual memory increase over the time
i hope this makes an easier investigation, since it dont require DRM we can also debugging

@glennguy
Copy link
Contributor

glennguy commented Jul 3, 2023

@CastagnaIT awesome work!
And thanks to everyone else for persisting in looking for a solution.
I'll dive in to this tomorrow night

@Ronny-nerd
Copy link

I'm curious to see what the future holds for us.

@glennguy
Copy link
Contributor

glennguy commented Jul 5, 2023

Very strange behaviour for VP9.

On windows with 4K test stream the memory usage as reported by task manager instantly jumps from around 130MB to 920MB. Usage keeps climbing to around 1250MB and then slowly creeps up, around 50MB per minute. This is all released upon stopping playback though and usage goes back to around 150MB.

There's no h264 4K to compare to, but with 1080p starting at 130MB usage:

  • VP9 - straight up to 350MB
  • H264 - goes to 190MB

Why does Kodi need so much RAM just to start playing a 4K VP9 stream?

@CastagnaIT
Copy link
Collaborator

CastagnaIT commented Jul 5, 2023

Why does Kodi need so much RAM just to start playing a 4K VP9 stream?

yes it is strange, but if you use from this
https://github.com/CastagnaIT/inputstream.adaptive/commits/test_mem_vp9fix
this dont happens...

however take note that on android its used HEVC stream, not VP9

on this comment you can see the results of some tests based on branch commits:
#1295 (comment)
see/find the commit hashes on github history (there is the list of hashes in filenames above) to identify from which point problem starts

i make a lot of attempts changing code on AdaptiveStream::DownloadImpl thinking that for some reason was badly allocated on buffer but found nothing, also the length of segment data downloaded seem ok

i also tried change vector of pointer of AdaptiveStream::segment_buffers_ to a uniqueptr to avoid possible loss of the pointer due to vector, but also this has not solved

@CastagnaIT
Copy link
Collaborator

one interesting thing that i found today is that if you set
on ISA expert setting type of media stream played to "only audio" or "only video"
the memory leak seem dont happen anymore

@Liqianyu
Copy link
Author

Liqianyu commented Jul 5, 2023

one interesting thing that i found today is that if you set on ISA expert setting type of media stream played to "only audio" or "only video" the memory leak seem dont happen anymore

I remember trying "only video", but it didn't work.
I'm not sure though, others can test it again.
Do I think "only audio" is pointless? Because Kodi doesn't load high bitrate video streams.

@MauriceW67
Copy link

@CastagnaIT I tried using "video only" on a 4K HEVC movie in the Disney+ addon, but the memory usage still went to 95% within seconds, causing Kodi to crash.

@Ronny-nerd
Copy link

Ronny-nerd commented Jul 6, 2023

If for example under
Netflix
movie: The Mother
the system memory usage increases after a short time...then the
Render speed: = fps goes down.

Normal: Render speed: 23.4 fps - 24.4 fps
When the render speed: = 19.5 fps and then this movie starts rotate and Kodi crashes.

What amazes me is with Amazon with the movie: "Dune" this movie it does not matter how it looks at the render speed, but it runs stable.

Translated with www.DeepL.com/Translator (free version)

@CastagnaIT
Copy link
Collaborator

we have found one of problems that cause high memory allocation
following test build will work better, but there is still something that need to investigate/fix
https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1323/2/artifacts

@Ronny-nerd
Copy link

Ronny-nerd commented Jul 11, 2023

TOP CastagnaIT

My test:

Netflix Addon = OK
Disney+ Addon = OK
Amazon VOD Addon = OK
MagentaTV Addon = OK

System memory utilisation: = OK

After two or three minutes = No Kodi Crash

@Ronny-nerd
Copy link

following test build will work better, but there is still something that need to investigate/fix

What else should be investigated and fixed?

@CastagnaIT
Copy link
Collaborator

mmh i think was memory difference due to debug build vs release build
the release version now seems good like old ISA version
let me know if you find memory or other weirdness until the end of playback

@Kampfader
Copy link

Fire cube test

Play 2h 10min StarWars IX = all good

Memory usage start 68%
End is at 77%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants