Amazon VOD: Kodi crashes opening a video with inputstream.adaptive >21.1.2

[DDuck007]

Starting a video, kodi crashes in 5 of 10 attempts if using inputstream.adaptive with versions > 21.1.2 - mainly if resuming a video.
I compiled the version 21.1.2 for actual Kodi 21a3 and it works - changing to higher versions Kodi sometimes (very often) crashes - also with most actual "Omega"-version.
See also this bug-report where i was advised go create an issue here.
Attached please find a crashlog using actual Omega-version of inputstream.adaptive.
kodi_crashlog_20231007200445.log

[CastagnaIT]

i have no account to test amazon vod and the log unfurnately is not so precise to allow identify the problem

since i see that you are able to compile ISA you can try help us to identify what PR cause the problem
you can make incremental builds starting from PR/commits over 21.1.2 version bump
https://github.com/xbmc/inputstream.adaptive/commits/Omega?after=c40794a10702f6e0ad5120f5959044e26200f3d7+139&branch=Omega&qualified_name=refs%2Fheads%2FOmega

[DDuck007]

oK I will test - will take some time, but will stay tuned and report back …

[DDuck007]

Did some tests today - started from 21.1.3 downwards an still crashing Kodi.
What I saw is that the crash occurs by 100% if the name of the episode includes a german Umlaut like "ä", "ü" - perhaps someone has an idea to reduce the number of compiles to test ...

[CastagnaIT]

usually on these cases from a "list" of commits i choose always a commit in the middle of the list,
then if still dont work, i choose the middle commit of the previous middle list, and so on

this is the list of commits between the working 21.1.2 to 21.1.3
21.1.2-Omega...21.1.3-Omega

so [addon] Add Team Kodi to provider name for sure is working (just after 21.1.2 bump)

i suggest to do test by using always a commit in the middle, e.g.
[DashTree] Fix replaced init segment from SegmentTemplate

if testing this still crash, i could suggest try compile with and without [AdaptiveTree] First rework to cleanup segments

What I saw is that the crash occurs by 100% if the name of the episode includes a german Umlaut like "ä", "ü"

i dont know what say about this, maybe if i can see the relative manifest
but data like the video title usually there are no in to ISA

[DDuck007]

... did it exactly this way ...
The problem was, that I needed a reboot every time I installed a new version - because installing a crashing version after a functional version also resulted in functioning one until reboot.
But now I identified the root-cause - it is this commit:
239525e

[DDuck007]

... just built version 21.3.0-Omega with attached patch and it does not crash ;-) ...
AmVOD_Crash.patch

[CastagnaIT]

nope is not that one the problem,
than commit commit
or you revert it, or you need to cherry-pick fixes from PR [wvdecrypter] Fix memory heap corruption #1346
(should be enough 75b9724)
otherwise you get the memory heap corruption that is another thing

[DDuck007]

Sorry, I do not understand …
It is clear that there is no crash before this commit - or do you mean my patch is Not „perfect“?

[CastagnaIT]

commit 239525e
its cause of another different crash called memory heap corruption (already fixed on PR 1346)
that is a different of the crash on this issue

then,
you need temporary revert 239525e and continue your tests

[DDuck007]

I used the current Omega now and reverted 239525e - will test …

[CastagnaIT]

has no sense that you revert that commit on the top of upstream Omega branch...
you are confusing old bugs already solved with this (different) one

[DDuck007]

I do not know the code so I think I cannot help more than identifying the commit that introduced the problem …
But of course I can test if you tell me what to test …

[CastagnaIT]

i got discount for amazon week account, so i can now test video playback for next 5 days

tried as suggested 10 "video resume" attempts with two tvshows, but no crashes,
have you a movie or tvshow title to suggest for test? (that is not geolocked to your location and not to be paied)

im using ISA builded from current "Omega" branch (that has not been released yet),
if you are able you can try build ISA by using the Omega branch here, for your LibreELEC version to check if upstream github version works correctly

[DDuck007]

There is one movie that crashes every time:
Thunderbirds are go, season 1, episode 21 (Kometenjäger).
I will try, but unfortunately not before Friday :-(

[CastagnaIT]

tried open "Thunderbirds are go" but unfurnately website say that is not available in my country

[DDuck007]

:-( - unfortunately I have no idea what makes a movie crashing - if you can open one movie once, it is working until next reboot - so I was happy to find an episode that crashes by 100%.

[nandbert]

Not sure if this is related, but I also see crashes on Amazon VOD movies. It seems like some work never, some work sometimes, some always.

Log:

2023-11-25 17:00:25.010 T:6641  critical <general>: [script.module.inputstreamhelper] ['ldd', '/storage/.kodi/cdm/libwidevinecdm.so'] cmd failed.
2023-11-25 17:00:25.010 T:6641  critical <general>: [script.module.inputstreamhelper] Failed to check for missing Widevine libraries.

kfoo:~ # ldd /storage/.kodi/cdm/libwidevinecdm.so 
/storage/.kodi/cdm/libwidevinecdm.so: error while loading shared libraries: /storage/.kodi/cdm/libwidevinecdm.so: DT_RELR without GLIBC_ABI_DT_RELR dependency

Happy to provide more information if this is the right thread / issue

[CastagnaIT]

selected lines of log shown has no relation with ISAdaptive crash

would have been helpful to know:
a non-geolocked movie title that make crashes
and if the crash happens only on linux or also on Windows

my account is expired so i cannot test anymore
if i will receive the trial coupon again i will do the trial subscription, no idea if/when

[nandbert]

"The Witches of Eastwick" is a safe crash for me. It seems that I can play it from my Amazon.de and Amazon.com accounts, not sure if this is sufficient (de: crashes, com: buy or rent, so not tested if it really plays) . Is there a better way to tell if it is non-geolocked?

The crash happens on LibreELEC 11.0.3 on a Raspberry Pie 4, no idea about windows

Is there any other helpful information that I could provide?

[andioddi]

Hi all,

I also experience this issue on rpi4 with libreelec 11.0.3 and Kodi 20. I would also like to support by providing debug information to help find the root cause.

[Targunitoth]

It seems to be a Raspberry Pi 4 problem.
Here are more bug reports: Sandmann79/xbmc#716

[fivebanger]

I can confirm this issue:

Amazon streams often crashes on start. LibreELEC re-starts without a reboot. Issue remains after LibreELEC re-start.

Setup:

Raspberry Pi 4
LibreELEC v11.03

Amazon VOD v1.0.2+matrix.1

InputStream Adaptiv = v20.3.11.1 -> plays without problems
InputStream Adaptiv > v20.3.11.1 -> crashes often when starting a stream

[rptaylor]

Yesterday I had the same crash as reported in Sandmann79/xbmc#716 - apparently this reports that the problem started between 20.3.11 and 20.3.12. Comparing commits: 20.3.11-Nexus...20.3.12-Nexus

I think this is also the same as Sandmann79/xbmc#706 too? This one reports that the problem happened between 21.1.2 and 21.1.3: 21.1.2-Omega...21.1.3-Omega

There are only 11 commits between 20.3.11 and 20.3.12 so maybe it would be easier to pin down in v20.3 than v21.1.

It happened on a new RPi 5, with LibreELEC 11.0.4, kernel 6.1.68, kodi 20.2, and inputstream.adaptive Version: 20.3.16.1 that was released only a few days ago. The segfault looks the same:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `/usr/lib/kodi/kodi.bin --standalone -fs'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0xd2370f70 in WV_CencSingleSampleDecrypter::DecryptSampleData(unsigned int, AP4_DataBuffer&, AP4_DataBuffer&, unsigned char const*, unsigned int, unsigned short const*, unsigned int const*) () from /storage/.kodi/addons/inputstream.adaptive/libssd_wv.so
[Current thread is 1 (Thread 0xc65a8280 (LWP 5693))]

Full log file: kodi_crash.log

It was consistently triggered by any show in season 2 of Vox Machina on primevideo, but oddly any show in season 1 worked fine.

[rptaylor]

Trying to follow up from #1394 (comment)
If I understand correctly, although @DDuck007 's bisection work identified a commit that causes a crash, that commit was already known to cause a (potentially different) crash, which had a fix that already went into the next InputStream Adaptiv version ?

• commit 239525e is from [FragmSampleReader] Removed extra memory reserved #1310
• commit 75b9724 is from [wvdecrypter] Fix memory heap corruption #1346

So in the bisection effort to identify where the problem came from, both of those commits need to be either left out or kept in together? @CastagnaIT can you confirm if that is right?
Then there should be two possibilities: the problem can't be reproduced anymore with both of those commits, so you have to proceed to add more later commits to discover when the problem started. Or, it is reproduced with both of those commits, which would indicate the changes to remove extra reserved memory and fix the memory heap corruption are somehow incomplete and still responsible for causing a crash.

[CastagnaIT]

So in the bisection effort to identify where the problem came from, both of those commits need to be either left out or kept in together?

@rptaylor yes
the fix commit 75b9724
fix a crash caused by the old commit 239525e

therefore if you build from an old GH history point where there is no the fix commit 75b9724
you need: add it in your build (cherry pick), or, otherwise you can build by reverting also the 239525e

in short if you build from an old GH history or you add both or you revert both commits,
Don't make a build by keeping only one of these two commits,
otherwise you will cause a bug that is already known and already fixed, and we waste time for nothing

it is possible that the problem is around the memory allocation here,
or also could happens before these commits idk

[DDuck007]

I unfortunately did not understand, what CastagnaIT said ...
I only can say I use the current version(s) with extra memory reserved (keep adding 4096/1024 that were removed in 239525e and everything is working fine ...

[rptaylor]

@DDuck007 Okay, although it may work before 239525e , that isn't specific enough to identify the problem, because 239525e causes a different problem unrelated to this issue.

To know for sure, can you add back that commit 239525e AND 75b9724 together, and test again?

[DDuck007]

Sorry for the late reply - I'll take a look at it as soon as the health is better...

[CastagnaIT]

@rptaylor on your comment above #1394 (comment)
you assume that "season 2 of Vox Machina" cause crasheses every time you play?
can you provide the log with kodi debug enabled? your log have debug disabled

i have for a month primevideo so i can try make some tests
i made a first test on ubuntu, x86, with kodi 21 and unfurnately no crashes
i start suspecting that crash could be catch only on arm CPU devices
i will need to resume my old coreelec arm device and try on it

UPDATE:
i catch the crash by using my old arm with coreelec after trying to play some more episodes, seem the same crash of your log
now the hard thing is find out whats happens

[rptaylor]

I tried updating to 20.3.16.1 again and enabled debug logging, and tried a few of those same videos but the problem didn't happen. It may just be sporadic. I was originally using 20.3.16 when the problem first happened.

[CastagnaIT]

i tried to make a fix (for kodi 21 atm, backport later), see PR #1464
i made some fast test with amazon on aarch64 coreelec and no crashes

here there are some test builds:
https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1464/2/artifacts
but i assume its needed to test on libreelec arm device, that i cant build for you

anyway i attach here my selfbuilded test build for Coreelec aarch64/arm64 cpus
inputstream.adaptive-21.4.1.1_fix_test.zip
maybe its can work also on RPI aarch64 Libreelec (but no idea)

UPDATE EDIT: see my new comment below with new builds

hoping someone is able to give a feedback

[DDuck007]

Thank you - due to a case of illness in the family, it will unfortunately take some time, but I will test it as soon as possible ...

[rptaylor]

The problem randomly started happening again with a different show today. I was able to reproduce it 3 times and captured debug logs. Two of those times, I reproduced it using the same episode. @CastagnaIT I sent them with google drive to the address on your beacons.ai page.

With the logs that failed on the same episode, the logged output stops after processing exactly the same data chunk.

$ tail -n 5 kodi_crashlog_20240131221611.log kodi_crashlog_20240131223526.log

I tried to find the core files for the stack trace /storage/.cache/cores/core.!usr!lib!kodi!kodi.bin.1706769317.847 but they were missing, I think that directory must get erased after kodi restarts.
Hope it helps find the problem!

[CastagnaIT]

@rptaylor you have to use the new isa test builds with kodi 21
as said on my new comment above:
#1394 (comment)
and compile ISA fix PR code by yourself for your device platform if needed

[CastagnaIT]

i think to have found an oversight in my changed code
please use the new test build or rebuild by syncing again with PR
test builds: https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1464/5/artifacts
Coreelec aarch64 build: inputstream.adaptive-21.4.1.1_ce_aarch64.zip

[DDuck007]

Just built a version for RPi5 with your PR as patch and it did not crash when playing episodes that crashed always with previous versions …

[CastagnaIT]

perfect thank you for the feedback