MPEG 2 TS Encapsulation

Extracting and decoding MPEG-H in Transport Stream (TS)

When receiving MPEG-H 3D Audio multiplexed within TS, there are 3 different cases that need to be handled by a receiving client:

• Case 1 Each PES packet contains exactly 1 MPEG-H Audio access unit. This case corresponds to the TS files in our github repository within the directories "single". Since each PES packet contains exactly 1 access unit, the PES payload can be fed directly into the decoder library using mpeghdecoder_process. The timing information from the TS layer needs to be provided to the API via the 'timestamp' parameter. After processing, mpeghdecoder_getSamples can be used to obtain the decoded samples. These can then be played out or written to a wav file by the application layer.
• Case 2 Each PES packet contains (potentially) multiple MPEG-H Audio access units. This case corresponds to the TS files in our github repository within the directories "multi". In this case, each PES packet potentially contains multiple access units. Since the decoder API requires the access units to be fed one by one along with a timestamp, the application layer needs to separate the access units within the PES packet and compute the correct timestamps before being able to feed them to the decoder. This can roughly be done as follows:
  1. Open an MHAS parser library (e.g. https://github.com/Fraunhofer-IIS/mmtmhasparserlib) and feed the PES packet data into the parser.
  2. Set the timestamp to whatever was signalled on the TS layer for the given PES packet.
  3. Extract MHAS packets until a packet of type PACTYP_MPEGH3DAFRAME is extracted.
  4. Feed all packets extracted this way along with the timestamp into the decoder via mpeghdecoder_process.
  5. Compute the duration of the extracted access unit and increase the timestamp accordingly.
  6. Extract decoded samples using mpeghdecoder_getSamples.
  7. If the parser has MHAS packets left, go to 2.
• Case 3 Each PES contains (potentially) multiple MPEG-H Audio access units. It is also allowed to carry a fractional number of access units per PES packet, i.e. MPEG-H access units can be spread across 2 (or even more) PES packets. This case corresponds to the TS files in our github repository within the directories "cont". This is the most involved case since it requires to keep track of the timestamp across multiple PES packets. Roughly speaking, we need to open an MHAS parser to parse the MHAS packets as in the previous case, however since access units can be spread across multiple PES packets, we need to keep the parser open between receiving PES packets and assure that the timestamp handling is done correctly (i.e. for each access unit that commences in a given PES packet, use the timestamp from the TS layer, for every other one compute the correct timestamp from the PES packet timestamp as well as the MPEG-H access unit durations).

The mmtmhasparserlib includes the CPesParser class which supports all 3 cases listed above. It takes a PES packet stream as input and produces MHAS access units with additional timestamp information extracted from the PES packets. See the corresponding example code in pestomhm.cpp for how to use this parser.

For reference, the AndroidX media repository also contains code that can handle all of the above cases. It can be found here:

• MPEG-H Reader (Java)
• MPEG-H Utils (Java)

A good starting point for reading the code may be Line 155 of the MPEG-H Reader.