8277309: Add support for H.265/HEVC to HTTP Live Streaming

[sashamatveev]

• Added support for fragmented MP4 with HEVC/H.265, H.264 and AAC.
• Added support for elementary AAC streams without any container for audio only streams.
• Added "aacparse" plugin from GStreamer. Required on Linux, since decoder cannot handle AAC elementary streams directly. DirectShow decoder works without it.
• DirectShow H.264 decoder on Windows and H.265/H.264 decoder on Linux will be reloaded when fMP4 stream changes resolution. Dynamic format change did not worked for these streams on Windows and Linux.

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed
• Change requires a CSR request to be approved

Issues

• JDK-8277309: Add support for H.265/HEVC to HTTP Live Streaming
• JDK-8281288: Add support for H.265/HEVC to HTTP Live Streaming (CSR)

Reviewers

• Kevin Rushforth (@kevinrushforth - Reviewer)
• Ambarish Rapte (@arapte - Reviewer)
• Johan Vos (@johanvos - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jfx pull/726/head:pull/726
$ git checkout pull/726

Update a local copy of the PR:
$ git checkout pull/726
$ git pull https://git.openjdk.java.net/jfx pull/726/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 726

View PR using the GUI difftool:
$ git pr show -t 726

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jfx/pull/726.diff

[bridgekeeper]

👋 Welcome back almatvee! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[mlbridge]

Webrevs

• 02: Full - Incremental (49b6b2e4)
• 01: Full - Incremental (08ce8569)
• 00: Full (01720c79)

[kevinrushforth]

/reviewers 2

[openjdk]

@kevinrushforth
The number of required reviews for this PR is now set to 2 (with at least 1 of role reviewers).

[yososs]

Why not use BitSet instead of ArrayList<Boolean>?

---

Because I forgot to add Markdown escaping, the
ArrayList <Boolean> was not being displayed.

[kevinrushforth]

Why not use BitSet instead of ArrayList?

Can you be more specific? The only use of ArrayList that I see in the patch is in code that uses a few already-existing lists. Changing that to some other data structure would be an unrelated change that seems out of scope for this fix.

[yososs]

modules/javafx.media/src/main/java/com/sun/media/jfxmedia/locator/HLSConnectionHolder.java

I noticed that in this change, ArrayList<Boolean> is rewritten as ArrayList<>.
The BitSet rewrite needs some thought, so it might be a good idea to make it a separate issue.
In the file I pointed out, it is used in many more places than the one I pointed out.

[kevinrushforth]

All my testing looks good. I spot checked the new code, and it looks good as well. I left a couple questions for you, but I'll approved it as is.

[sashamatveev]

Fixed several issues found during additional testing:

• sendHeader was true by default which caused sending first and second segment together for MP2TS and MP3/AAC elementary streams. This variable should be true only for fragmented MP4.
• Removed G_SIGNAL_NO_RECURSE flag from javasource signals. From GLib doc "G_SIGNAL_NO_RECURSE - Signals being emitted for an object while currently being in emission for this very object will not be emitted recursively, but instead cause the first emission to be restarted." Basically it caused skipping data segments since in some cases loadNextSegment() was called twice. We do use signal mechanism from separate thread when reading duration. This issue is intermittent. All GStreamer provided elements do not use G_SIGNAL_NO_RECURSE flag, so not sure why javasource was using it.
• Fixed duration calculation. We should not count duration of header, since header does not contain any data.
• Added code which will adjust PTS during rendering to start with 0, otherwise (after duration fix) we will drop frames with PTS > duration. Note: In fMP4 HLS streams PTS starts with N and goes upto duration + N. Our rendering code expects start with 0 and stop at duration.
• Minor code cleanup and removal of unnecessary "if (segment.start != 0)" check in hlsprogressbuffer.

[johanvos]

This looks really great. I'm doing a few more tests before approving, but I don't expect issues.
I wonder if we can add some simple resources that allow testing the different protocols on different platforms?

[openjdk]

@sashamatveev This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8277309: Add support for H.265/HEVC to HTTP Live Streaming

Reviewed-by: kcr, arapte, jvos

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 27 new commits pushed to the master branch:

• eb7fa5d: 8270867: Debug build of WebKit 613.1 fails on Linux
• ff75917: 8283183: Skip failing PredefinedMeshManagerTest tests
• 4afc8af: Merge
• df43d2b: 8282766: Create release notes for JavaFX 18
• d28f3d7: 8282093: LineChart path incorrect when outside lower bound
• c6069d6: 8271054: [REDO] Wrong stage gets focused after modal stage creation
• 1beb323: 8280840: Update libFFI to 3.4.2
• ae419d7: 8282100: Missed top/left bouncing for ScrollPane on Raspberry Pi with Touchscreen
• 263db3d: 8280020: Underline and line-through not straight in WebView
• 2338821: 8269115: WebView paste event contains old data
• ... and 17 more: https://git.openjdk.java.net/jfx/compare/929e7c9217ff7ea04f9bb31bd5c1bf7d74086752...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[kevinrushforth]

I wonder if we can add some simple resources that allow testing the different protocols on different platforms?

I can't think of a good way to do that, especially for this feature, since it requires a web server.

[sashamatveev]

/integrate

[openjdk]

Going to push as commit 424aeba.
Since your change was applied there have been 27 commits pushed to the master branch:

• eb7fa5d: 8270867: Debug build of WebKit 613.1 fails on Linux
• ff75917: 8283183: Skip failing PredefinedMeshManagerTest tests
• 4afc8af: Merge
• df43d2b: 8282766: Create release notes for JavaFX 18
• d28f3d7: 8282093: LineChart path incorrect when outside lower bound
• c6069d6: 8271054: [REDO] Wrong stage gets focused after modal stage creation
• 1beb323: 8280840: Update libFFI to 3.4.2
• ae419d7: 8282100: Missed top/left bouncing for ScrollPane on Raspberry Pi with Touchscreen
• 263db3d: 8280020: Underline and line-through not straight in WebView
• 2338821: 8269115: WebView paste event contains old data
• ... and 17 more: https://git.openjdk.java.net/jfx/compare/929e7c9217ff7ea04f9bb31bd5c1bf7d74086752...master

Your commit was automatically rebased without conflicts.

[openjdk]

@sashamatveev Pushed as commit 424aeba.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

[mlbridge]

Mailing list message from Scott Palmer on openjfx-dev:

On Mar 23, 2022, at 8:20 AM, Kevin Rushforth <kcr at openjdk.java.net> wrote:

On Tue, 22 Mar 2022 08:54:45 GMT, Johan Vos <jvos at openjdk.org> wrote:

I wonder if we can add some simple resources that allow testing the different protocols on different platforms?

I can't think of a good way to do that, especially for this feature, since it requires a web server.

Just wondering why that?s an issue, specifically for HLS. Java?s built-in web server, the old one at com.sun.net.httpserver.HttpServer (i.e. it wouldn?t need JDK 18), would work fine.

E.g. something not much more complicated than:

int port = 8888;
Path testFilesRoot = Paths.get("media");
HttpServer server = HttpServer.create(new InetSocketAddress(port), 0);
server.createContext("/hls", new HttpHandler() {
@Override
public void handle(HttpExchange exchange) throws IOException {
Path path = Paths.get(exchange.getRequestURI().getPath()).getFileName();
boolean isPlaylist = path.endsWith(".m3u8");
boolean isTS = path.endsWith(".ts");
Path assetPath = testFilesRoot.resolve(path);
byte[] data = Files.readAllBytes(assetPath);
exchange.getResponseHeaders().add("Content-Type", isPlaylist
? "application/x-mpegURL"
: isTS ? "video/MP2T" : "video/mp4");
exchange.sendResponseHeaders(200, data.length);
OutputStream os = exchange.getResponseBody();
os.write(data);
os.close();
}
});
server.start();

The content would all be short static test files, easily fitting in the byte array. You could grab something from a public domain source (just edit the paths in the .m3u8 files as needed)

Cheers,

Scott

[kevinrushforth]

Interesting idea. Btw, we don't have any media tests at all in our repo at present, so there is a more basic issue that would need to be solved. We could file an RFE, but there are some challenges in doing anything that is useful enough to justify the effort.