Skip to content

/ servo

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GStreamer plugin should allow GPU video encoding #24831

Closed
asajeffrey opened this issue Nov 21, 2019 · 14 comments
Closed

GStreamer plugin should allow GPU video encoding #24831

asajeffrey opened this issue Nov 21, 2019 · 14 comments
Labels
P-gst-plugin

Comments

@asajeffrey
Copy link
Member

@asajeffrey asajeffrey commented Nov 21, 2019

At the moment we use readback to get each frame out of GPU memory and into main memory. We shouldn't.
@ferjm ferjm added the P-gst-plugin label Nov 22, 2019
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Nov 26, 2019

@sdroege I'm trying to work out how to get the plugin to generate frames in GLMemory. I wrote a simple version https://github.com/asajeffrey/my-gst-plugin/blob/2c346d729ce599a85e44209af1b5e8c6f12510c5/src/myglsrc.rs which casts the output buffer to GstGLMemory and draws on the texture. Unfortunately, running it just produces a black window, no video output :( My command line on macos was:

GST_PLUGIN_PATH=target/debug gst-launch-1.0 myglsrc ! video/x-raw\(memory:GLMemory\),framerate=25/1,width=1024,height=512 ! queue ! autovideosink

Any idea what I'm doing wrong? Can you point me to an example GStreamer source that produces GLMemory?
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Nov 26, 2019

PS: the real thing, which also doesn't work, is at https://github.com/asajeffrey/servo/blob/785e243f1a82818a0cad5ccc97afe1c445540a8f/ports/gstplugin/servosrc.rs
@jdm
Copy link
Member

@jdm jdm commented Nov 27, 2019

https://github.com/asajeffrey/my-gst-plugin/blob/2c346d729ce599a85e44209af1b5e8c6f12510c5/src/myglsrc.rs#L178 is using an alpha of 0, right?
@jdm
Copy link
Member

@jdm jdm commented Nov 27, 2019

Changing 0.0 to 1.0 yields a non-black screen, but it looks like random texture memory.
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Nov 27, 2019

Aaargh, this used to be just RGBx, not RGBA. Random video memory is still not good!
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Nov 27, 2019

Trying it on linux:

gst-launch-1.0 myglsrc ! video/x-raw\(memory:GLMemory\),width=50,height=99 ! gldownload ! ximagesink

produces

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Got context from element 'gldownloadelement0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Setting pipeline to PLAYING ...
ERROR: from element /GstPipeline:pipeline0/MyGLSrc:myglsrc0: Internal data stream error.
Additional debug info:
../subprojects/gstreamer/libs/gst/base/gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/MyGLSrc:myglsrc0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.000034192
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Nov 27, 2019

To be completely accurate, the command line set a slew of env vars, it was actually:

GST_GL_XINITTHREADS=1 \
GST_PLUGIN_PATH=$PWD/target:$PWD/../servo/support/linux/gstreamer/gst/lib \
GST_PLUGIN_SCANNER=$PWD/../servo/support/linux/gstreamer/gst/libexec/gstreamer-1.0/gst-plugin-scanner \
LD_LIBRARY_PATH=$PWD/../servo/support/linux/gstreamer/gst/lib \
  ../servo/support/linux/gstreamer/gst/bin/gst-launch-1.0 \
    myglsrc ! video/x-raw\(memory:GLMemory\),width=50,height=99 ! gldownload ! ximagesink
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Nov 27, 2019

Ah, the problem is the queue transform, if we remove that (and set the alpha to be 1.0) then myglsrc works! Let's see if that applies to servo too...
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Nov 27, 2019

OK, I got it to the point where I can generate GLMemory and feed it into glimagesink, yay! Not so yay is the flickering, which I think is caused by black frames when servo can't keep up with the window's framerate. Adding a videorate element just resulted in a black window.
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Nov 27, 2019

The WIP is https://github.com/asajeffrey/servo/blob/gstplugin-glmemory/ports/gstplugin/servosrc.rs
@asajeffrey asajeffrey mentioned this issue Dec 3, 2019
Merged
4 of 4 tasks complete
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Dec 3, 2019

Fixed the flickering. Submitted a PR.
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Dec 3, 2019

It's not perfect, in particular the plugin expects to be fed GLMemory, and sometimes fails due to pipelines which provide non-GL memory for some reason.
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Dec 3, 2019

Also there's a memory leak, sigh.
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Dec 3, 2019

Ah, the memory leak is probably caused by using into_ptr() to get at the *mut GstGLMemory, but this then doesn't run any drop code. I should be using as_ptr() instead.
bors-servo added a commit that referenced this issue Dec 5, 2019
@bors-servo
Auto merge of #25048 - asajeffrey:gstplugin-glmemory, r=ferjm
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
0d2503e 
GStreamer plugin should use GLMemory

<!-- Please describe your changes on the following line: -->

Get the gstreamer servosrc plugin to generate frames in GLMemory rather than main memory.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix #24831
- [x] These changes do not require tests because it's an embedding perf issue

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
bors-servo added a commit that referenced this issue Dec 5, 2019
@bors-servo
Auto merge of #25048 - asajeffrey:gstplugin-glmemory, r=ferjm
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
b4db5c4 
GStreamer plugin should use GLMemory

<!-- Please describe your changes on the following line: -->

Get the gstreamer servosrc plugin to generate frames in GLMemory rather than main memory.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix #24831
- [x] These changes do not require tests because it's an embedding perf issue

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
bors-servo added a commit that referenced this issue Dec 5, 2019
@bors-servo
Auto merge of #25048 - asajeffrey:gstplugin-glmemory, r=ferjm
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
38eb89c 
GStreamer plugin should use GLMemory

<!-- Please describe your changes on the following line: -->

Get the gstreamer servosrc plugin to generate frames in GLMemory rather than main memory.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix #24831
- [x] These changes do not require tests because it's an embedding perf issue

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
bors-servo added a commit that referenced this issue Dec 5, 2019
@bors-servo
Auto merge of #25048 - asajeffrey:gstplugin-glmemory, r=<try>
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
78120f1 
GStreamer plugin should use GLMemory

<!-- Please describe your changes on the following line: -->

Get the gstreamer servosrc plugin to generate frames in GLMemory rather than main memory.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix #24831
- [x] These changes do not require tests because it's an embedding perf issue

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
@bors-servo bors-servo closed this in 2d2cd2b Dec 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P-gst-plugin
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

GStreamer plugin should use GLMemory
3 participants
@jdm @asajeffrey @ferjm
You can’t perform that action at this time.