New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Request] A way to fetch the previous frames of a surface. #489

Open
Luke-Nukem opened this Issue Sep 21, 2018 · 19 comments

Comments

Projects
None yet
3 participants
@Luke-Nukem

Luke-Nukem commented Sep 21, 2018

The number one reason for this is to have the ability to pass previous frames to shaders, eg:

final_surf.shader.set_texture_param("source1", frame[0]); // current
final_surf.shader.set_texture_param("source2", frame[2); // three frames ago
@Luke-Nukem

This comment has been minimized.

Show comment
Hide comment
@Luke-Nukem

Luke-Nukem Sep 21, 2018

I did try using things like swap and add_clone combined with a frame tick, but it seems swap/clone update at the same frequency.

Being able to clone a single frame via Squirrel script could work wonders for all sorts of things, but being able to get an array of the last 3 frames or something would be best.

I'm willing to give this a shot at implementing but I'm not sure where in the code-base to start.

Luke-Nukem commented Sep 21, 2018

I did try using things like swap and add_clone combined with a frame tick, but it seems swap/clone update at the same frequency.

Being able to clone a single frame via Squirrel script could work wonders for all sorts of things, but being able to get an array of the last 3 frames or something would be best.

I'm willing to give this a shot at implementing but I'm not sure where in the code-base to start.

@oomek

This comment has been minimized.

Show comment
Hide comment
@oomek

oomek Sep 21, 2018

Collaborator

I was thinking about implementing multipass shaders and/or give an access to the frambuffer of the current frame. Can you give me some examples of the possible uses of having access to the the previous backbuffer?

Collaborator

oomek commented Sep 21, 2018

I was thinking about implementing multipass shaders and/or give an access to the frambuffer of the current frame. Can you give me some examples of the possible uses of having access to the the previous backbuffer?

@Luke-Nukem

This comment has been minimized.

Show comment
Hide comment
@Luke-Nukem

Luke-Nukem Sep 21, 2018

Sure, something like phosphor burn/fade effects would be cool to see on videos or art switching etc. Motion blur when scrolling. Burn effects, that sort of thing.

It's all about looking pretty I guess, some effects just aren't possible unless you have a backbuffer.

Luke-Nukem commented Sep 21, 2018

Sure, something like phosphor burn/fade effects would be cool to see on videos or art switching etc. Motion blur when scrolling. Burn effects, that sort of thing.

It's all about looking pretty I guess, some effects just aren't possible unless you have a backbuffer.

@zpaolo11x

This comment has been minimized.

Show comment
Hide comment
@zpaolo11x

zpaolo11x Sep 24, 2018

Multipass shaders would be a godsend. Also, having access to the framebuffer would allow to, for example, create a "frosted glass" effect over the display? Because I'm struggling to implement such an effect for overlay menus saving a screenshot and loading it into a surface with shaders... It works but it's cumbersome and would be much cleaner if I had access to the framebuffer

screen216

zpaolo11x commented Sep 24, 2018

Multipass shaders would be a godsend. Also, having access to the framebuffer would allow to, for example, create a "frosted glass" effect over the display? Because I'm struggling to implement such an effect for overlay menus saving a screenshot and loading it into a surface with shaders... It works but it's cumbersome and would be much cleaner if I had access to the framebuffer

screen216

@Luke-Nukem

This comment has been minimized.

Show comment
Hide comment
@Luke-Nukem

Luke-Nukem Oct 2, 2018

@zpaolo11x we can do some multipass shaders via the use of multiple surfaces with one shader per surface.

Luke-Nukem commented Oct 2, 2018

@zpaolo11x we can do some multipass shaders via the use of multiple surfaces with one shader per surface.

@zpaolo11x

This comment has been minimized.

Show comment
Hide comment
@zpaolo11x

zpaolo11x Oct 2, 2018

@Luke-Nukem absolutely true, but can be a bit cumbersome sometimes and it would be much cleaner to be able to apply multiple shaders to the same entity. Imagine if you could just add an image and define a transparent "padding" plus a couple of shaders: gaussian blur would be so easier :D

zpaolo11x commented Oct 2, 2018

@Luke-Nukem absolutely true, but can be a bit cumbersome sometimes and it would be much cleaner to be able to apply multiple shaders to the same entity. Imagine if you could just add an image and define a transparent "padding" plus a couple of shaders: gaussian blur would be so easier :D

@Luke-Nukem

This comment has been minimized.

Show comment
Hide comment
@Luke-Nukem

Luke-Nukem Oct 2, 2018

@zpaolo11x yeah I guess so. A simple frame buffer seems like the best way to go? For multi-pass shaders we can write the last pass in to a location in the frame buffer, and for shaders wanting previous frames they'd just read n frames backward.

Just started learning SFML a little more, and it looks like Attract needs to create the frame buffers per surface - how many though? 3-5 frames back?

Luke-Nukem commented Oct 2, 2018

@zpaolo11x yeah I guess so. A simple frame buffer seems like the best way to go? For multi-pass shaders we can write the last pass in to a location in the frame buffer, and for shaders wanting previous frames they'd just read n frames backward.

Just started learning SFML a little more, and it looks like Attract needs to create the frame buffers per surface - how many though? 3-5 frames back?

@Luke-Nukem

This comment has been minimized.

Show comment
Hide comment
@Luke-Nukem

Luke-Nukem Oct 4, 2018

@oomek did you start working on this at all?

Luke-Nukem commented Oct 4, 2018

@oomek did you start working on this at all?

@oomek

This comment has been minimized.

Show comment
Hide comment
@oomek

oomek Oct 4, 2018

Collaborator

I'm still in a process of figuring out the best way of implementing multipass shaders as it's not that easy to make it to work and not screw up the blending of the images and surfaces that have transparent pixels.

Collaborator

oomek commented Oct 4, 2018

I'm still in a process of figuring out the best way of implementing multipass shaders as it's not that easy to make it to work and not screw up the blending of the images and surfaces that have transparent pixels.

@Luke-Nukem

This comment has been minimized.

Show comment
Hide comment
@Luke-Nukem

Luke-Nukem Oct 4, 2018

Would it be possible to implement a back-buffer per surface first then do multipass shaders? Or are those two things intrinsically linked?
I'm still browsing the source myself and trying to learn how it all works, but after using Rust lang for so long it's become Greek 🙄

Luke-Nukem commented Oct 4, 2018

Would it be possible to implement a back-buffer per surface first then do multipass shaders? Or are those two things intrinsically linked?
I'm still browsing the source myself and trying to learn how it all works, but after using Rust lang for so long it's become Greek 🙄

@oomek

This comment has been minimized.

Show comment
Hide comment
@oomek

oomek Oct 4, 2018

Collaborator

I would like to avoid blitting the backbuffer as a base for multipass, but if there is no other way I will fo with that and a flipflop rendertextures. If I go this route I don't see a reason of not going full programmable blending and skip the output merger blending completely. On Opengl ES 2.0 you can just sample the backbuffer with just 2 simple commands, but on normal opengl where all the fragments of many shaders are called simultaneously you come to the common problem with glitches caused by racing conditions.

Collaborator

oomek commented Oct 4, 2018

I would like to avoid blitting the backbuffer as a base for multipass, but if there is no other way I will fo with that and a flipflop rendertextures. If I go this route I don't see a reason of not going full programmable blending and skip the output merger blending completely. On Opengl ES 2.0 you can just sample the backbuffer with just 2 simple commands, but on normal opengl where all the fragments of many shaders are called simultaneously you come to the common problem with glitches caused by racing conditions.

@oomek

This comment has been minimized.

Show comment
Hide comment
@oomek

oomek Oct 7, 2018

Collaborator

Phosphor retention effect, and other relying on a previous frame may be actually possible right now. I've just come up with the way of achieving it by accident:
https://youtu.be/prWnEQ0Da9s
Will post the souce later on.

Collaborator

oomek commented Oct 7, 2018

Phosphor retention effect, and other relying on a previous frame may be actually possible right now. I've just come up with the way of achieving it by accident:
https://youtu.be/prWnEQ0Da9s
Will post the souce later on.

@oomek

This comment has been minimized.

Show comment
Hide comment
@oomek

oomek Oct 7, 2018

Collaborator

It's possible by exploiting the surface's one frame delay and making a feedback loop between 2 surfaces.

layout.nut

local flw = fe.layout.width
local flh = fe.layout.height

local su = []
local sh = []
local width = flw /2.0
local height = flh /1.0
local retention = 0.95

su.push( fe.add_surface( width, height ))
su.push( fe.add_surface( width, height ))

local snap = su[0].add_artwork("snap", 0, 0, width, height )

sh.push( fe.add_shader( Shader.Fragment, "shader.frag" ))
sh.push( fe.add_shader( Shader.Fragment, "shader.frag" ))

snap = su[1].add_clone( snap )
sh[0].set_texture_param( "texold", su[0] )
sh[0].set_param( "retention", retention )
snap.shader = sh[0]

snap = su[0].add_clone( snap )
sh[1].set_texture_param( "texold", su[1] )
sh[1].set_param( "retention", retention )
snap.shader = sh[1]

su[0].set_pos( flw / 2.0, flh / 2.0 )
su[0].origin_x = su[0].width / 2.0
su[0].origin_y = su[0].height / 2.0
su[1].visible = false

shader.frag

uniform sampler2D texture;
uniform sampler2D texold;
uniform float retention;

void main()
{
    vec4 pixel;
    vec4 pixelold;
	vec2 uv = gl_TexCoord[0];
	vec2 uvo = gl_TexCoord[0];
	uvo.y = 1.0 - uvo.y;
	pixel = texture2D( texture, uv );
	pixelold = texture2D( texold, uvo );
	pixel = max( pixel, pixelold * retention );
    gl_FragColor = gl_Color * pixel;
}

Enjoy and post your experiments.

Collaborator

oomek commented Oct 7, 2018

It's possible by exploiting the surface's one frame delay and making a feedback loop between 2 surfaces.

layout.nut

local flw = fe.layout.width
local flh = fe.layout.height

local su = []
local sh = []
local width = flw /2.0
local height = flh /1.0
local retention = 0.95

su.push( fe.add_surface( width, height ))
su.push( fe.add_surface( width, height ))

local snap = su[0].add_artwork("snap", 0, 0, width, height )

sh.push( fe.add_shader( Shader.Fragment, "shader.frag" ))
sh.push( fe.add_shader( Shader.Fragment, "shader.frag" ))

snap = su[1].add_clone( snap )
sh[0].set_texture_param( "texold", su[0] )
sh[0].set_param( "retention", retention )
snap.shader = sh[0]

snap = su[0].add_clone( snap )
sh[1].set_texture_param( "texold", su[1] )
sh[1].set_param( "retention", retention )
snap.shader = sh[1]

su[0].set_pos( flw / 2.0, flh / 2.0 )
su[0].origin_x = su[0].width / 2.0
su[0].origin_y = su[0].height / 2.0
su[1].visible = false

shader.frag

uniform sampler2D texture;
uniform sampler2D texold;
uniform float retention;

void main()
{
    vec4 pixel;
    vec4 pixelold;
	vec2 uv = gl_TexCoord[0];
	vec2 uvo = gl_TexCoord[0];
	uvo.y = 1.0 - uvo.y;
	pixel = texture2D( texture, uv );
	pixelold = texture2D( texold, uvo );
	pixel = max( pixel, pixelold * retention );
    gl_FragColor = gl_Color * pixel;
}

Enjoy and post your experiments.

@Luke-Nukem

This comment has been minimized.

Show comment
Hide comment
@Luke-Nukem

Luke-Nukem Oct 7, 2018

I have a couple of questions since some things seem non-intuitive:

You set the shader twice?

snap.shader = sh[0]
...
snap.shader = sh[1]

Does this not overwrite the previously set shader? Or is it keeping two shaders active on 'snap'?

You set only the "texold" texture parameter, sh[0].set_texture_param( "texold", su[0] ).
Does this mean that "texture" is automatically assigned?

Luke-Nukem commented Oct 7, 2018

I have a couple of questions since some things seem non-intuitive:

You set the shader twice?

snap.shader = sh[0]
...
snap.shader = sh[1]

Does this not overwrite the previously set shader? Or is it keeping two shaders active on 'snap'?

You set only the "texold" texture parameter, sh[0].set_texture_param( "texold", su[0] ).
Does this mean that "texture" is automatically assigned?

@oomek

This comment has been minimized.

Show comment
Hide comment
@oomek

oomek Oct 7, 2018

Collaborator

I set the shader twice on purpose. Because of using add_clone there are actually 2 instances of snap processed and I need 2 shaders to have the unique texture attached to them.
The texture parameter is automaticaly assigned in surfaces and images and contains what you are intending to draw initially.

Collaborator

oomek commented Oct 7, 2018

I set the shader twice on purpose. Because of using add_clone there are actually 2 instances of snap processed and I need 2 shaders to have the unique texture attached to them.
The texture parameter is automaticaly assigned in surfaces and images and contains what you are intending to draw initially.

@oomek

This comment has been minimized.

Show comment
Hide comment
@oomek

oomek Oct 7, 2018

Collaborator

I could use a different name for each clone, but because I'm interested only in the final result I chose to overwrite the variable name.

Maybe there is a way to simplify it. If you can find it feel free to post here or on the forum.
http://forum.attractmode.org/index.php?topic=2496.0

Collaborator

oomek commented Oct 7, 2018

I could use a different name for each clone, but because I'm interested only in the final result I chose to overwrite the variable name.

Maybe there is a way to simplify it. If you can find it feel free to post here or on the forum.
http://forum.attractmode.org/index.php?topic=2496.0

@Luke-Nukem

This comment has been minimized.

Show comment
Hide comment
@Luke-Nukem

Luke-Nukem Oct 7, 2018

I actually tried something quite similar a while ago, but never realised that "texture" parameter was automatically set.

Luke-Nukem commented Oct 7, 2018

I actually tried something quite similar a while ago, but never realised that "texture" parameter was automatically set.

@Luke-Nukem

This comment has been minimized.

Show comment
Hide comment
@Luke-Nukem

Luke-Nukem Oct 8, 2018

Hmm, definitely need a back-buffer for more advanced effects, and so you don't trash the previous frame.

Had a little bit of fun with the loop, and it seems you can do multipass shaders this way, and some basic effects... But damn, really do need a back-buffer if only for unmolested textures and accurate sizes.

Luke-Nukem commented Oct 8, 2018

Hmm, definitely need a back-buffer for more advanced effects, and so you don't trash the previous frame.

Had a little bit of fun with the loop, and it seems you can do multipass shaders this way, and some basic effects... But damn, really do need a back-buffer if only for unmolested textures and accurate sizes.

@oomek

This comment has been minimized.

Show comment
Hide comment
@oomek

oomek Oct 8, 2018

Collaborator

Define unmolested textures :)

Collaborator

oomek commented Oct 8, 2018

Define unmolested textures :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment