Audio Visualization

[panxinmiao]

I've recently written some audio visualization code using pygfx out of interest. It might need some tidying up and refactoring.
I'm wondering if you'd be interested in this kind of audio visualization example in pygfx? Or perhaps as a built-in utility in pygfx?

Something like this: ( Remember to turn on the sound :) )

audio.visualizer.mp4

[almarklein]

That's cool! Would an example with sound be possible?

Maybe the logic is a bit niche. You could create pygfx-audio similar to the shadertoy spin-off :)

[Vipitis]

shadertoy will eventually support some audio features too. This includes inputs and outputs. It's on my to-do list but requires me to finish the Multipass PR first. So I will gladly look at some references

[kushalkolar]

This is cool! I'm curious how you synced the live audio with the visualization!

[panxinmiao]

This is cool! I'm curious how you synced the live audio with the visualization!

I have opened a draft PR #765

[panxinmiao]

shadertoy will eventually support some audio features too. This includes inputs and outputs. It's on my to-do list but requires me to finish the Multipass PR first. So I will gladly look at some references

Here, I would like to take a moment to talk about Shadertoy in passing.

It's great to see that you are also interested in this and have continued to expand and maintain it, turning it into an independent project.

Shadertoy originally started as a small built-in tool and example for wgpu-py that I created out of personal interest. I was fascinated by the concept of "using mathematical formulas to create amazing images" on Shadertoy website, so I set out to replicate some of the magical examples from the Shadertoy website using wgpu and WGSL.

After completing the initial version, I did consider adding support for channel inputs and multiple render passes. The functionality wasn't particularly complex to implement, and I even wrote some examples of this case. However, those examples remained on my computer, and those additional features were only noted as to-dos in the code comments.

The main reasons for this were:

1. The tool's initial purpose was not to fully replicate all features of the Shadertoy website. As a built-in tool for wgpu-py, it was intended to serve as a minimal example, and should maintain a minimalistic implementation, demonstrating certain aspects of wgpu-py.

2. Some of the most fascinating examples on Shadertoy require only a single render pass and some seemingly simple code, with no any additional inputs. All the complexity and magic are generated through straightforward mathematical formulas.

3. The most important reason is that supporting these additional features would complicate the tool's usage, making it less suitable as an API.
   As a built-in API tool for wgpu-py, it only needed to take a piece of code (a string), and voila, magical things happen. Supporting channel inputs and multiple render passes would require users to manage these resources (creating input resources, configuring them, assembling resources with render passes, etc.), which isn't very convenient or intuitive through API. To replicate the full functionality of the Shadertoy website, I believe a GUI application would be more appropriate.

After taking a break from work and returning, I was delighted to see Shadertoy become an independent project with you maintaining it and planning new features. I initially intended to tidy up my old code and submit some PRs, but noticed that you had already implemented some of these features or had them in your roadmap. I understand that there are different ways to achieve the same capabilities, and everyone has their own approach and architecture. It's important to maintain consistency and completeness in the overall concept of the program, so I didn't proceed further.

Recently, I also organized my code a bit and uploaded it to GitHub for reference. :)

See: https://github.com/panxinmiao/wgpu-py/blob/shadertoy/wgpu/utils/shadertoy.py

[panxinmiao]

Here are some examples use channel inputs or multiple render passes:

https://github.com/panxinmiao/wgpu-py/blob/shadertoy/examples/shadertoy_glsl_rainforest.py

Video_071713.mp4

https://github.com/panxinmiao/wgpu-py/blob/shadertoy/examples/shadertoy_glsl_dawn.py

Video_071815.mp4

https://github.com/panxinmiao/wgpu-py/blob/shadertoy/examples/shadertoy_glsl_mountain.py

Video_071925.mp4

https://github.com/panxinmiao/wgpu-py/blob/shadertoy/examples/shadertoy_glsl_lover.py

Video_072113.mp4

In this typical multi channel input and multi rendering passes example (Original is here), The API is used in this way, 😅 :

    shader = Shadertoy(
        main_code,
        common_code=common_code,
        buffer_a_code=buffer_a_code,
        buffer_b_code=buffer_b_code,
        buffer_c_code=buffer_c_code,
        buffer_d_code=buffer_d_code,
    )

    shader.buffer_a_pass.channel_0 = shader.buffer_a_pass
    shader.buffer_a_pass.channel_3 = shader.buffer_d_pass

    shader.buffer_b_pass.channel_0 = shader.buffer_a_pass
    shader.buffer_b_pass.channel_1 = shader.buffer_b_pass

    shader.buffer_c_pass.channel_0 = shader.buffer_a_pass
    shader.buffer_c_pass.channel_1 = shader.buffer_b_pass

    shader.buffer_d_pass.channel_0 = shader.buffer_a_pass
    shader.buffer_d_pass.channel_1 = shader.buffer_b_pass
    shader.buffer_d_pass.channel_2 = shader.buffer_c_pass
    shader.buffer_d_pass.channel_3 = shader.buffer_d_pass

    shader.main_pass.channel_0 = shader.buffer_a_pass
    shader.main_pass.channel_1 = shader.buffer_b_pass
    shader.main_pass.channel_2 = shader.buffer_c_pass
    shader.main_pass.channel_3 = shader.buffer_d_pass

    shader.show()

[Vipitis]

See: https://github.com/panxinmiao/wgpu-py/blob/shadertoy/wgpu/utils/shadertoy.py

Thanks for the references, I will gladly take a look tomorrow.

Don't feel discouraged to open PRs or review mine :)

[almarklein]

Closed by #765