Feature/metal (Draft)

[vade]

Hi - I want to use this thread to coordinate some Metal work on Lygia

I see 2 ways to do a metal port

1. Dupe files, replace vec3's with float3's etc and ensure things work, and just have a separate set of implementations.
2. Try to get fancy and typedef float3 to vec3 etc in some main metal include, and then import files. There is likely going to be issues like some functions wont clip, pad, or work the same.

Most of this PR for now is replacing vec2/vec3/vec4 with float2/float3/float4 and duping files for *msl and updating any include paths

Todo:

Sampler Logic Metal sampler functions differently than GLSL / HLSL

• Filtering
• Min / Mag Filter
• Metal shader considerations vs GLSL

Bonus Points

• Swift Package Manager implementation - might be nice to point Xcode to this project and get metal code linked up and ready to use?

[patriciogonzalezvivo]

Anton! I'm so glad to see your PR! Can I merge it? or are you still working on it?

This is going to be so much fun!

To your question, Yes!

1 is brute force. Tedious. But have some pros I learned from GLSL/HLSL parity: it's easier to debug and optimize for a specific platform. Also it's best for reducing code size (which is only important if you are doing NFTs using GLSL/WGSL). Maybe better for collaboration, given that the developer only need to know the target language?

2 is definitely interesting. I don't know so much about MSL to see the pitfalls or from what of the current versions GLSL, HLSL, WGSL or CUDA have more commonalities. Have you seen lygia/sample.glsl or lygia/math/transpose this kind of things are my poor man's polyfill attempt.

I sometimes do the mental exercise of thinking on an architecture that only have one source of truth on release get's transcompile to the others. That seams to lead to creating a new more abstract language... or some sort of preprocessing idiom that require parsing. At that point I abandon the idea thinking it will prevent people for contributing.

I had used naga which have some cool GLSL/HLSL/WGLS conversions, BUT I have run into a lot of issues with it.

So I at the moment I keep on the 1 path. Just because get's the job done. But you are an expert on MSL, if you think there is an opportunity and we can pull it off, let's do it! It could be something we start working for a 2.0.0 version.

Let me know know when you are ready for merging, I was going to push the merge button but the "(Draft)" made me think you where still working on it.

I'm excited!

[vade]

Hey! Yes, it's still very much Draft!

I want to try to get this moving a bit and working in a 3rd party code base and document my findings. Theres work to be done on the sampler side, and im trying to remember my MSL nuances, so I def would say this is highly untested just for now!

[apsalin]

Hi guys, do you need any help?

[vade]

Absolutely! Ive been poking at this in my spare (haha whats that) time and TBH it's a low priority for me.

Help is ABSOLUTELY appreciated!

Happy to run this however. I can add you to the WIP Branch thats the source of this PR?

[patriciogonzalezvivo]

What if we merge this? and we work on top of it in patches, instead of trying to do the entire migration all at once?

[vade]

We could do that, but the reality is

• the above is completely untested
• it currently isnt really usable
• no strategy on some metal only stuff rn

How do you plan on coordinating efforts?

[patriciogonzalezvivo]

Good points! Let's work in this branch and use this thread to align efforts until we have some minimum amount of functions that are usable and which implementation can be use as a guidance for other migrations.

How that sounds?

[apsalin]

Sounds good.
@vade yes, please add me to the branch, do you have any suggestions where to start?

[vade]

Hey! Sorry for the slow response!

I'd say for now, its probably best to think about the sampler / sampler args and how we want to convert the function definitions for metal so they are compatible or drop in.

Then we can try to wire up the includes and ensure that groups of functionality work

[patriciogonzalezvivo]

@vade do you have an example of a sampler type / functions in metal?

[apsalin]

@vade thank you for adding me to the branch, in addition to Patricio's question, do you have any preferences for setting up unit tests?

[vade]

No prefs! It would be great to add some Swift tests to compile shaders.

@patriciogonzalezvivo https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf section 2.10

[patriciogonzalezvivo]

I see. What if we declare a default in sampler.mls following sampler.glsl pattern

The others can be wrapped on functions like:

https://github.com/patriciogonzalezvivo/lygia/blob/main/sample/repeat.glsl
https://github.com/patriciogonzalezvivo/lygia/blob/main/sample/zero.glsl
https://github.com/patriciogonzalezvivo/lygia/blob/main/sample/clamp2edge.glsl

Samplers can be "over-writen" per function using #defines. Ex. : https://github.com/patriciogonzalezvivo/lygia/blob/main/filter/bilateral.glsl#L18

But I'm sure I'm not following correctly or missing important parts. Be patient with me folks : )

[vade]

I was thinking the same thing, and we can define overloaded function declarations which allow users to override the border, mip map and wrapping calls while providing sane defaults.

[patriciogonzalezvivo]

Morning folks! any updates in this front? There is a third potential developer that is interested on translating LYGIA to Metal. I was wondering if it's ok to merge what we have so far. So can works as a model for the rest and we move this conversation to an Issue

[apsalin]

I have none; I will follow your lead.

[ronerez]

Hi Everyone,
I would be happy to contribute to the Metal effort. I communicated with Patricio about this earlier today. I'll update once I have more progress. I think I'll start by running through the examples in lygia. I only came across lygia yesterday for the first time so I still don't have a full grasp of the scope of lygia. For example I came across

https://lygia.xyz/generative/voronoi

and thought this would be very nice to translate to metal and use SwiftUI since lately it has become very easy to use metal in SwiftUI using colorEffect, distortionEffect and layerEffect.

https://developer.apple.com/documentation/swiftui/view/coloreffect(_:isenabled:)
https://developer.apple.com/documentation/swiftui/view/distortioneffect(_:maxsampleoffset:isenabled:)
https://developer.apple.com/documentation/swiftui/view/layereffect(_:maxsampleoffset:isenabled:)

I'll be in touch the moment I make some progress and will begin sharing code.

Best,
Ron

[vade]

FWIW I picked up some steam on my personal project that was driving this effort.

On my roadmap. I might try to do something today, but basically:

• samplers
• ensuring linking works in an actual metal driven app
• porting image processing tooling vs more geometry related things for now

Sadly I looked at using Swift package manager and it enforces specific repository code folder layouts that would be major breaking changes, so I think stuff will need to be manually set up in each project that imports Lygia / Metal.

[ronerez]

Sounds good, I'll be happy to write code in metal that will be demonstrated in SwiftUI. For example, possibly bearing some similarities to the format appearing in:

https://github.com/twostraws/Inferno

For example a folder containing the metal shaders and a folder containing the SwiftUI code demonstrating each of the shaders. Of course I'm open to any other suggestions.

I'll also have a look at the format for SPM.