Thoughts on the Kage API

[tinne26]

There are currently multiple open issues about Kage (#2639, #2640, #1431). This has led to some extense debate over whether the current API is easy enough to use, practical, what changes may be needed, which parts are too complex and may actually be considered "bugs", etc.

The following was a comment on one of these issues, but it was rather out of scope, so I'm moving it here instead. The comment is basically a "proposal" of my "ideal model" for Kage in terms of usability. There are some implementation and compatibility hurdles that make it not viable for v2 and probably also impractical or undesirable even for v3, but I think it's useful as a "what would be nice to have" reference and to have some discussion over the direction we are heading towards in the long run. For a comparison, Hajime's current proposal can be found in this google doc, though one really needs to read through the whole doc and issues to have a good grasp on the situation.

Notice that this discussion is in the spirit of "maybe now is a good time to try to find a user-friendlier long-term plan", not "Hajime's proposal is wrong, we should be doing this instead". This proposal is focused on the model, not the implementation or syntax. Let's keep it constructive.

---

My model considers a func Fragment() vec4 with no inputs. The main goal of the model is to offer two sets of functions: one predictable and easy to use that doesn't care much about performance and batching breaks, and another more "low level". The syntax is given as pseudo-code, not final.

The "easy to use" set of functions:

TargetCoords() vec4 // in pixels, =~ `position` - atlas_offsets - border_offsets - subimage_offsets
SubTargetCoords() vec4 // in pixels, =~ `position` - atlas_offsets - border_offsets
TargetSize() vec2 // in pixels
Textures0Mapping() vec2 // src image coords mapped to the current fragment, in pixels
Textures0At(vec2) vec4 // in pixels. refers to source images.
Textures0UnsafeAt(vec2) vec4
Textures0Size() vec2 // all would return the same
Attributes0() float // up to 16 values per vertex

The "low-level" set of functions:

RawTargetCoords() vec4 // in pixels. =~ current `position`
Textures0RawTexelMapping() vec2 // removes some magic
Textures0AtRawTexels(vec2) vec4 // or pixels if that's equally or more efficient
Textures0RawSizeInTexels() vec2 // or pixels if that's equally or more efficient
Textures0UnsafeAtRawTexels(vec2) vec4 // or pixels if that's equally or more efficient

Notice that actual functionality and calls are almost the same that we already have or that Hajime's proposal adds (most parts of this proposal could be implemented through a pre-processor, e.g., on kage-desk/display). The only notable points are:

• Clear separation of easy to use and low-level APIs.
• Removing input arguments from Fragment() is not a direct goal of my model, but it spares the average user from having to deal with input arguments that require understanding internal intricacies of Ebitengine (e.g. atlases and borders) to be used properly.
• Currently in kage there's only one imageSrcRegionOnTexture() function and return value, which is a lie / hack (as different source images can have different origins) that's compensated later on imageSrcNAt(). I dislike this and prefer the more explicit and magic-free Textures[N].RawTexelMapping(). For those wondering, Textures[N].Mapping() would return the same value for all source textures instead (could also be written as TextureMapping() or something).
• Pixels are preferred and prioritized over texels. This matches Hajime's intent on #1431 and I think we are all pretty much on the same page on this.
• Generalization of Attributes is also suggested (see #2640). This is not a core part of this model, but it's motivated by the fact that many shaders operate with simple quads, and when you only have four vertices, being able to pass some values that would typically be hidden uniforms as attributes can actually make sense (not too much overhead, no batching breaking). We would still need to expose something like Image.AtlasOrigin() on the CPU side to make this really useful, but that's a story for another day.

---

The main debate and interest point for the discussion of this "proposal" is mostly whether Kage users think a simpler model is required or not. I feel the users in the current issue discussions are too knowledgeable of Kage and may be slightly biased to understimate the complexity of the shaders, both in their current version and with the suggested changes to fix the withstanding issues. As the main writer of the kage-desk tutorial, I often take more the stance of "how would you explain this to noobs", and in this sense, I think we can do better. But maybe not! Please share your opinions on the topic.

[hajimehoshi]

Removing input arguments from Fragment() is not a direct goal of my model, but it spares the average user from having to deal with input arguments that require understanding internal intricacies of Ebitengine (e.g. atlases and borders) to be used properly.

Unfortunately we cannot do that due to Metal's restrictions: all the arguments are arguments, and Metal shaders cannot treat global variables.

[tinne26]

To be fair, this is probably the least important of the 5 points I made for the general model and ideas. Also, as I say, removing the input parameters is not a direct goal. I think it may be more productive to re-state this point as "avoid having low level, tricky to use input arguments" (EDIT: actually, the original statement says this clearly enough: "[...] spares the average user from having to deal with input arguments that require understanding internal intricacies of Ebitengine". That's the key idea there). I'm sure there are many other ideas to get around this that don't cause problems with globals. An example may be passing the arguments as special types that can only be used as arguments to the built-in functions, which solves both globals and complicated input arguments (at least for the user side... internally it may take some work to do this nicely, of course). And I'm sure there may be other ways to achieve the goals. Whether it ends up being too much trouble for the gain is the other part of the question, but I'm not really specifically advocating for or requesting globals or anything like that.

[hajimehoshi]

I'm not sure what you are suggesting... As long as we cannot change inpute parameters to be global functions, what are you suggesting to change them?

[tinne26]

By "special types" I meant something like this:

func Fragment(position VertexPosition, ...) vec4 {
    _ = TargetCoords(position) // or position.TargetCoords() or whatever
    // ...
}

So we don't need globals, and VertexPosition would be the special type that you basically can only use in predefined, expected ways. I know this also has some implementation complications, but I was only giving it as an example of alternative ideas that avoid trouble with globals while achieving the goal of "sparing the user from having to deal with input arguments that require understanding internal intricacies".

[hajimehoshi]

As we will no longer have to treat texels (#1431), do we still need Textures0RawSizeInTexels() vec2?

[tinne26]

Not necessarily, yeah. My position on the topic could be summarized as: "if texels can be used to make operations more performant than with pixels, then they should be exposed; otherwise only pixels sounds good to me". But this is dependent on details of the underlying graphic libraries. Some of the "raw" functions could be removed, others replaced by the "pixels" version.

[SolarLune]

Following up the discussion from #1870, I think it's imperative that multiple image sizes are supported. Is this possible under current circumstances? If not, how can this be resolved?

For example, would it be resolved if we turned off atlasing for all source and destination images when using a custom shader? Coincidentally, this would fix all inconsistencies and problems with the current Kage API, and simplify things considerably (so there's no need to know about or work with the internal atlas, including origins, texture sizes vs atlas sizes, etc).

[hajimehoshi]

EDIT: However, even an unmanaged image has a thin border on a texture. This is needed to prevent bleeding edges when the image is used as source.

I am sorry but this was my misunderstanding. An unmanaged image didn't have any borders.

ebiten/internal/atlas/image.go

Lines 259 to 264 in 2a1d23d

	func (i *Image) paddingSize() int {
	if i.imageType == ImageTypeRegular {
	return 1
	}
	return 0
	}

[SolarLune]

These questions are where the model I present shines, I think:

* No magic on `imageSrcNAt()`. This means that if different size images are implemented, then the proposal of `Textures0Mapping()` and `Textures0Size()` (ignore syntax / naming) works because each image has independent properties.

* Whether atlasing is on or off, the average user doesn't care because the "easy to use" set of functions already handle that transparently. Then, the other lower-level / raw functions can be used as required if atlasing is known to be off under your current setup, or if you have other ways to deal with it (e.g. passing origin coords within the atlas through vertex attributes when simple geometry is used (for example, on shaders operating on a simple rectangle, where sending the data as vertex attributes at the cost of x4 may be worth it in order to not break batching... but we may need `flat` attributes for this and exposing `image.AtlasOrigin()` or something like that)).

I don't really like having two sets of functions for a single API, if possible, though I do like not having to think about atlasing. My opinion doesn't matter for that much, though, as I'm just looking at it from the perspective of what I personally would like to have, as opposed to what's necessarily possible to implement or what would be backwards compatible.

Maybe it would be best to test out the switch to pixels and see if that solves some of the problems I've had with the current API. Tetra3D is the project of mine that uses shaders heaviest, by far, and it's working fine (though being able to use images of varying sizes should allow me to optimize it a bit more, I think).

[tinne26]

That's actually what I'm most interested on hearing people talk about. Why do you not like two sets of functions? Could you flesh out the issue? We can probably agree that from a design perspective it doesn't sound very elegant, but it seems like an undeniable fact that we do have two situations: when you need something reliable and simple, and when you need to go low level for performance. If you want to serve both use-cases, you need functions for both.

I agree with seeing how things go with the switch to pixels, but yeah, being a heavy and regular user of Kage, you may be less likely to be bothered by usability / complexity issues as you already know a lot about Ebitengine and Kage.

Other random notes:

• Saying "two sets" is only a conceptual division anyway.
• Yeah, I'm really not talking about backwards compatibility here, I openly admitted this doesn't make sense for v2.
• Even without a friendly API we can still have all the necessary functionality, so that's the main point of this discussion: how important is it to have a friendly API, and how that may look like.

[SolarLune]

we do have two situations: when you need something reliable and simple, and when you need to go low level for performance. If you want to serve both use-cases, you need functions for both.

Do we? I don't think we've struggled with performance in a Kage shader itself. I thought the fundamental problem was essentially that Kage doesn't work consistently, and that it has a very "batteries-not-included" feel for building shaders, not that it was slow or needed optimization. I feel like, in general, if most shaders need optimization, it wouldn't be in Kage's functions that sample textures or convert coordinates, but rather in the body of the shader, which is the developer's responsibility - what the developer is doing on the GPU, and how.

As for the API itself, the fewer and simpler the functions provided, the better, in my opinion. If we have more functions (i.e. one "set" for performance and one "set" for usability), that just increases mindspace and the overall footprint of the API. Your proposed model is kind of chunky in the total number of functions - 7 functions for sampling textures just for the "easy-to-use" set feels like too much. Currently, there's 6, and I would probably reduce that down to, like, 2, if possible. Get rid of imagesrcNAt in favor of just the unsafe version (or vice-versa, if undefined behavior could result in a crash on the GPU (?)), replace imageSrcTextureSize() / imageDstTextureSize() with a single function to get the size of any image, get rid of the functions that tell you the image's region on the atlas (as that's an internal detail), etc.

I agree with seeing how things go with the switch to pixels, but yeah, being a heavy and regular user of Kage, you may be less likely to be bothered by usability / complexity issues as you already know a lot about Ebitengine and Kage.

I'm on the side of usability and simplicity; I'd rather the proposed API be simple and easy-to-use, even if it comes at the cost of a bit of a performance hit in terms of actuating the shaders - after all, Ebitengine is a framework for 2D games, first-and-foremost. It's generally unlikely that we need to squeeze absolutely every bit of GPU performance out of custom shaders for 2D games.

Fundamentally, I'm kind of mixed on the changes currently; it feels like I don't grasp the issues at hand enough to offer valid or constructive input. I think I just don't really like the API the way it is currently, but I feel like I don't have much to contribute past that. I still need to re-examine what Kage's current state is to determine my feelings on it.

EDIT:

Saying "two sets" is only a conceptual division anyway.

I see what you mean, I understand.

Yeah, I'm really not talking about backwards compatibility here, I openly admitted this doesn't make sense for v2.

I'm generally anti-backwards compatibility when it comes to game development frameworks; if changes need to be made, I'd make them immediately (unless a lot of people relied extremely heavily on Kage, which I don't think is the case for a primarily 2D game development framework).

Even without a friendly API we can still have all the necessary functionality, so that's the main point of this discussion: how important is it to have a friendly API, and how that may look like.

Understood. If we don't get a friendly API, then for now, I guess my opinion is that it can just stay the way it is, unless things are missing from the current API with the pixels change?

[tinne26]

Oh ok, your previous comment could be interpreter either way so I misunderstood. I'd personally keep both the simple and performant options, but I agree that that could be discussed. I would be hard pressed to decide which one to take if I could only have one. I think you really need the low level ones... not only to make shaders fast by themselves, but also to avoid breaking batching, which may be equally or even more important in many cases.

On the topic of 7 functions being chunky, I think that's misrepresenting it. TargetCoords() and Texture0Mapping() are the "no surprises" equivalents for the current position vec2 and texCoord vec2 input args (and both texCoord and color are just "special" attributes for Ebitengine, so even some of that could be simplified). Then, another 4 functions map to what we already have (I also get rid of texture regions in the atlas), so I was only "adding" SubTargetCoords() as a practical option for working with subimages in some situations. But yeah, that one is not essential. I don't think you can collapse source and destination texture sizes into a single function, but I agree that *Unsafe*() access could be on the "low level" set instead, which you could remove without losing functionality, only performance. I think your minimalism is a bit too extreme in this case, but it's an interesting take and I don't dislike it in spirit.

In terms of back-compatibility, I agree with you, but we know that's not going to happen. Actually, now I was thinking about a maybe crazy idea, but here we go: it may be very interesting to remove all shader functions and option structs and everything from the ebiten package, and then adding a single ShaderSettings field or similar to DrawTriangleOptions and DrawImageOptions, and move shaders to a separate ebiten/kage package and be able to iterate more freely or provide more support for experimentation, debug and customization there.