Design of the rewrite

[Friz64]

This discussion is intended for deciding on design decisions of the rewrite.

Background: #344

[Friz64]

Abstract design

The current erupt generator works by building an intermediate representation of the data from the vk.xml and the Vulkan C headers. This is then used to generate Rust code.

The reason for why erupt uses the Vulkan C headers in addition to vk.xml, is that vk.xml contains C declarations that need to be accurately parsed. When converting these declarations into the intermediate representation, it's best to use a correct C parser (the erupt generator uses https://github.com/vickenty/lang-c) to achieve that. It's easy to feed it the complete C headers, and then pick out the data you need.

I like this approach and think it's a good idea to keep it.

[Ralith]

Ironically, the drawbacks you highlight for vk-parse are almost exactly my impression of lang-c, which is certainly far more complex, general, and hence difficult to use than a parser specialized for our use case. I generally favor not reinventing the wheel regardless.

That said, I'm not too concerned about misparses either, as we're manually reviewing output, and we don't need to consume arbitrary code, just whatever specifically arises in vk.xml.

[Friz64]

The wheel was already invented over 2 years ago when I initially wrote the current version of the erupt generator. lang-c has served me very well here. It's done and it's tested in the wild. It's future proof. Using the new code would be reinventing it.

[MarijnS95]

I'd argue that it adds more maintenance work, because it converts every XML element and attribute into a Rust representation (even if not needed). This is redundant, as the analysis part of the rewrite (when splitting the generator in analysis and codegen) will have it's own representation anyways. You'd end up with less total code when reading vk.xml directly through an XML parsing library

@Friz64 In this case, do you still describe the vk.xml schema in Rust and run it through a parser like serde? IMHO I don't want to conflate all the business/analysis logic of the high-level meaning of vk.xml with low-level iterating of XML elements and attributes, especially if the parsing logic looks anything verbose like vk-parse.

At the same time we might want to refer back to earlier XML elements and attributes, which would need to be re-parsed this way (but more likely: design analysis in such a way that can rely on earlier parsed structures, vk.xml is ordered that way iirc).

[Friz64]

Working with the XML directly isn't as verbose as the code in vk-parse, I've had great success using https://crates.io/crates/roxmltree in erupt. Describing the schema in Rust may still be a good idea though, because of the reasons you have mentioned. It allows us to be independent of the XML order. There are many crates that do this, some using serde, some not (as doing XML through serde is a bit hacky).

[Friz64]

Essentially where I am at now is that I am failing to find a Rust crate that allows us to correctly represent the vk.xml schema in Rust. They always seem to trip up at the problem of XML elements between text, which is very common in vk.xml, e.g.

<type requires="VkFormatFeatureFlagBits" category="bitmask">typedef <type>VkFlags</type> <name>VkFormatFeatureFlags</name>;</type>

I think we should try to use roxmltree directly again. Even when the schema is in Rust, you'd still work through the XML the same way (accessing attributes and children etc.), it's not that much less verbose.

---

Otherwise, some stuff came up when testing the idea of running each C declaration through lang-c individually.

1. The Vulkan headers contain C bitfields, an example for this is:

       <type category="struct" name="VkAccelerationStructureInstanceKHR">
           <comment>The bitfields in this structure are non-normative since bitfield ordering is implementation-defined in C. The specification defines the normative layout.</comment>
           <member><type>VkTransformMatrixKHR</type>                                    <name>transform</name></member>
           <member><type>uint32_t</type>                                                <name>instanceCustomIndex</name>:24</member>
           <member><type>uint32_t</type>                                                <name>mask</name>:8</member>
           <member><type>uint32_t</type>                                                <name>instanceShaderBindingTableRecordOffset</name>:24</member>
           <member optional="true"><type>VkGeometryInstanceFlagsKHR</type>              <name>flags</name>:8</member>
           <member><type>uint64_t</type>                                                <name>accelerationStructureReference</name></member>
       </type>

   You can't parse bitfields without your declaration being in the context of a struct. This requires hacks to work around.

2. The lang-c library has some internal state, where every time it encounters a symbol being defined, it remembers whether it is a type name (e.g. a typedef) or an identifier (e.g. a variable). It later uses this information to identify typedefs in declarations. So, whenever a typedef is referenced when parsing standalone, it errors out. This would again be very hacky to work around.

All this points us in the direction of again parsing everything at once.

[MarijnS95]

Analysis stage

The analysis stage takes parsed vk.xml/video.xml and correlates all the data. Think about propagating type information such as lifetimes up through structs that have other structs as members, or setting up lookup tables for aliases (and any other datastructure that is currently passed around at random or recomputed within Ash's Generator).

Multiple codegen backends

A codegen backend should be able to trivially take this information and generate Rust code from it, without doing lots of data transformation or searching across the datastructures to quickly adjust behaviour.

In addition to the "regular" ash low-level wrapper code, and high-level helpers (currently hand-written, but should be automated with the new generator), I'd also like to do various codegen (into separate crates) to help with building Vulkan layers in Rust (automatically generating type-correct dispatch tables like what the validation layers and tools do).
Each of these could be their own distinct codegen module, operating on (a borrow of) the analysis data, hopefully making it efficient and hassle-free to write.

(@Friz64 If I understood you correctly Erupt already implements a design similar to this 👍)

[Friz64]

I suggest the following rough architecture for the analysis stage:

• The C header is parsed and all potentially useful information is packed into (e.g.) HashMaps in such a way that it can easily be accessed later on.
  • This is built on the assumption that we'll parse all header information at once (see other comment).
• The items in the <types>, <enums> and <command> portions of vk.xml are also parsed and again packed into (e.g.) HashMaps in such a way that they can easily be accessed later on.
• The <feature> and <extensions> portions of vk.xml, which reference gathered items and dictate where exactly in Vulkan they belong to, are iterated, retrieving data gathered in the previous steps, to further process and then pack into the analysis output.
• Post processing, such as lifetime propagation and generation of lookup tables etc.

I still find it difficult to structure code, but I think something like this could work and hopefully continue to be maintainable into the future. Another thing important to keep in mind is video.xml and its accompanying headers. Essentially, it's structured very similarly to the normal Vulkan headers, so being able to run it through the same pipeline would be a win.

[MarijnS95]

Structuring it properly seems to be the hardest task, but could hopefully lead to us being able to cfg-guard things properly without accidentally breaking code-dependencies referencing a certain item without guarding. #448 is somewhat relevant.

[cheako]

I'm once again working on my abstraction above ash. Is there interest in ash-tray's vk-helper module? Recently I was working on implementing a type-state system for command buffers, but I'm finding there are more states than those "outlined" by the spec, specifically inside and outside a render pass has the effect of enabling and disabling some commands... but for whatever reason this is not part of the overview of command buffer lifecycle. I'm curious what happens to a bound pipeline when a renderpass starts or ends? Also binding a pipeline introduces a layout, so there is another state to track.

Does that sound like I'm asking the right questions?

[cheako]

You can use xlst to perform a first-pass to re-write the data in something more manageable, and then use nom on that. I'm just saying it's an option, not that it's best.

[Lokathor]

I was invited to post that I have made a working parser which can scrape all the info out of vk.xml: https://github.com/Lokathor/vkvk/blob/main/generator/src/vk_dot_xml_parser.rs

• It's uses one other small XML tokenizing library to walk the XML nodes, but everything else is all in that one file so you shouldn't have to look a lot of other places to understand it.
• It doesn't ever look at C code.
• The parsing logic should be easy enough to follow along for anyone who is familiar with the XML. Nearly all the work it does is just walk the XML until an interesting part and then record the interesting part within the appropriate struct.
• The whole thing is perhaps too simple, it just panics on anything it doesn't understand in a lot of places. I figured it's better to just crash loudly and the generator can be adjusted as necessary when there's any significant change to the XML layout.

Feel free to make use of some or all of it if you like. The whole thing took like 2 days to do. The parser is really the easy part. It's the code generator that's the hard part.

[Lokathor]

when I take a closer look at how you've currently implemented type parsing I don't believe it to be sustainable. Either better code for this gets written or we use a proper C parser.

@Friz64 i missed this second part of your post before and I'm not sure what you mean by unsustainable.

[Friz64]

@Lokathor Let's discuss this on Discord after I get home from work in a few hours.

[Lokathor]

Update: We had the meeting and didn't take specific notes but here's my version of the notes from what I can remember:

• A parser of the vulkan xml does not need support for any possible XML file with a dash of C code. It only needs to handle one specific file. Any capabilities of the parser that don't get used by the specific current version of vk.xml might just be needless complexity that should be discarded.
• In the past, Friz made a version of Erupt that put every extension under its own crate feature. In practice, build times didn't really go down. Crate features are also a pain in the butt to use because simply put Rust doesn't support them well. Conclusion: using crate features is probably not a good path to go down.
• Rustc has extremely poor parallelism. Cargo has very good parallelism. Breaking big crates into smaller crates will almost always reduce build times when parallelism is available (though it might not be on a free CI runner). Conclusion: It might be advised to split parts of ash into sub crates that can build in parallel (eg: bitflags and enumerations can both build in parallel with each other), then just re-export them from the main ash crate so that users don't have to think about the difference.
• Friz said that he likely will write a new parser rather than trying to salvage/adapt any of the older parsers or use the one I made in February.

[MarijnS95]

• A parser of the vulkan xml does not need support for any possible XML file with a dash of C code. It only needs to handle one specific file. Any capabilities of the parser that don't get used by the specific current version of vk.xml might just be needless complexity that should be discarded.

That's what I expect for a generator here. After all, even the current implementation ad-hoc implements parsers for a few specific C constructs: any new syntax/changes in vk.xml would have to be added manually, and that's fine. Seems it's only a "few" expressions and the types.

[Lokathor]

Well I extracted the xml parser part into a crate called vk_dot_xml_parser. It parses all the data, though it doesn't collect and arrange the data as much as I'd maybe want. Still, it works. Even a perfect parser doesn't make a perfect generator, so for the moment I've moved on to working on the generator part of my stuff. I think other would find it usable if they wanted to try.