Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A proposed alternative model for the registry #931

Open
zyrolasting opened this issue Feb 24, 2019 · 10 comments
Assignees
Labels

Comments

@zyrolasting
Copy link
Contributor

@zyrolasting zyrolasting commented Feb 24, 2019

New here and lacking historical context, so please bring your tribal knowledge. CC: @oddhack

While old, #19 is interesting. I'm seeing a need to define the registry such it appears in the form most conducive to its consumers while keeping it maintainable. Unless there's a development I missed, right now things look lose-lose for a few cases.

On my end I could not just use an XML parser to back FFI binding generation. I needed to parse C too (1, 2). Right now if I want to simplify my job I need to either get buy-in to add more stuff in vk.xml for my purposes at maintainer expense, or spend enough time on a hybrid parser that I may as well write bindings by hand.

Spitballing here, but what if the registry starts with a Scheme that only models a dependency graph for named concepts? A reverse topological sort then yields a sequence of declarations to inform generation of C headers, or a conversion of the spec to XML, JSON, etc. Using a Scheme is just meant for language extensions when semantics get strained.

I think this model can change the user/maintainer relationship in a good way. In that scenario the onus is on maintainers to model the registry in a language-agnostic way and still generate vulkan.h etc., but it can also place the onus on consumers to do something else without needing maintainers to juggle implementation details for external projects in the same place.

It's possible that I am not communicating what I'm envisioning clearly and can understand if there are still doubts about the benefits. But if I prepared a POC of smaller scope, would you want to see it?

@oddhack

This comment has been minimized.

Copy link
Contributor

@oddhack oddhack commented Feb 25, 2019

The truth is that we don't want to maintain a language-neutral representation. Vulkan is explicitly defined as a C API, and it is much easier for the maintainers to write that at the leaf nodes of vk.xml, than to invent some other representation and then have to convert it to and from C. In particular, every time someone came in to write a Vulkan extension, they'd have to remember how to convert it into whatever the abstract format is, and ensure that the abstract format was getting converted back into the correct C, and that is a big burden. It's almost the only reason the old SGI ".spec" format that we used to use for OpenGL got updated so rarely.

AIUI there's a Python package providing an interface to clang to parse C++ code. That's probably the most direct way to pull out some additional information about member and function parameter types. Alternatively, the complexity of these type declarations is very low, and doing regexp matching and special-casing after stripping out the XML tags would probably suffice; perhaps occasionally there would be new declarations requiring an addition to that. Adding a hook in the scripts to perform this simple-minded parsing and return a parse tree or other data structure informing your FFI would hopefully address the issue at hand.

It is unlikely we will adopt any large-scale changes to the XML format at this point. Khronos is the producer, but also the major consumer. It is deeply embedded in the toolchains for specs, headers, validation layers, C++ bindings, and other stuff; is being used by multiple Khronos APIs; and basically does what we need, with occasional additions as new requirements come up. If there are some annotations that could help inform your use case and that can't be recovered by parsing the C declaration that would probably be fine; stuff like tagging parameters "inout" in the few cases that happens, might be an example.

I'm not sure what you mean by "historical context", but I have more familiarity with the history of this stuff than anyone else.

@zyrolasting

This comment has been minimized.

Copy link
Contributor Author

@zyrolasting zyrolasting commented Feb 25, 2019

Thank you so much for the detailed answer

[...] it is much easier for the maintainers
[...] the complexity of these type declarations is very low

I know you are right. Bear with me, I just need a little more info on what my best play can be here. The docs said this was an XML specification with an implied "it's just XML", which I admit is my misunderstanding. Still, to me that warrants a big asterisk now because I did not see anything in the materials that clarify, in summary, how much the XML part of vk.xml really tells me about the API.

It seems that if I want to generate FFI bindings, then I should just parse the C headers and go off that parse tree unless the XML tells me important things that the C does not.

  • If vk.xml has data for FFI purposes outside of the C bits, then it makes sense to play off your suggestion and support transforming the inner text of vk.xml leaf nodes to an XML form of a C parse tree. Yes, I can lean on a scaffold of regexp casework (as some python packages seem to do), but I'd rather play the long game by working with a complete and unambiguous representation of the spec in a consistent format that can evolve in more precise and correct ways over time. If we're saying to a user, "Use this XML to generate your goods", they are reasonable to expect XML and only XML to serve their needs regardless of our approach.
  • If vk.xml does NOT have data for FFI purposes outside of the C bits, then I think the solution is simpler: Change the doc to caution FFI devs to just parse the C headers directly to generate their stuff and forget they saw the XML.

Knowing my biases and interests as a user, which of the above do you think is a better investment of my time as a contributor? If I did the former, would that be good enough to add as a script here for the people who care to use it?

@krOoze

This comment has been minimized.

Copy link
Contributor

@krOoze krOoze commented Feb 25, 2019

support transforming the inner text of vk.xml leaf nodes to an XML form of a C parse tree.

Somewhat like https://github.com/NicolBolas/New-Vulkan-XML-Format? Maybe you could cooperate with @NicolBolas on it.

@zyrolasting

This comment has been minimized.

Copy link
Contributor Author

@zyrolasting zyrolasting commented Feb 25, 2019

@krOoze That's a star for me, thank you! But if I can request a closing criteria for the issue: Of the bullet points I mentioned, do either sound like good contributions to this repository? I can go third-party, but I'd like to do something that helps everyone at once if I can.

@oddhack

This comment has been minimized.

Copy link
Contributor

@oddhack oddhack commented Feb 25, 2019

Some of the XML annotations on members and parameters might be used to inform your FFI / bindings layer. It really depends on the target language. Some of them, like the 'len' attribute, are useful for some bindings. Some are really specific to validation and probably are irrelevant unless you plan to do some validation as part of the bindings. And if there's actual missing information needed by some language bindings that doesn't exist in either the C declaration or the current attributes, that's something we can talk about adding.

Talking about this in the abstract doesn't give much to move forward with - what specific information are you lacking to do what you want to do?

@oddhack

This comment has been minimized.

Copy link
Contributor

@oddhack oddhack commented Feb 25, 2019

If there's someplace in specific that you would like it called out that making use of vk.xml may require parsing above and beyond an XML library, we can do that. Just point to the place. Note that right at the beginning of the registry.txt document, there are a couple of paragraphs describing "Schema Choices" that speak to this; anywhere else, I'd just add a reference to this language.

@zyrolasting

This comment has been minimized.

Copy link
Contributor Author

@zyrolasting zyrolasting commented Feb 25, 2019

Talking about this in the abstract doesn't give much to move forward with - what specific information are you lacking to do what you want to do?

@oddhack I pointed to an example (#930) re: distinguishing pointer types and non-pointer types for <member> elements in reply to a quote of interest in #19 (comment). Probably not the trickiest example, but enough to be surprising for me mid-project

You expressed interest in partial parsing of the C information anyway, hence me wanting to confirm if that's still your position given your comment there is 3 years old. I think I can add that to the python scripts so that the user can to opt to start from a more fleshed out XML doc.

In that light, all I need from you is to confirm if that's still helpful here. If so, I have what I need to do something and know I'm not wasting any time making something that won't be accepted.

@oddhack

This comment has been minimized.

Copy link
Contributor

@oddhack oddhack commented Mar 3, 2019

To implement this in a way that's non-intrusive to existing use cases, you could write a Python procedure that could be called with a <member> or <param> element and return an AST of the declaration it contains. That's probably what I was alluding to several years ago.

Or, for people who are doing their own parsing and traversal of the XML and not working in our generator framework (and possibly not even in Python), a procedure that would replace the mixed-mode member or param tag contents with an XML representation of the AST. If someone wants to work only in XML then they could apply the latter globally.

Since the use cases for this are not things we're doing ourselves, it's hard for me to judge what is helpful. But if you write a PR that does one of these things and doesn't otherwise affect the scripts, I expect we'll be happy to include it in the hope it will serve your and other outside consumer's needs.

@oddhack

This comment has been minimized.

Copy link
Contributor

@oddhack oddhack commented Mar 18, 2019

@zyrolasting assigning this back to you, if you want to make a proposal along lines of previous discussion. Note that we made some significant internal changes to the generator scripts in the 1.1.104 spec update, and probably some more coming to validitygenerator.py and extensionmetadocgenerator.py soon. So if you're doing this, please make sure you're synced up with master.

@zyrolasting

This comment has been minimized.

Copy link
Contributor Author

@zyrolasting zyrolasting commented Apr 3, 2019

@oddhack Understood, thanks much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.