[C++] Investigate automatic generation of YAML for Substrait extension functions & types

[asfimport]

Arrow supports data types that are not in the Substrait spec (e.g. unsigned integers) and supports functions that are not in the Substrait spec.

As a convenience for consumers we can generate a YAML document that describes all the things Arrow supports that are not part of the Substrait spec. ARROW-15238 has an incomplete example of what this YAML might look like.

This YAML can allow for things like better ahead-of-time validation by milddleman validators and optimizers. It is not strictly required to use these features.

We should investigate our ability to generate this automatically. However, this can get fairly complicated. For example, we don't want to blindly generate YAML for all functions and types because many of the functions and types map to existing functions and types in the core Substrait spec.

Reporter: Weston Pace / @westonpace

Subtasks:

• [C++] Automatically convert from Substrait options to Arrow options object
• [C++] Automatically convert from Substrait options to Arrow options object

_{Note: This issue was originally created as ARROW-15535. Please see the migration documentation for further details.}

[asfimport]

Jeroen van Straten / @jvanstraten:
As a followup to this (or maybe a predecessor, not sure which makes more sense?), it may also be useful to expose an API to generate additional YAML for custom compute functions that the user registers at runtime.

[asfimport]

Weston Pace / @westonpace:
Agreed. We may get that for free or we might not depending on how we tackle this. We can create an additional JIRA / follow-up as we know more.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.