feat: create README.mdx generation command

[jvallesm]

What is the purpose of this PR?

Because

• Component docs get outdated easily and keeping them up-to-date is a (mostly) mechanical task.

This PR

• Introduces compogen, a README.mdx generation tool (and potentially more) for components.

compogen

compogen is a generation tool for Instill AI component schemas. It uses the
information in a component schema to automatically generate the component
documentation.

Installation

git clone https://github.com/instill-ai/component
cd component/tools/compogen
go install .

Generate the documentation of a component

compogen can generate the README of a component by reading its schemas. The
format of the documentation is MDX, so the generated files can directly be used
in the Instill AI website.

compogen readme path/to/component/config path/to/component/README.mdx

Validation & guidelines

In order to successfully build the README of a component, the definitions.json
and tasks.json files must be present in the component configuration directory.

The definitions.json file must contain an array with one object in which the
following fields must be present and comply with the following guidelines:

• id.
• title.
• description - It should contain a single sentence describing the component.
  The template will use it next to the component title ({{ .Title }}{{ .Description }}.) so it must be written in third person, present tense.
• version - Must be valid SemVer 2.0.0.
• type - Connector definitions must contain this field and its value must
  match one of the (string) values defined in protobufs.
• available_tasks - This array must have at least one value, which should be
  one of the root-level keys in the tasks.json file.
• source_url - Must be a valid URL. It must not end with a slash, as the
  definitions path will be appended.

Certain optional fields modify the document behaviour:

• public, when true, will set the draft property to false.
• For connector components, the content of prerequisites will be displayed in
  an info block next to the resource configuration details.
  • Note that this section only applies when a connector is being documented,
    i.e. when the --connector flag is passed.`
• A table will be built for the spec.resource_specification properties. They
  must contain an instillUIOrder field so the row order is deterministic.

TODO

• Support oneOf schemas for resource properties, present in, e.g., the Airbyte
  or the REST API connectors.
  • We might leverage some Go implementation of JSON schema. Some candidates:
    • santhosh-tekuri/jsonschema
    • omissis/go-jsonschema
    • invopop/jsonschema
    • swaggest/jsonschema-go
  • The schema loading carried out by the component/base package in
    LoadConnectorDefinitions or LoadOperatorDefinitions might also be
    useful, although it is oriented to transforming the data to a structpb.Struct
    rather than to define the object structure.
• In the "supported tasks" tables, provide better documentation for nested
  arrays and objects (currently the type doesn't support nesting).
• If task definitions contain examples for the (required) input and output
  fields, generate param samples as in https://github.com/instill-ai/instill.tech/blob/dedaaa3/docs/v0.12.0-beta/vdp/ai-connectors/openai.en.mdx#L148
• Implement a way to inject extra sections if a component needs further
  documentation (e.g. by adding a doc.json file with a structured array that
  describes the position and content of the new section.

Next steps

• compogen validate might be used validate any component configuration.
• compogen new [--operator] might be used to generate the skeleton of a component.

[linear]

INS-3584 Create component README through templates

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 46.15%. Comparing base (52804d4) to head (ef16230).

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #59      +/-   ##
==========================================
+ Coverage   45.89%   46.15%   +0.26%     
==========================================
  Files           6        6              
  Lines        1120     1118       -2     
==========================================
+ Hits          514      516       +2     
+ Misses        517      513       -4     
  Partials       89       89              

Flag	Coverage Δ
unittests	46.15% <100.00%> (+0.26%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.