airbyte-lib: Connector documentation

[flash1293]

Add capabilities to effectively document airbyte-lib-enabled connectors to our doc system:

• Check the metadata whether the connector is published on pypi and adds an icon to the availability row in the header decoration
• Introduce the components AirbyteLibExample and SpecSchema which can be used in documentation files to render a code sample on how to use the connector and a collapsible element describing the full config schema

The new components are picked up by a new remark plugin that's injecting the spec of the connector as a prop to the component so it can be used to render out a generated sample config / the spec schema.

This adds the following dependencies:

• json-schema-faker to generate an example config object only based on the schema
• @headlessui/react for the collapsible implementation
• sanitize-html to safely render HTML from the schema in the docs

See the following connector doc pages for examples:

• Google Drive: Complex config
• Stripe: Medium config
• Apify Dataset: Simple config

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 23, 2024 9:43am

[flash1293]

TODO: Reimplement json schema rendering

[nataliekwong]

Honestly looks way better designed than the out of the box solution! Looks great.
Some minor things I don't feel we have control over, but maybe worth noting: fields are listed in order of appearance (I think), and that means that we list optional fields before required ones. We also mix the schema configuration with the connector configuration settings, so a user is context switching between the two "types" of configuration options.

[flash1293]

Thanks @nataliekwong !

fields are listed in order of appearance (I think), and that means that we list optional fields before required ones

We can actually control this, added some logic to show the required ones first (this is the advantage of using a custom-built solution here)

We also mix the schema configuration with the connector configuration settings, so a user is context switching between the two "types" of configuration options

Not sure I follow, could you elaborate on that point?

[nataliekwong]

Oh nice, thanks for reorganizing! I think it may have resolved itself (in this example, at least) with that change:

I think previously the streams were listed between two other settings, but now it's at the top (Should it be at the bottom? That would then break the "list required fields at the top" rule). I think of the connector's overall configuration settings differently from the streams, because I'd expect the connector configuration to determine the streams I have available. But perhaps that's an incorrect assumption in the case of airbyte-lib. Anyway, following that assumption, I'd then expect to set the connector-level settings before defining specific streams.

[flash1293]

@timroes I adopted @nora-airbyte s design and made it more table-y:

IMHO it looks pretty cool. Could you give it a review?

[timroes]

Why are we putting MacOS specifically here as an example to install those, but no other system?

[flash1293]

This is just an example, I will remove it from this PR before merging (only the pure implementation of the components will go on, rolling it out to connectors on a separate PR)

[timroes]

Does this quickstart guide already exist, so we could link to it?

[flash1293]

It doesn't yet - I will work on this on a separate PR. As mentioned, this component will not be rendered yet, in this PR I just want to get the first version of the components ready.

[timroes]

Generally I feel this looks pretty good. The only thing irking me a bit is how we display arrays. We somehow nest a items[x] key once you expand the actual array, so it looks like the following:

Specifically for arrays of primitive datatypes like string[] this looks imho more confusing then helpful:

Is there any chance for simple arrays to render it as array<string> as a type and don't have the collapsed items[x]. Also for complex ones I think we should not have the separate items[x], since imho it makes it more confusing and looks like the path would be streams.items[0] to access something. Could we just render them as array<object> and have the child properties directly nested when expanding the array?

[flash1293]

Thanks @timroes !

Your suggestions for arrays make a lot of sense to me - I will implement that.

[flash1293]

@timroes Done - it's now showing the properties of the array items directly. I did a similar thing for constant values - they are rendered as the "type" now:

[flash1293]

If you are OK with the basic component I will remove the actual usages so we can get the pre-requisite in.