CDK: Generate HTTP stream catalog from OpenAPI spec

[sherifnada]

Tell us about the problem you're trying to solve

One of the biggest pain points when writing HTTP connectors is creating JSON Schema from OpenAPI specs. We should provide a very easy way to go from OpenAPI spec to JsonSchema, either completely programmatically or via an external tool.

┆Issue is synchronized with this Asana task by Unito

[james-blitzm]

Hello. I’m in the process of selecting an ELT solution and this feature is a significant deciding factor.

[davinchia]

Hi @james-blitzm this is probably slated for next quarter. Can you say more about why this is a significant deciding factor?

[james-blitzm]

Hi Davin,

Thanks for the quick response. In my experience, most of the effort associated with implementing/maintaining an ELT solution is in maintaining the integrations with the data sources (which are often Rest APIs in JSON format). I feel this effort would be greatly reduced if it were possible to broadly standardise these integrations. OpenAPI is one such standard which could be used to automate significant parts of that integration.

In my use case, many of the data sources I'm interested in have first-party OpenAPI specifications. For those APIs which don't have official OpenAPI specifications, it is usually trivial to find unofficial specs from third-parties on Github, etc. Having these specifications enables automation in model generation, parameterisation and code generation, which greatly reduces development and maintenance effort.