Support Custom Export with Jinja

[chanyou0311]

Background and Motivation

The current export formats support a wide variety of data types, making it very convenient to get started. However, incorporating custom logic into the output requires defining a Custom Exporter, which involves implementing and maintaining Python scripts—a significant burden in practice.

For instance, at my company, the product team manages hundreds of tables in an RDBMS, and the schemas change almost daily. After loading these tables into BigQuery, our data platform team sets up a staging layer in dbt.

Because schema changes occur so frequently across hundreds of tables, manually updating the staging SQL files for dbt is quite labor-intensive. We therefore use the Data Contract CLI to automate these updates.

Although Data Contract CLI offers a dbt-staging-sql output format, it doesn’t easily accommodate our needed pre-processing—such as data type–specific conversions or parsing date/time values. To address this, we introduced a Jinja template file and used a Custom Exporter to generate our own custom format.

In scenarios like this, having official support for Jinja templates within the dbt export command would greatly simplify producing outputs that include custom logic.

Proposed Feature

I propose introducing a custom-template format, as shown below:

datacontract export datacontract.yaml --format custom-template --template template.sql --output output.sql

With this command, a Jinja template file specified via the --template option would be used for the export. I'm not particularly attached to the names custom-template or --template; if there are more suitable names, I'm happy to adopt them.

Expected Benefits

By enabling this functionality, injecting unique logic into the output becomes much easier.

Beyond generating dbt resources as described above, you could, for example, produce Markdown files to help maintain a data catalog or other documentation, complete with custom headings or formatting. I believe this approach would be broadly useful for various scenarios.

Potential Concerns and Open Questions

Since this proposal differs from existing formats, I'd like to discuss any considerations or potential issues before it’s fully implemented.

If the proposal is accepted, I’d be happy to create and submit a Pull Request myself.

Thank you for your time, and I look forward to your feedback.

[jochenchrist]

I like the idea 👍

[jochenchrist]

I would go with

datacontract export --format custom --template template.sql --output output.sql

[jochenchrist]

Would highly appreate a PR

[chanyou0311]

Thank you for your response.
I’ll try creating a Pull Request with the format name set to custom 👍

[simonharrer]

Key here is a good documentation so people know how to use it (document the fields that we make available to the jinja template, have a minimal example template on how to use it with a for look over some fields). And how we offer a stable API for the custom templates with the availability of the Open Data Contract Standard in addition to the Data Contract Specification.

[chanyou0311]

@simonharrer
I also agree that documentation is important. I plan to add a brief usage section to the README, and I’d appreciate your feedback once I open a PR.

I’m not very familiar with the Open Data Contract Standard, and I wasn’t entirely sure what you meant about making ODCS available.
My current understanding is that the export command is designed to read YAML files following the Data Contract Specification, so it doesn’t directly support ODCS. If you wanted to export from ODCS in a custom format, I believe you’d first convert ODCS into the Data Contract Specification and then export to the custom format.

Could you please provide more details on your implementation approach for ODCS support? Any clarification would be very helpful.

[PadenZach]

This sounds promising. Would this allow for rendering markdown found in descriptions? I often find myself wanting to embed additional links, images, tables, etc. into our contracts.

[simonharrer]

Regarding markdown: I‘d welcome contributions to add markdown support for the standard html export.

Regarding ODCS: this require more work but we want to provide an export for the ODCS format as well. In that case, the conversion from ODCS to DCS may happen internally but ODCD yaml strcuture is kept so it can be passed to the jinja template. But this can happen in another PR.

[chanyou0311]

I opened PR #616. I'd like you to review it.

@PadenZach
Yes, you can export the markdown as any format.
data_contract variable is available in jinja template as below example.

$ cat template.txt
title: {{ data_contract.info.title }}

$ datacontract export --format custom --template template.txt datacontract.yaml
title: Orders Latest

I added dbt model template example in README.
https://github.com/datacontract/datacontract-cli/pull/616/files?short_path=b335630#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5