HyperConformance report version from payload

[kevinwallimann]

Background

Currently, in the HyperConformance plugin, the enceladus_info_version can only be set through report.version with a static value. It is a desired feature from source teams to set enceladus_info_version dynamically based on a field from the payload, similarly how event.timestamp.column can be used to set the value for enceladus_info_date.

Feature

Add an attribute (e.g. report.version.column) that accepts the name of an Int-column. That Int-column should be used as the value for enceladus_info_version.

Example

Given a dataframe

key	value	custom_version
100	abc	1
101	xyz	1
100	ABC	2
101	XYZ	2

With HyperConformance configuration

• dataset.name=Some_Dataset
• dataset.version=1
• ...
• report.version.column=custom_version

The conformed dataframe should look like this

key	value	custom_version	enceladus_info_date	enceladus_info_date_string	enceladus_info_version
100	abc	1	...	...	1
101	xyz	1	...	...	1
100	ABC	2	...	...	2
101	XYZ	2	...	...	2

(Dates are omitted)

[AdrianOlosutean]

Release notes
New way of having HyperConformance info_version_column from an existing dataframe column by specifying report.version.column