You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It's likely that there might be a use-case in the future which requires multiple chainable transformers.
The configuration parameter component.transformer could take a comma-separated list instead of just one value. The order of the list would specify the order of execution of the transformers. That wouldn't support the same transformer to be used multiple times.
Why the prefixes component.transformer.class and transformer? This prevents name conflicts
At runtime, transformers would only receive their specific config subset, i.e. in the above example, my-transformer gets property.a => "value a", property.b => "value b" in the transform method instead of the full configuration like now. If cross-component configuration is necessary, the HyperdriveContext may be utilized.
The order of the transformers is determined by the number after component.transformer.id. An error is thrown if it's not an integer. The number may be negative. Order numbers need not be consecutive, i.e. no error is thrown if one transformer has component.transformer.id.2 and the other component.transformer.id.-1
Tasks
StreamTransformerAbstractFactory.build should return a list of transformers.
build should call the apply method of the companion object only with a configuration subset using the id (e.g. csst in the above example)
SparkIngestor.ingest should accept a list of transformers and loop through them (fold)
Update tests
Note
As a by-product, transformers will be optional. If no transformer is specified, the list of transformers will be empty, thus the dataframe will directly be passed from the decoder (reader from v4.0.0) to the writer.
How to migrate ColumnSelectorStreamTransformer
ConfigurationsKeys.ColumnSelectorStreamTransformerKeys.KEY_COLUMNS_TO_SELECT should be "columns.to.select"
Currently, only one transformer can be specified.
It's likely that there might be a use-case in the future which requires multiple chainable transformers.
The configuration parameterThat wouldn't support the same transformer to be used multiple times.component.transformer
could take a comma-separated list instead of just one value. The order of the list would specify the order of execution of the transformers.Components should be configured like this:
Why the prefixes
component.transformer.class
andtransformer
? This prevents name conflictsAt runtime, transformers would only receive their specific config subset, i.e. in the above example,
my-transformer
getsproperty.a => "value a", property.b => "value b"
in thetransform
method instead of the full configuration like now. If cross-component configuration is necessary, theHyperdriveContext
may be utilized.The order of the transformers is determined by the number after
component.transformer.id.
An error is thrown if it's not an integer. The number may be negative. Order numbers need not be consecutive, i.e. no error is thrown if one transformer hascomponent.transformer.id.2
and the othercomponent.transformer.id.-1
Tasks
StreamTransformerAbstractFactory.build
should return a list of transformers.build
should call theapply
method of the companion object only with a configuration subset using the id (e.g.csst
in the above example)SparkIngestor.ingest
should accept a list of transformers and loop through them (fold
)Note
How to migrate
ColumnSelectorStreamTransformer
ConfigurationsKeys.ColumnSelectorStreamTransformerKeys.KEY_COLUMNS_TO_SELECT
should be "columns.to.select"with
with
Alternatively, the whole column selector transformation config can be removed if all of the jobs only use select all.
HyperConformance
za.co.absa.enceladus.conformance.HyperConformanceAttributes
, search and replaces"$keysPrefix.
with"
Replace
with
Transformer specific configuration already happens to be correct
The text was updated successfully, but these errors were encountered: