Extractor attributes and "dumb" extractors

[andrus]

I recently wrote a non-open source extractor (and a matching connector) that generates a stream of data by listing S3 bucket contents. S3 object keys are parsed by a stage listener into a set of columns that match the target DB object properties (this required a new @AfterSourceRowsExtracted annotation per #192).

However this change confuses RowConverter in its attempt to "normalize" the columns. If no explicit attributes are defined in the extractor, it tries to match S3 object attributes to the target entity (this is a meaningless operation, and for coincidentally overlapping names it tries to do type conversions!). If transformed attributes are defined in the extractor XML, they are incorrectly applied to the source S3 columns.

I think the problem here is in the implicit assumption in the RowConverter is that the Extractor is flexible enough to produce attributes that are close enough to the target structure. This works for JDBC extractor, that allows SQL scripting in the XML, but does not work for the S3 extractor.

To solve this we may possibly need an extra processing stage, and / or the ability to redefine RowAttribute[] dynamically from stage to stage.