Issue #728: Adding asterisk for metadata transfer

[michaeldinzinger]

Hello all

Goal:
When having a lot of metadata fields that all start with the same prefix, e.g. parse.[...], one can simply write

metadata.persist:
- parse.*

instead of listing them all.

Besides that, making mdToTransfer, mdToPersistOnly, trackPath and trackDepth protected instead of private. This makes it easier to create a Custom Metadata Transfer class.

[jnioche]

Thanks! this would be very helpful. See comments on possible improvements

[jnioche]

That's fab, thanks @michaeldinzinger!

I think the change would affect both the metadata to transfer and the metadata to persist, in which case ideally we would like to have a test demonstrating the impact on the latter. We'd also need to make it explicit in the default config as well as the configs generated by the archetypes.

[michaeldinzinger]

Thank you again for your quick reply.

I think the change would affect both the metadata to transfer and the metadata to persist, in which case ideally we would like to have a test demonstrating the impact on the latter.

How could such a demonstration test look like? For now, I extended the method testTransfer to cover the metadata.transfer scenario, whereas the method testFilter tests the metadata.persist scenario.

We'd also need to make it explicit in the default config as well as the configs generated by the archetypes.

I added a comment line in each of the crawler-default.yaml files.

[jnioche]

Perfect! Thanks a lot @michaeldinzinger, this is a great addition to StormCrawler