EFK with transform for tabular data

[ryandawsonuk]

Alternative to #610

In this version the engine can do a transform to put tabular data into key-value pairs for searching. We can then do e.g. value-range searches:

It checks whether there's a 'data.names' array before transforming but need to do more checks to make sure this won't break other types of requests (e.g. what can we assume about the data value type and array shape). Also need to consider performance.

[ryandawsonuk]

For images we may be better to store the image on disk or in a bucket and keep a searchable reference to it. I'd like to investigate computing metadata about the image before storing it so that you could search by metadata (e.g. presence of certain colours) - see https://stackoverflow.com/questions/30440224/possible-to-store-images-in-elasticsearch

We should discuss whether to do this in the engine or in a wrapper library or even start with just an example of how it can be done in the end-user code

[ryandawsonuk]

Next steps:

• use thread to make async
• consider more examples - e.g. config option for column-primary, data.names as comma-sep in env var, size cap on transform/enrichment, percentage sampling
• revisit whether to split out batches or provide option to
• look at binary, tensor and str cases
• Look at how parameterised - what env vars and what defaults and how to apply (think example helm charts)
• If single engine for graph is that ok that we won’t know which component? Will we know anyway? Do we need to? What about MABs?
• run on Cloud
• look at custom metadata
• document how it works (mention [filter_kubernetes] enhancement: provide mechanism to exclude containers from fluent bit via annotations fluent/fluent-bit#737 )
• consider KF-serving where engine not present - could it use same logic in python as part of its base image/s? Or should we publish a py lib?

[ryandawsonuk]

New idea - look at performing the enrichment in a separate batch job that updates the entries in elasticsearch or an elasticsearch transform script - preferably we'd do something python-based to simplify working with the arrays. In that case the transformation/enrichment part in the engine for this PR would be taken out.

[ryandawsonuk]

Related to SeldonIO/seldon-operator#13 but not a dependency

[ryandawsonuk]

Depends on SeldonIO/seldon-operator#17