Flux commands for analyzing input data

[fsteeg]

See https://gitlab.com/oersi/oersi-etl/-/issues/238

[fsteeg]

The first module is ready for functional review, assigning @TobiasNx. Deployed to playground for testing: your sample from #294, using the new fix-list-paths command.

From the documentation annotation:

Lists all paths found in a record. These paths can be used in a Fix to address fields. Options: count (output occurence frequency of each path, sorted by highest frequency first; default: true), index (output individual repeated subfields and array elements with index numbers instead of '*'; default: false)

(If we count it's always sorted by highest frequency, because if we count we'll automatically sort internally, which would make it hard to have a consistent implementation with separate control over the sorting. I think this way it's both conceptually simple and hopefully covers your use cases.)

[fsteeg]

And the second module, for listing values, is ready for review too. Deployed to playground: the example from above, but calling fix-list-values with one of the resulting paths.

From the documentation annotation:

Lists all values found for the given path. The paths can be found using fix-list-paths. Options: count (output occurence frequency of each value, sorted by highest frequency first; default: true)

(Sorting behavior is as described for fix-list-paths above.)

[TobiasNx]

Implementation looks good.
Only the option count= should behave differently. Only the template should change and count-triples should not be dropped. So that you have the list of values/paths but not the counted occurence. The path/values would be uniq and no duplicate should be given:

something like this:

// This part can be adjusted to your own data source. The example here is marc21
"https://raw.githubusercontent.com/metafacture/metafacture-core/master/metafacture-runner/src/main/dist/examples/read/marc21/10.marc21"
| open-http
| as-lines
| decode-marc21

// This part can be reused for all incoming data streams to list all fix paths.
| fix("nothing()",repeatedFieldsToEntities="true", entityMemberName="*")
| flatten
| stream-to-triples
| count-triples(countBy="predicate")
| template("${s}")
| print;

[TobiasNx]

Also perhaps the template should be | template("${o}\t|\t${s}") ( when count=false then | template("${s}"))

[fsteeg]

Only the template should change and count-triples should not be dropped

Deployed to playground for testing:

• fix-list-paths(count="false")
• fix-list-values("68900.0.*", count="false")

Also perhaps the template should be | template("${o}\t|\t${s}")

I've changed both templates to use \t|\t instead of \t :

• fix-list-paths
• fix-list-values("68900.0.*")

(It seems I currently have to open these playground links in a new window, otherwise it's empty.)

[TobiasNx]

Great paths are unique now. +1
Concerning the template I am not sure if I prefer [occurece] | [path] more. This may be a simple preference. So I am okay with that. +1

Concerning the sorting I thought about it again.

| stream-to-triples
| count-triples(countBy="predicate")
| template("${s}")
| print;

This does not sort by occurence automatically. How does your function deviate from this? The alphabetical sorting helps to get an understanding of the source data and its schema. Therefore I would prefer if we also would support alphabetical sorting too.

[fsteeg]

Concerning the template I am not sure if I prefer [occurece] | [path] more.

I'm only trying to implement your requirement, I don't have my own preference here. Could you give me an example of the desired output for both commands?

[blackwinter]

Just a side note: Wouldn't it make sense to make the template configurable?

[TobiasNx]

Concerning the template I am not sure if I prefer [occurece] | [path] more.

I'm only trying to implement your requirement, I don't have my own preference here. Could you give me an example of the desired output for both commands?

[path] | [occurrence]
6890 .5.* | 18

vs
[occurrence] | [path]
18 | 6890 .5.*

[TobiasNx]

Concerning the template I am not sure if I prefer [occurece] | [path] more.

I'm only trying to implement your requirement, I don't have my own preference here. Could you give me an example of the desired output for both commands?

[path] | [occurrence] 6890 .5.* | 18

vs [occurrence] | [path] 18 | 6890 .5.*

but keep it the way you implement it

[fsteeg]

The alphabetical sorting helps to get an understanding of the source data and its schema.

I've changed it to sort alphabetically if count is false:

fix-list-paths(count="false")

Concerning the template I am not sure if I prefer [occurece] | [path] more.

I've changed it to use ${o}\t|\t${s} as a common default template for both commands:

fix-list-paths
fix-list-values("68900.0.*")

With an option to set it (thanks for the input @blackwinter):

fix-list-paths(template="${s}\t${o}")

[TobiasNx]

Thanks, thats all great!

[blackwinter]

Left some (more or less) minor remarks.

[fsteeg]

Thank you for the review! Addressed your feedback in 172a701.