From e6c2bd559960e3eb46a7ff698632228082bed5a6 Mon Sep 17 00:00:00 2001 From: Phu Tu Date: Wed, 26 Feb 2025 15:43:31 +0100 Subject: [PATCH 01/10] Fix some typos, links, numberings of lessons --- 01_Getting_Metafacture.md | 11 +++++------ 02_Introduction_into_Metafacture-Flux.md | 6 +++--- 05-More-Fix-Concepts.md | 2 +- 06_MetafactureCLI.md | 2 +- README.md | 2 +- 5 files changed, 11 insertions(+), 12 deletions(-) diff --git a/01_Getting_Metafacture.md b/01_Getting_Metafacture.md index 526adbc..a65dd5e 100644 --- a/01_Getting_Metafacture.md +++ b/01_Getting_Metafacture.md @@ -8,17 +8,16 @@ It was initially developed by DNB starting in 2011 and is maintained since 2019 Metafacture can be used as a stand-alone application or as a Java library in other applications. The name Metafacture is a portmanteau of the words metadata and manufacture. -In this tutorial we are going to teach how to use Metafacture to peform simple and advanced data processing tasks. +In this tutorial we are going to teach how to use Metafacture to perform simple and advanced data processing tasks. At the beginning we will use the web application [Metafacture Playground](https://metafacture.org/playground/). So no installation is needed. The Playground is a web interface that helps you getting started. -It is useful to test, share and export metafacture workflows. +It is useful to test, share and export Metafacture workflows. -Starting with [Chapter 6](https://github.com/metafacture/metafacture-tutorial/blob/main/06_MetafactureCLI.md) -we can switch from using Playground to running Metafacture on our own Hardware. -But the examples are still provided in the playground. +Starting with [Chapter 6](./06_MetafactureCLI.md) we can switch from using Playground to running Metafacture on our own Hardware. +But the examples are still provided in the Playground. -To run Metafacture on your local maschine you need you need a Linux/Unix Bash Shell (part of every Linux, MacOS and Windows >=10) with Metafacture Core installed. In this course we are not teaching you how to use the command line. For that see: +To run Metafacture on your local maschine you need a Linux/Unix Bash Shell (part of every Linux, MacOS and Windows >=10) with Metafacture Core installed. In this course we are not teaching you how to use the command line. For that see: TODO **Next lesson**: [02 Introduction into Metafacture Flux](./02_Introduction_into_Metafacture-Flux.md) diff --git a/02_Introduction_into_Metafacture-Flux.md b/02_Introduction_into_Metafacture-Flux.md index bfc9a4e..d39f37c 100644 --- a/02_Introduction_into_Metafacture-Flux.md +++ b/02_Introduction_into_Metafacture-Flux.md @@ -1,6 +1,6 @@ # Lesson 2: Introduction into Metafacture Flux -To perform data processing with Metafacture transformation workflows are configured with **Metafacture Flux**, a domain-specific scripting language(DSL). +To perform data processing with Metafacture transformation workflows are configured with **Metafacture Flux**, a domain-specific scripting language (DSL). With Metafacture Flux we combine different modules for reading, opening, transforming, and writing data sets. In this lesson we will learn about Metafacture Flux, what Flux workflows are and how to combine different Flux modules to create a workflow in order to process datasets. @@ -9,9 +9,9 @@ In this lesson we will learn about Metafacture Flux, what Flux workflows are and To process data Metafacture can be used with the command line, as JAVA library or you can use the Metafacture Playground. -For this introduction we will start with the Playground since it allows a quick start without additional installing. The [Metafacture Playground](https://metafacture.org/playground) is a webinterface to test and share Metafacture. The commandline handling will be subject in lesson 6. TODO: Add link. +For this introduction we will start with the Playground since it allows a quick start without additional installing. The [Metafacture Playground](https://metafacture.org/playground) is a web interface to test and share Metafacture workflows. The commandline handling will be subject in [lesson 6](./06_MetafactureCLI.md) -In this tutorial we are going to process *structured information*. We call data structured when it organised in such a way is that it easy processable by computers. Literary text documents like War and Peace are structured only in words and sentences, but a computer doesn’t know which words are part of the title or which words contain names. We had to tell the computer that. Today we will download a weather report in a structured format called JSON and inspect it with Metafacture. +In this tutorial we are going to process *structured information*. We call data structured when it organised in such a way that is easy processable by computers. Literary text documents like War and Peace are structured only in words and sentences, but a computer doesn’t know which words are part of the title or which words contain names. We had to tell the computer that. Today we will download a weather report in a structured format called JSON and inspect it with Metafacture. ## Flux Workflows diff --git a/05-More-Fix-Concepts.md b/05-More-Fix-Concepts.md index a430953..6c18837 100644 --- a/05-More-Fix-Concepts.md +++ b/05-More-Fix-Concepts.md @@ -1,4 +1,4 @@ -# Lesson 6: More Fix concepts +# Lesson 5: More Fix concepts We already learned about simple Fixes (or Fix Functions) but there are three additional concepts in Fix selector, conditionals and binds. These Fix concepts were introduced by Catmandu (see [functions](https://librecat.org/Catmandu/#functions), [selector](https://librecat.org/Catmandu/#selectors), [conditionals](https://librecat.org/Catmandu/#conditionals) and [binds](https://librecat.org/Catmandu/#binds)). But be aware Metafacture Fix does not support all of the specific functions, selectors, conditionals and binds from Catmandu. Check the documentation for an full overview of the supported [fix functions](https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#functions). diff --git a/06_MetafactureCLI.md b/06_MetafactureCLI.md index 61aab44..3a611f8 100644 --- a/06_MetafactureCLI.md +++ b/06_MetafactureCLI.md @@ -1,4 +1,4 @@ -# Lesson 7: Using Metafacture as Command Line Tool +# Lesson 6: Using Metafacture as Command Line Tool ## Get Metafacture Runner as CLI Tool diff --git a/README.md b/README.md index 6b3ddec..ee6f8ba 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ The Catmandu tutorial was created by [@phochste](https://github.com/phochste). I Since [Metafacture Fix](https://github.com/metafacture/metafacture-fix) is introducing a catmandu-fix like transformation language to Metafacture the addaption of the Catmandu Tutorial for Metafacture purposes seem to me obvioous. -It should help you to get accostumed with Metafacture Core and Metafacture Fix. +It should help you to get accustomed with Metafacture Core and Metafacture Fix. Have fun. Lesson plan: From 9c36ebcd98550b6dbbf661649967f387c2a8bd2c Mon Sep 17 00:00:00 2001 From: Phu Tu Date: Mon, 10 Mar 2025 15:48:18 +0100 Subject: [PATCH 02/10] Fix more typos, add some rewording --- 01_Getting_Metafacture.md | 4 +- 02_Introduction_into_Metafacture-Flux.md | 67 ++++++++++++------------ README.md | 2 +- 3 files changed, 37 insertions(+), 36 deletions(-) diff --git a/01_Getting_Metafacture.md b/01_Getting_Metafacture.md index a65dd5e..d317809 100644 --- a/01_Getting_Metafacture.md +++ b/01_Getting_Metafacture.md @@ -14,10 +14,10 @@ At the beginning we will use the web application [Metafacture Playground](https: installation is needed. The Playground is a web interface that helps you getting started. It is useful to test, share and export Metafacture workflows. -Starting with [Chapter 6](./06_MetafactureCLI.md) we can switch from using Playground to running Metafacture on our own Hardware. +Starting with [Chapter 6](./06_MetafactureCLI.md) we can switch from using Playground to running Metafacture on our own hardware. But the examples are still provided in the Playground. -To run Metafacture on your local maschine you need a Linux/Unix Bash Shell (part of every Linux, MacOS and Windows >=10) with Metafacture Core installed. In this course we are not teaching you how to use the command line. For that see: TODO +To run Metafacture on your local machine you need a Linux/Unix Bash Shell (part of every Linux, MacOS and Windows >=10) with Metafacture Core installed. In this course we are not teaching you how to use the command line. For that see: **TODO** **Next lesson**: [02 Introduction into Metafacture Flux](./02_Introduction_into_Metafacture-Flux.md) diff --git a/02_Introduction_into_Metafacture-Flux.md b/02_Introduction_into_Metafacture-Flux.md index d39f37c..bc68079 100644 --- a/02_Introduction_into_Metafacture-Flux.md +++ b/02_Introduction_into_Metafacture-Flux.md @@ -11,7 +11,7 @@ To process data Metafacture can be used with the command line, as JAVA library o For this introduction we will start with the Playground since it allows a quick start without additional installing. The [Metafacture Playground](https://metafacture.org/playground) is a web interface to test and share Metafacture workflows. The commandline handling will be subject in [lesson 6](./06_MetafactureCLI.md) -In this tutorial we are going to process *structured information*. We call data structured when it organised in such a way that is easy processable by computers. Literary text documents like War and Peace are structured only in words and sentences, but a computer doesn’t know which words are part of the title or which words contain names. We had to tell the computer that. Today we will download a weather report in a structured format called JSON and inspect it with Metafacture. +In this tutorial we are going to process *structured information*. We call data structured when it organised in such a way that is easy processable by computers. Literary text documents like *War and Peace* are structured only in words and sentences, but a computer doesn’t know which words are part of the title or which words contain names. We had to tell the computer that. Today we will download a weather report in a structured format called JSON and inspect it with Metafacture. ## Flux Workflows @@ -48,7 +48,7 @@ INPUT ; ``` -Copy this into the I window of your playground or just adjust your example. +Copy this into the window of your playground or just adjust your example. `INPUT` as a varibale is defined in the first line of the flux. And instead of the text string, the Flux-Workflow starts just with the variable `INPUT` without `"`. @@ -56,9 +56,7 @@ But the result is the same if you process the flux. Often you want to process data stored in a file. -The playground has a input area called `ìnputFile`-Content that pretends to be a local file. It can be adressed with the variable `inputFile`. In this `inputFile`-text area you can insert data samples.In the Playground you can put the variable `inputFile` at the beginning of a MF workflow to process the content of this imaginary file. The variable `inputFile` can be used at the beginning -of the workflow and it refers to the input that is written in the Data Window at the top of the playground. - +The playground has an input area called `ìnputFile-content`. In this text area you can insert data that you have usually stored in a file. The variable `inputFile` can be used at the beginning of the workflow and it refers to the input file. e.g. @@ -68,6 +66,7 @@ e.g. So lets use `inputFile` instead of `INPUT` and copy the value of the text string in the Data field above the Flux. Data: + `Hello, friend. I'am Metafacture!` Flux: @@ -78,11 +77,11 @@ inputFile ; ``` -Höm... There seems to be unusual output. Its a file path. Why? -Because hidden behind the variable `inputFile` is a path to a file. -But to read the content of the file we need to handle the incoming file path differently. +Oops... There seems to be unusual output. Its a file path. Why? +Because the variable `inputFile` refers to a file (path). +To read the content of the file we need to handle the incoming file path differently. -(How to open real files you will learn when we learn how to run metafacture on your command line in lesson 06.) +(You will learn how to process files on your computer in lesson 06 when we show how to run metafacture on the command line.) We need to add two additional Metafacture commands: `open-file` and `as-lines` @@ -96,7 +95,7 @@ inputFile ; ``` -The inputFile is opened as a file(`open-file`) and then processed line by line (`as-line`). +The inputFile is opened as a file (`open-file`) and then processed line by line (`as-line`). You can see that in this [sample](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-lines%0A%7Cprint%0A%3B&data=Hello%2C+friend.+I%27am+Metafacture%21). We usually do not start with any random text strings but with data. So lets play around with some data. @@ -113,16 +112,15 @@ This is data in JSON format. But it seems not very readable. But all these fields tell something about a publication, a book, with 268 pages and title Ordinary Vices by Judith N. Shklar. -Let's copy the text from the input into our data field. [And run it again](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-lines%0A%7Cprint%0A%3B&data=%7B%22publishers%22%3A+%5B%22Belknap+Press+of+Harvard+University+Press%22%5D%2C+%22identifiers%22%3A+%7B%22librarything%22%3A+%5B%22321843%22%5D%2C+%22goodreads%22%3A+%5B%222439014%22%5D%7D%2C+%22covers%22%3A+%5B413726%5D%2C+%22local_id%22%3A+%5B%22urn%3Atrent%3A0116301499939%22%2C+%22urn%3Asfpl%3A31223009984353%22%2C+%22urn%3Asfpl%3A31223011345064%22%2C+%22urn%3Acst%3A10017055762%22%5D%2C+%22lc_classifications%22%3A+%5B%22JA79+.S44+1984%22%2C+%22HM216+.S44%22%2C+%22JA79.S44+1984%22%5D%2C+%22key%22%3A+%22/books/OL2838758M%22%2C+%22authors%22%3A+%5B%7B%22key%22%3A+%22/authors/OL381196A%22%7D%5D%2C+%22ocaid%22%3A+%22ordinaryvices0000shkl%22%2C+%22publish_places%22%3A+%5B%22Cambridge%2C+Mass%22%5D%2C+%22subjects%22%3A+%5B%22Political+ethics.%22%2C+%22Liberalism.%22%2C+%22Vices.%22%5D%2C+%22pagination%22%3A+%22268+p.+%3B%22%2C+%22source_records%22%3A+%5B%22marc%3AOpenLibraries-Trent-MARCs/tier5.mrc%3A4020092%3A744%22%2C+%22marc%3Amarc_openlibraries_sanfranciscopubliclibrary/sfpl_chq_2018_12_24_run01.mrc%3A195791766%3A1651%22%2C+%22ia%3Aordinaryvices0000shkl%22%2C+%22marc%3Amarc_claremont_school_theology/CSTMARC1_barcode.mrc%3A137174387%3A3955%22%2C+%22bwb%3A9780674641754%22%2C+%22marc%3Amarc_loc_2016/BooksAll.2016.part15.utf8%3A115755952%3A680%22%2C+%22marc%3Amarc_claremont_school_theology/CSTMARC1_multibarcode.mrc%3A137367696%3A3955%22%2C+%22ia%3Aordinaryvices0000shkl_a5g0%22%2C+%22marc%3Amarc_columbia/Columbia-extract-20221130-001.mrc%3A328870555%3A1311%22%2C+%22marc%3Aharvard_bibliographic_metadata/ab.bib.01.20150123.full.mrc%3A156768969%3A815%22%5D%2C+%22title%22%3A+%22Ordinary+vices%22%2C+%22dewey_decimal_class%22%3A+%5B%22172%22%5D%2C+%22notes%22%3A+%7B%22type%22%3A+%22/type/text%22%2C+%22value%22%3A+%22Bibliography%3A+p.+251-260.\nIncludes+index.%22%7D%2C+%22number_of_pages%22%3A+268%2C+%22languages%22%3A+%5B%7B%22key%22%3A+%22/languages/eng%22%7D%5D%2C+%22lccn%22%3A+%5B%2284000531%22%5D%2C+%22isbn_10%22%3A+%5B%220674641752%22%5D%2C+%22publish_date%22%3A+%221984%22%2C+%22publish_country%22%3A+%22mau%22%2C+%22by_statement%22%3A+%22Judith+N.+Shklar.%22%2C+%22works%22%3A+%5B%7B%22key%22%3A+%22/works/OL2617047W%22%7D%5D%2C+%22type%22%3A+%7B%22key%22%3A+%22/type/edition%22%7D%2C+%22oclc_numbers%22%3A+%5B%2210348450%22%5D%2C+%22latest_revision%22%3A+16%2C+%22revision%22%3A+16%2C+%22created%22%3A+%7B%22type%22%3A+%22/type/datetime%22%2C+%22value%22%3A+%222008-04-01T03%3A28%3A50.625462%22%7D%2C+%22last_modified%22%3A+%7B%22type%22%3A+%22/type/datetime%22%2C+%22value%22%3A+%222024-12-27T16%3A46%3A50.181109%22%7D%7D). +Let's copy the JSON data into our `ìnputFile-content` field. [And run it again](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-lines%0A%7Cprint%0A%3B&data=%7B%22publishers%22%3A+%5B%22Belknap+Press+of+Harvard+University+Press%22%5D%2C+%22identifiers%22%3A+%7B%22librarything%22%3A+%5B%22321843%22%5D%2C+%22goodreads%22%3A+%5B%222439014%22%5D%7D%2C+%22covers%22%3A+%5B413726%5D%2C+%22local_id%22%3A+%5B%22urn%3Atrent%3A0116301499939%22%2C+%22urn%3Asfpl%3A31223009984353%22%2C+%22urn%3Asfpl%3A31223011345064%22%2C+%22urn%3Acst%3A10017055762%22%5D%2C+%22lc_classifications%22%3A+%5B%22JA79+.S44+1984%22%2C+%22HM216+.S44%22%2C+%22JA79.S44+1984%22%5D%2C+%22key%22%3A+%22/books/OL2838758M%22%2C+%22authors%22%3A+%5B%7B%22key%22%3A+%22/authors/OL381196A%22%7D%5D%2C+%22ocaid%22%3A+%22ordinaryvices0000shkl%22%2C+%22publish_places%22%3A+%5B%22Cambridge%2C+Mass%22%5D%2C+%22subjects%22%3A+%5B%22Political+ethics.%22%2C+%22Liberalism.%22%2C+%22Vices.%22%5D%2C+%22pagination%22%3A+%22268+p.+%3B%22%2C+%22source_records%22%3A+%5B%22marc%3AOpenLibraries-Trent-MARCs/tier5.mrc%3A4020092%3A744%22%2C+%22marc%3Amarc_openlibraries_sanfranciscopubliclibrary/sfpl_chq_2018_12_24_run01.mrc%3A195791766%3A1651%22%2C+%22ia%3Aordinaryvices0000shkl%22%2C+%22marc%3Amarc_claremont_school_theology/CSTMARC1_barcode.mrc%3A137174387%3A3955%22%2C+%22bwb%3A9780674641754%22%2C+%22marc%3Amarc_loc_2016/BooksAll.2016.part15.utf8%3A115755952%3A680%22%2C+%22marc%3Amarc_claremont_school_theology/CSTMARC1_multibarcode.mrc%3A137367696%3A3955%22%2C+%22ia%3Aordinaryvices0000shkl_a5g0%22%2C+%22marc%3Amarc_columbia/Columbia-extract-20221130-001.mrc%3A328870555%3A1311%22%2C+%22marc%3Aharvard_bibliographic_metadata/ab.bib.01.20150123.full.mrc%3A156768969%3A815%22%5D%2C+%22title%22%3A+%22Ordinary+vices%22%2C+%22dewey_decimal_class%22%3A+%5B%22172%22%5D%2C+%22notes%22%3A+%7B%22type%22%3A+%22/type/text%22%2C+%22value%22%3A+%22Bibliography%3A+p.+251-260.\nIncludes+index.%22%7D%2C+%22number_of_pages%22%3A+268%2C+%22languages%22%3A+%5B%7B%22key%22%3A+%22/languages/eng%22%7D%5D%2C+%22lccn%22%3A+%5B%2284000531%22%5D%2C+%22isbn_10%22%3A+%5B%220674641752%22%5D%2C+%22publish_date%22%3A+%221984%22%2C+%22publish_country%22%3A+%22mau%22%2C+%22by_statement%22%3A+%22Judith+N.+Shklar.%22%2C+%22works%22%3A+%5B%7B%22key%22%3A+%22/works/OL2617047W%22%7D%5D%2C+%22type%22%3A+%7B%22key%22%3A+%22/type/edition%22%7D%2C+%22oclc_numbers%22%3A+%5B%2210348450%22%5D%2C+%22latest_revision%22%3A+16%2C+%22revision%22%3A+16%2C+%22created%22%3A+%7B%22type%22%3A+%22/type/datetime%22%2C+%22value%22%3A+%222008-04-01T03%3A28%3A50.625462%22%7D%2C+%22last_modified%22%3A+%7B%22type%22%3A+%22/type/datetime%22%2C+%22value%22%3A+%222024-12-27T16%3A46%3A50.181109%22%7D%7D). The output in result is the same as the input and it is still not very readable. -So let's transform some stuff. Let us use some other serialization. How about YAML. -With the metafacture you can process this file to make it a bit easier readable by using a small workflow script. Lets turn the one line of JSON data into YAML. YAML is another format for structured information which is a bit easier to read for human eyes. In order to change the serialization of the data we need to decode the data and then encode the data. -Metafacture has lots of decoder- and encoder-modules for specific data formats that can be used in an Flux-Workflow. +Metafacture has lots of decoder and encoder modules for specific data formats that can be used in an Flux workflow. + Let's try this out. Add the module `decode-json` and `encode-yaml` to your Flux Workflow. The Flux should now look like this: @@ -215,13 +213,13 @@ last_modified: value: "2024-12-27T16:46:50.181109" ``` -This is better readble right? +This is better readable, right? -But we cannot only open the data we have in our data field we also can open stuff on the web: +But we cannot only open the data we have in our `ìnputFile-content` field, we also can open stuff on the web: -Instead of using `inputFile` lets read the live weather data which is provided by the URL from above: +Instead of using `inputFile` lets read the book data which is provided by the URL from above: -Clear your playground and copy the following Flux-Workflow: +Clear your playground and copy the following Flux workflow: ``` "https://openlibrary.org/books/OL2838758M.json" @@ -233,18 +231,16 @@ Clear your playground and copy the following Flux-Workflow: ; ``` -The [result in the playground](https://metafacture.org/playground/?flux=%22https%3A//openlibrary.org/books/OL2838758M.json%22%0A%7C+open-http%0A%7C+as-lines%0A%7C+decode-json%0A%7C+encode-yaml%0A%7C+print%0A%3B) should be the same as before but with the module `open-http` you can get the text that is provided via an url. +The [result in the playground](https://metafacture.org/playground/?flux=%22https%3A//openlibrary.org/books/OL2838758M.json%22%0A%7C+open-http%0A%7C+as-lines%0A%7C+decode-json%0A%7C+encode-yaml%0A%7C+print%0A%3B) should be the same as before but with the module `open-http` you can get the text that is provided via an URL. -But lets understand what a Flux Workflow does. -The Flux-Workflow is combination of different moduls to process incoming semi structured data. In our example we have different things that we do with these modules: +Let's take a look what a Flux workflow does. The Flux workflow is combination of different moduls to process incoming structured data. In our example we have different things that we do with these modules: -First we have a URL. -The URL state the location of the data on the web. -First we tell Metafacture `open-http` to request the stated url. -Then how to handle the data that is incoming: Since the report is writen in one line, we tell Metafacture to regard every new line as a new record with `as-lines` -Afterwards we tell Metafacture to `decode-json` in order to translate the incoming data as json to the generic internal data model that is called metadata events -Lastly Metafacture should serialize the metadata events as YAML with `encode-yaml` -Finally we tell MF to `print` everything. +- First, we have a URL as an input. The URL state the location of the data on the web. +- Second, We tell Metafacture to request the stated url using `open-http`. +- Then how to handle the incoming data: since the JSON is written in one line, we tell Metafacture to regard every new line as a new record with `as-lines` +- Afterwards we tell Metafacture to `decode-json` in order to translate the incoming data as json to the generic internal data model that is called metadata events +- Then Metafacture should serialize the metadata events as YAML with `encode-yaml` +- Finally, we tell MF to `print` everything. So let's have a small recap of what we done and learned so far. We played around with the Metafacture Playground. @@ -257,7 +253,11 @@ Now take some time and play around a little bit more and use some other modules. 1) Try to change the Flux workflow to output as formeta (a metafacture specific data format) and not as YAML. 2) Configure the style of formeta to multiline. -3) Also try not to print but to write the output and call the file that you write weather.xml. +3) Also try not to print but to write the output to a file called book.xml. + +
+ +Click to see the new workflow ``` "https://openlibrary.org/books/OL2838758M.json" @@ -265,9 +265,10 @@ Now take some time and play around a little bit more and use some other modules. | as-lines | decode-json | encode-formeta(style="multiline") -| write("test.xml") +| write("book.xml") ; ``` +
What you see with the modules `encode-formeta` and `write` is that modules can have further specification in brackets. These can eiter be a string in `"..."` or attributes that define options as with `style=`. @@ -277,7 +278,7 @@ The most abstract and most common process resemble the following steps: **→ read → decode → transform → encode → write →** -This process is one that Transforms incoming data in a way that is changed at the end. +This process is one that transforms incoming data in a way that is changed at the end. Each step can be done by one or a combination of multiple modules. Modules are small tools that do parts of the complete task we want to do. @@ -285,12 +286,12 @@ Each modul demands a certain input and give a certain output. This is called sig e.g.: The fist modul `open-file` expects a string and provides read data (called reader). -This reader data can be passed on to a modul that accepts reader data e.g. in our case `as-lines` +This reader data can be passed on to a modul that accepts reader data e.g. in our case `as-lines`. `as-lines` outputs again a string, that is accepted by the `decode-json` module. If you have a look at the flux modul/command documentation then you see under signature which data a modul expects and which data it outputs. -The combination of moduls is a Flux-Workflow. +The combination of moduls is a Flux workflow. Each module is separated by a `|` and every workflow ends with a `;`. Comments can be added with `//`. diff --git a/README.md b/README.md index ee6f8ba..bf7df28 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ This project is early-stage. This is a selective copy of the [Catmandu tutorial](https://librecatproject.wordpress.com/2014/12/01/day-1-getting-catmandu/) but adapted for [Metafacture](https://github.com/metafacture). The Catmandu tutorial was created by [@phochste](https://github.com/phochste). It is great if you want to learn [Catmandu](https://github.com/LibreCat/Catmandu). So try it out. -Since [Metafacture Fix](https://github.com/metafacture/metafacture-fix) is introducing a catmandu-fix like transformation language to Metafacture the addaption of the Catmandu Tutorial for Metafacture purposes seem to me obvioous. +Since [Metafacture Fix](https://github.com/metafacture/metafacture-fix) is introducing a catmandu-fix like transformation language to Metafacture the adaption of the Catmandu Tutorial for Metafacture purposes seem to me obvious. It should help you to get accustomed with Metafacture Core and Metafacture Fix. Have fun. From b0e2bca7091584e4d567a86807428db928675b1e Mon Sep 17 00:00:00 2001 From: Phu Tu Date: Tue, 11 Mar 2025 10:11:57 +0100 Subject: [PATCH 03/10] Corrections for lesson 3 see also TODO --- 03_Introduction_into_Metafacture-Fix.md | 50 +++++++++++++------------ 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/03_Introduction_into_Metafacture-Fix.md b/03_Introduction_into_Metafacture-Fix.md index 6825e91..128ebee 100644 --- a/03_Introduction_into_Metafacture-Fix.md +++ b/03_Introduction_into_Metafacture-Fix.md @@ -1,15 +1,15 @@ # Lesson 3: Introduction into Metafacture Fix -In the last session we learned about Flux-Moduls. -Flux-Moduls can do a lot of things. They configure the the "high-level" transformation pipeline. +In the last session we learned about Flux moduls. +Flux moduls can do a lot of things. They configure the "high-level" transformation pipeline. -But the main transformation of incoming data at record-, elemenet- and value-level is usually done by the transformation moduls: `fix` or `morph` as one step in the pipeline. +But the main transformation of incoming data at record, elemenet and value level is usually done by the transformation moduls `Fix` or `Morph` as one step in the pipeline. -What do we mean when we talk about transformation, e.g.: +By transformation we mean things like: -* Manipulating element-names and element-values +* Manipulating element names and element values * Change hierachies and structures of records -* Lookup values in concordance list. +* Lookup values in concordance list But not changing serialization that is part of encoding and decoding. @@ -45,7 +45,7 @@ The `fix` module in Metafacture is used to manipulate the input data filtering f HINT: As long as you embedd the fix functions in the Flux Workflow, you have to use double quotes to fence the fix functions, and single quotes in the fix functions. As we did here: `fix ("retain('title')")` -Now let us additionally keep the info that is given in the element `"publish_date"` and in the subfield `"key"` as well as the subfield `"key"` in `'type'` by adding `'publish_date', 'type.key'` to `retain`: +Now let us additionally keep the info that is given in the element `"publish_date"` and the subfield `"key"` in `'type'` by adding `'publish_date', 'type.key'` to `retain`: ``` "https://openlibrary.org/books/OL2838758M.json" @@ -68,11 +68,12 @@ type: key: "/type/edition" ``` +**TODO**: In diesem Beispiel macht es im Ergebnis keinen Unterscheid, ob man `type` oder `type.key` adressiert. Vielleicht ein anderes Beispiel wählen wie z. B. `identifiers`, wo mehrere Unterfelder vorhanden sind? -When manipulating data you often need to create many fixes to process a data file in the format and structure you need. With a text editor you can write all fix functions in a singe separate fix-file. +When manipulating data you often need to create many fixes to process a data file in the format and structure you need. With a text editor you can write all fix functions in a singe separate Fix file. -The playground has an transformationFile-content area that can be used as if the fix is in a separate file. -In the playground we use the variable `transformationFile` to adress the fix file in the playground. +The playground has an transformationFile-content area that can be used as if the Fix is in a separate file. +In the playground we use the variable `transformationFile` to adress the Fix file in the playground. Like this. @@ -84,16 +85,16 @@ Fix: retain("title", "publish_date", "type.key") ``` -With this separate fix-file it will be a bit easier to write many fix-functions and it does not overcrowd the flux-workflow. +Using a separate Fix file is recommended if you will need to write many Fix functions. It will keep the Flux workflow clear and legible. -To add more fixes we can again edit the fix file. +To add more fixes we can again edit the Fix file. Lets add these lines in front of the retain function: ``` move_field("type.key", "pub_type") ``` -Also change the `retain` function, so that you keep the new element `"pub_type"` instead of the not existing nested `"key"` element. +Also change the `retain` function so that you keep the new element `"pub_type"` instead of the not existing nested `"key"` element. ``` move_field("type.key","pub_type") @@ -109,14 +110,14 @@ publish_date: "1984" pub_type: "/type/edition" ``` -So with `move_field` we moved and renamed an existing element. +With `move_field` we moved and renamed an existing element. As next step add the following function before the `retain` function. ``` replace_all("pub_type","/type/","") ``` -If you execute your last workflow with the Process-Button again, you should now see as ouput: +If you execute your last workflow with the "Process" button again, you should now see as ouput: ```YAML --- @@ -125,20 +126,21 @@ publish_date: "1984" pub_type: "edition" ``` -We cleaned up the `"pub_type"` element, so that we can better read it. +We cleaned up the value of `"pub_type"` element for better readability. [See the example in the playground.](https://metafacture.org/playground/?flux=%22https%3A//openlibrary.org/books/OL2838758M.json%22%0A%7C+open-http%0A%7C+as-lines%0A%7C+decode-json%0A%7C+fix+%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=move_field%28%22type.key%22%2C%22pub_type%22%29%0Areplace_all%28%22pub_type%22%2C%22/type/%22%2C%22%22%29%0Aretain%28%22title%22%2C+%22publish_date%22%2C+%22pub_type%22%29) -Metafacture contains many fix function to manipulate data. Also there are many flux commands/modules that can be used. +Metafacture contains many Fix functions to manipulate data. Also there are many Flux commands/modules that can be used. -Check the documentation to get a complete list of [flux command](https://github.com/metafacture/metafacture-documentation/blob/master/flux-commands.md) and [fix functions](https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#functions). This post only presented a short introduction into Metafacture. In the next posts we will go deeper into its capabilities. +Check the documentation to get a complete list of [Flux commands](https://metafacture.github.io/metafacture-documentation/docs/flux/flux-commands.html) and [Fix functions](https://metafacture.github.io/metafacture-documentation/docs/fix/Fix-functions.html). This post only presented a short introduction into Metafacture. In the next posts we will go deeper into its capabilities. -Besides fix functions you can also add as many comments and linebreaks as you want to a fix. +Besides Fix functions you can also add as many comments and linebreaks as you want to a Fix. -Comments are good if you want to add descriptions to you transformation. Like the following. -Comments in Fix start with a hashtag `#`, while in Flux they start with `//` +Adding comments will save you a lot of time and effort when you look at your code in the future. -e.g.: +Comments in Fix start with a hash mark `#`, while in Flux they start with `//`. + +Example: ``` # Make type.key a top level element. @@ -162,9 +164,9 @@ Have a look at the fix functions: https://metafacture.org/metafacture-documentat
Answer -[See here](https://metafacture.org/playground/?flux=%22https%3A//openlibrary.org/books/OL2838758M.json%22%0A%7C+open-http%0A%7C+as-lines%0A%7C+decode-json%0A%7C+fix+%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=move_field%28%22type.key%22%2C%22pub_type%22%29%0Areplace_all%28%22pub_type%22%2C%22/type/%22%2C%22%22%29%0Aadd_field%28%22mape_date%22%2C%222025-11-11%22%29%0Aretain%28%22title%22%2C+%22publish_date%22%2C+%22by_statement%22%2C+%22pub_type%22%29) +[See here](https://metafacture.org/playground/?flux=%22https%3A//openlibrary.org/books/OL2838758M.json%22%0A%7C+open-http%0A%7C+as-lines%0A%7C+decode-json%0A%7C+fix+%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=move_field%28%22type.key%22%2C%22pub_type%22%29%0Areplace_all%28%22pub_type%22%2C%22/type/%22%2C%22%22%29%0Aadd_field%28%22map_date%22%2C%222025-11-11%22%29%0Aretain%28%22title%22%2C+%22publish_date%22%2C+%22by_statement%22%2C+%22pub_type%22%2C+%22map_date%22%29) -or [use timestamp](https://metafacture.org/playground/?flux=%22https%3A//openlibrary.org/books/OL2838758M.json%22%0A%7C+open-http%0A%7C+as-lines%0A%7C+decode-json%0A%7C+fix+%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=move_field%28%22type.key%22%2C%22pub_type%22%29%0Areplace_all%28%22pub_type%22%2C%22/type/%22%2C%22%22%29%0Atimestamp%28%22mape_date%22%2Cformat%3A%22yyyy-MM-dd%27T%27HH%3Amm%3Ass%22%2C+timezone%3A%22Europe/Berlin%22%29%0Aretain%28%22title%22%2C+%22publish_date%22%2C+%22by_statement%22%2C+%22pub_type%22%29) +or [use timestamp](https://metafacture.org/playground/?flux=%22https%3A//openlibrary.org/books/OL2838758M.json%22%0A%7C+open-http%0A%7C+as-lines%0A%7C+decode-json%0A%7C+fix+%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=move_field%28%22type.key%22%2C%22pub_type%22%29%0Areplace_all%28%22pub_type%22%2C%22/type/%22%2C%22%22%29%0Atimestamp%28%22map_date%22%2Cformat%3A%22yyyy-MM-dd%27T%27HH%3Amm%3Ass%22%2C+timezone%3A%22Europe/Berlin%22%29%0Aretain%28%22title%22%2C+%22publish_date%22%2C+%22by_statement%22%2C+%22pub_type%22%2C+%22map_date%22%29)
Next lesson: [04 Fix Path](./04_FIX-Path.md) From 23d23e57f7a6a3f4a0df1cafee8d971412611e81 Mon Sep 17 00:00:00 2001 From: Phu Tu Date: Tue, 11 Mar 2025 10:34:37 +0100 Subject: [PATCH 04/10] Minor typo fix --- 03_Introduction_into_Metafacture-Fix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/03_Introduction_into_Metafacture-Fix.md b/03_Introduction_into_Metafacture-Fix.md index 128ebee..ccf9ad1 100644 --- a/03_Introduction_into_Metafacture-Fix.md +++ b/03_Introduction_into_Metafacture-Fix.md @@ -85,7 +85,7 @@ Fix: retain("title", "publish_date", "type.key") ``` -Using a separate Fix file is recommended if you will need to write many Fix functions. It will keep the Flux workflow clear and legible. +Using a separate Fix file is recommended if you need to write many Fix functions. It will keep the Flux workflow clear and legible. To add more fixes we can again edit the Fix file. Lets add these lines in front of the retain function: From 7352b3051b2573032becc7f509fe07f75f1c744f Mon Sep 17 00:00:00 2001 From: Phu Tu Date: Wed, 12 Mar 2025 13:00:55 +0100 Subject: [PATCH 05/10] Revision for lesson 4 --- 04_Fix-Path.md | 178 +++++++++++++++++-------------------------------- 1 file changed, 63 insertions(+), 115 deletions(-) diff --git a/04_Fix-Path.md b/04_Fix-Path.md index 23e8f44..d7d261e 100644 --- a/04_Fix-Path.md +++ b/04_Fix-Path.md @@ -1,8 +1,10 @@ -# Lesson 4: Fix Path and more complex transfromations in Fix +# Lesson 4: FixPath and more complex transformations in Fix -Last sessions we learned the how to construct a metafacture workflow, how to use the Playground and how and how Metafacture Flux and Fix can be used to parse structured information. Today we will go deeper into Metafacture Fix and describe how to pluck data out of structured information. +Over the last lessons we learned how to construct a Metafacture workflow, how to use the Playground and how Metafacture Flux and Fix can be used to parse structured information. We saw how you can use Flux to transform the JSON format into the YAML format which is easier to read and contains the same information. We also learned how to retrieve information out of the JSON file using a Fix function like `retain("title", "publish_date", "type.key")`. -Today will we fetch of a new book with the Metafacture Playground: +In this lesson we will go deeper into Metafacture Fix and describe how to pluck data out of structured information. + +First, let's fetch of a new book with the Metafacture Playground: ``` "https://openlibrary.org/books/OL27333998M.json" @@ -14,89 +16,52 @@ Today will we fetch of a new book with the Metafacture Playground: ; ``` -We also saw in the previous post how you can use Metafacture to transform the JSON format into the YAML format which is easier to read and contains the same information. - -We also learned some fixes e.g. to retrieve information out of the JSON file like `retain("title", "publish_date", "type.key")`. +The printed output in YAML format contains a collection of key:value pairs. -In this post we delve a bit deeper into ways how to point to fields in a JSON or a YAML file: +There are top level fields like: `title` or `publish_date` which contain only text values or numbers: ```YAML --- -publishers: -- "Simon & Schuster" -number_of_pages: "368" -subtitle: "A Theory" -covers: -- "8798647" -physical_format: "paperback" -full_title: "Bullshit Jobs A Theory" -key: "/books/OL27333998M" -authors: -- key: "/authors/OL1395062A" -source_records: -- "amazon:1501143336" -- "bwb:9781501143335" -- "marc:marc_columbia/Columbia-extract-20221130-034.mrc:71583959:3725" -- "promise:bwb_daily_pallets_2023-05-10:W8-BRV-242" title: "Bullshit Jobs" -notes: "Source title: Bullshit Jobs: A Theory" -identifiers: - amazon: - - "1501143336" -publish_date: "May 07, 2019" -works: -- key: "/works/OL20153626W" -type: - key: "/type/edition" -local_id: -- "urn:bwbsku:W8-BRV-242" -isbn_10: -- "1501143336" -isbn_13: -- "9781501143335" -lccn: -- "2021276048" -oclc_numbers: -- "1056738022" -classifications: {} -lc_classifications: -- "HF5549.5.J63 G73 2019" -languages: -- key: "/languages/eng" -latest_revision: "6" -revision: "6" +``` + +There are also fields like `created` that contain a nested structure where the value is again a key:value pair: + +```YAML +--- created: type: "/type/datetime" value: "2019-10-04T04:03:07.194846" -last_modified: - type: "/type/datetime" - value: "2023-08-05T12:37:41.711036" ``` -`type.key` is called a *Path* that is JSON Path-like and points to a part of the data set - here our Yaml record - you are interested in. The data, as shown above, is structured like a tree. - -There are top level simple fields like: `title`, `publish_date`, `notes` and `latest_revision` which contain only text values or numbers. Depending on the context simple fields can also be named: elemente, properties, attribute or key. +And fields like `source_records[]` do have a list as value: -There are also fields like `created` that contain a deeper structure like `type` and `value`.Nested elements that contain one or more subfields or subelements are also called objects or hash. -And there are lists like `source_records[]` to which I come back later. +```YAML +--- +source_records: +- "amazon:1501143336" +- "bwb:9781501143335" +- "marc:marc_columbia/Columbia-extract-20221130-034.mrc:71583959:3725" +- "promise:bwb_daily_pallets_2023-05-10:W8-BRV-242" +``` -Metafacture Fix is using Fix Path, a path-syntax that is JSON Path like but not identical. It also uses the dot notation but there are some differences with the path structure of arrays and repeated fields. Especially when working with JSON or YAML. +Metafacture Fix is using FixPath to access and selectively extract data out of YAML and JSON documents. It is JSONPath like but not identical. It also uses the dot notation but there are some differences with the path structure of arrays and repeated fields. -Using a JSON path you can point to every part of the JSON file using a dot-notation. For simple top level fields the path is just the name of the field: +You can point to every part of the YAML file using a dot-notation. For simple top level fields the path is just the name of the field: * `title` * `publish_date` * `notes` * `latest_revision` -For the nested objects with deeper structure you add a dot `.` to point to the subfields: +For nested objects with deeper structure you add a dot `.` to point to the subfields: * `type.key` * `created.type` * `last_modified.value` -So for example. If you would have a deeply nested structure like this object: +If you would have a deeply nested structure like this object: ```YAML x: @@ -107,15 +72,15 @@ x: c: Hello :-) ``` -Then you would point to the c field with the path to reference the element would be `x.y.z.a.b.c`. +Then you would point to the c field with this path: `x.y.z.a.b.c`. So lets do some simple excercises: -[Try and complete the fix functions. Transform the element `a` into `title` and combine the subfields of `b` and `c` to the element `author`.](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-yaml%0A%7Cfix%28transformationFile%29%0A%7Cencode-json%28prettyPrinting%3D%22true%22%29%0A%7Cprint%0A%3B&transformation=move_field%28%22a%22%22%2C+%22title%22%29%0Apaste%28%22author%22%2C+%22...%22%2C+...%2C+%22~from%22%2C+...%29%0Aretain%28%22title%22%2C+%22author%22%29&data=---%0Aa%3A+Faust%0Ab+%3A%0A++ln%3A+Goethe%0A++fn%3A+JW%0Ac%3A+Weimar%0A%0A---%0Aa%3A+R%C3%A4uber%0Ab+%3A%0A++++ln%3A+Schiller%0A++++fn%3A+F%0Ac%3A+Weimar) +[Try and complete the fix functions. Transform the element `a` into `title` and combine the subfields of `b` and `c` to the element `author`.](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-yaml%0A%7Cfix%28transformationFile%29%0A%7Cencode-json%28prettyPrinting%3D%22true%22%29%0A%7Cprint%0A%3B&transformation=move_field%28%22a%22%2C+%22title%22%29%0Apaste%28%22author%22%2C+%22...%22%2C+...%2C+%22~from%22%2C+...%29%0Aretain%28%22title%22%2C+%22author%22%29&data=---%0Aa%3A+Faust%0Ab+%3A%0A++ln%3A+Goethe%0A++fn%3A+JW%0Ac%3A+Weimar%0A%0A---%0Aa%3A+R%C3%A4uber%0Ab+%3A%0A++++ln%3A+Schiller%0A++++fn%3A+F%0Ac%3A+Weimar)
Answer -[See here](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-yaml%0A%7Cfix%28transformationFile%29%0A%7Cencode-json%28prettyPrinting%3D%22true%22%29%0A%7Cprint%0A%3B&transformation=move_field%28%22a%22%22%2C+%22title%22%29%0Apaste%28%22author%22%2C+%22...%22%2C+...%2C+%22~from%22%2C+...%29%0Aretain%28%22title%22%2C+%22author%22%29&data=---%0Aa%3A+Faust%0Ab+%3A%0A++ln%3A+Goethe%0A++fn%3A+JW%0Ac%3A+Weimar%0A%0A---%0Aa%3A+R%C3%A4uber%0Ab+%3A%0A++++ln%3A+Schiller%0A++++fn%3A+F%0Ac%3A+Weimar) +[See here](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-yaml%0A%7Cfix%28transformationFile%29%0A%7Cencode-json%28prettyPrinting%3D%22true%22%29%0A%7Cprint%0A%3B&transformation=move_field%28%22a%22%2C+%22title%22%29%0Apaste%28%22author%22%2C+%22b.fn%22%2C+%22b.ln%22%2C+%22~from%22%2C+%22c%22%29%0Aretain%28%22title%22%2C+%22author%22%29&data=---%0Aa%3A+Faust%0Ab+%3A%0A++ln%3A+Goethe%0A++fn%3A+JW%0Ac%3A+Weimar%0A%0A---%0Aa%3A+R%C3%A4uber%0Ab+%3A%0A++++ln%3A+Schiller%0A++++fn%3A+F%0Ac%3A+Weimar)
## Repeated fields and arrays @@ -125,21 +90,13 @@ There are two extra path structures that need to be explained: * repeated fields * arrays -In general: Repeated fields as well arrays are both handled as arrays. They can also call these internal arrays lists. -Both names (list and array) are reflected in some fix functions (e.g. `add_array` or the `list`-Bind.) +For both repeated fields and arrays you need to use an **index** to select an element. -In an data set an element sometimes can have multiple instances. Different data models solve this possibility differently. XML-Records can have all elements multiple times, element repition is possible and in many schemas it is (partly) allowed. E.g. the subject element exists three times: +For arrays specifically you also need to use an **array marker**. ### Working with repeated fields -```XML -Metadata -Datatransformation -ETL -``` - - -Repeatable elements also exist e.g. in JSON and YAML but are unusual: +In a data set an element sometimes can have multiple instances: ```YAML creator: Justus @@ -147,22 +104,22 @@ creator: Peter creator: Bob ``` -In our two examples the `subject`- and `creator`-element exists three times. To point to one of the elements you need to use an index. The index is one-based: The first index has value 1, the second the value 2, the third the value 3. So, the path of the creator Bob would be `creator.3`. (This is a main difference between Catmandu and Metafacture because Catmandu has an zero based index.) +To point to one of the `creator` elements you need to use an index. The first index has value 1, the second the value 2, the third the value 3. So, the path of the creator Bob would be `creator.3`. (In contrast, Catmandu uses an zero based index starting with 0 as the first index.) -If you want to refer to all creators then you can use the array wildcard `*` which can replace the concrete index number: `creator.*` refers to all creator elements. You can also select the the first instance with the array wildcard `$first` and the last `$last`. This is espacially handy if you do not know how often an element is repeated. When adding an additional repeated element you usually use the `$append` wildcard. +If you want to refer to all creators then you can use the `*` sign as a wildcard: `creator.*` refers to all creator elements. The first instance can be selected by the `$first` wildcard and the last by `$last`. This is espacially handy if you do not know how often an element is repeated. When adding an additional repeated element you can use the `$append` or `$prepend` wildcards. -[Prepend the correct last name to the three investigators: Justus Jonas, Peter Shaw and Bob Andrews. Also append Investigator to all of them.](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-yaml%0A%7Cfix%28transformationFile%29%0A%7Cencode-json%28prettyPrinting%3D%22true%22%29%0A%7Cprint%0A%3B&transformation=&data=---%0Acreator%3A+Justus%0Acreator%3A+Peter%0Acreator%3A+Bob%0A) +[Append the correct last name to the three investigators: Justus Jonas, Peter Shaw and Bob Andrews. Also prepend "Investigator" to all of them.](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-yaml%0A%7Cfix%28transformationFile%29%0A%7Cencode-json%28prettyPrinting%3D%22true%22%29%0A%7Cprint%0A%3B&transformation=&data=---%0Acreator%3A+Justus%0Acreator%3A+Peter%0Acreator%3A+Bob%0A)
Answer [See here](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-yaml%0A%7Cfix%28transformationFile%29%0A%7Cencode-json%28prettyPrinting%3D%22true%22%29%0A%7Cprint%0A%3B&transformation=append%28%22creator.1%22%2C%22+Jonas%22%29%0Aappend%28%22creator.2%22%2C%22+Shaw%22%29%0Aappend%28%22creator.3%22%2C%22+Andrews%22%29%0Aprepend%28%22creator.%2A%22%2C%22Investigator+%22%29&data=---%0Acreator%3A+Justus%0Acreator%3A+Peter%0Acreator%3A+Bob%0A)
-### Working with JSON and Yaml arrays +### Working with arrays -In JSON or YAML element repetion is possible but unusual. Instead of repeating elements repetition is constructed as list so that an element can have more than one value. This is called an array and looks like this in YAML: +In JSON or YAML element repetion is possible but unusual. Instead of repeating elements an element can have a list or array of values. -In our book example e.g. we have the following array: +In our book example we have an array as value: ``` source_records: @@ -172,7 +129,7 @@ source_records: - "promise:bwb_daily_pallets_2023-05-10:W8-BRV-242" ``` -Our example from above would look like this if creator was a list instead of an repeated field: +Our example from above would look like this if creator was a list instead of a repeated field: ```YAML creator: @@ -181,17 +138,7 @@ creator: - Bob ``` -or: - -```YAML -my: - colors: - - black - - red - - yellow -``` - -Also lists can be deeply nested, if they are not just lists of strings (array of strings) but of objects (array of objects). +Lists can be deeply nested if the values are not just strings (list of strings) but objects (list of objects): ```YAML characters: @@ -203,48 +150,49 @@ characters: role: Research & Archive ``` -In the colour example above you see a field `my` which contains a deeper field `colors` which has 3 values. To point to one of the colors you need to use an index but also genuin arrays have a marker in Metafacture: `[]`. Also here the first index in a array has value 1, the second the value 2, the third the value 3. The array markers are generated by the [JSON-Decoder](https://github.com/metafacture/metafacture-documentation/blob/master/flux-commands.md#decode-json) and the [YAML-Decoder](https://github.com/metafacture/metafacture-documentation/blob/master/flux-commands.md#decode-yaml). Also if you want to generate an array in the target format, then you need to add `[]` at the end of an list-element like `newArray[]`. (While sofare the path handling of Catmandu and Metafacture are similar, they differ at this point.) +Here is another example: -So, the path of the `red` would be: `my.color[].2` -And the path for `Peter` would be `characters[].2.name` +```YAML +my: + colors: + - black + - red + - yellow +``` -There is one array type in our JSON report from our example at the beginning above and that is the `weather` field. To point to the description of the weather you need the path `weather[].1.description`. +To point to one of the colors you need to use an index but also an array marker: `[]`. -| elements | objects | array/repeated field | -|---|---|---| -| need path | need dots to mark nested structure | need index/array-wildcards to refer to specific position | -| `id` | `title.subtitle` | `author.*.firstName` | -| `name` | `very.nested.element` | `my.color.2` | +So, the path of the `red` would be: `my.colors[].2` -Excercise: +And the path for `Peter` would be `characters[].2.name` + +Also if you want to generate an array in the target format, then you need to add `[]` at the end of an list element like `newArray[]`. + +## Excercise: -[Only `retain` the elements of title, the element of the series and the role of Bob Andrews. You have to identify the paths for said elements.](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-json%0A%7Cfix%28transformationFile%29%0A%7Cencode-yaml%0A%7Cprint%0A%3B&transformation=retain%28...%29&data=%7B%0A++%22title%22+%3A+%22The+Secret+of+Terror+Castle%22%2C%0A++%22isPartOf%22+%3A+%7B%0A++++%22series%22+%3A+%22The+Three+Investigators%22%2C%0A++++%22volume%22+%3A+%221%22%0A++%7D%2C%0A++%22releaseDate%22+%3A+%221964%22%2C%0A++%22author%22+%3A+%22Robert+Arthur%22%2C%0A++%22characters%22+%3A+%5B+%7B%0A++++%22name%22+%3A+%22Jupiter+Jones%22%2C%0A++++%22role%22+%3A+%22Investigator%22%0A++%7D%2C+%7B%0A++++%22name%22+%3A+%22Peter+Crenshaw%22%2C%0A++++%22role%22+%3A+%22Investigator%22%0A++%7D%2C+%7B%0A++++%22name%22+%3A+%22Bob+Andrews%22%2C%0A++++%22role%22+%3A+%22Research+%26+Archive%22%0A++%7D+%5D%0A%7D) +[Only `retain` the title and the series plus the name and role of Bob Andrews. You have to identify the paths for said elements.](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-json%0A%7Cfix%28transformationFile%29%0A%7Cencode-yaml%0A%7Cprint%0A%3B&transformation=retain%28...%29&data=%7B%0A++%22title%22+%3A+%22The+Secret+of+Terror+Castle%22%2C%0A++%22isPartOf%22+%3A+%7B%0A++++%22series%22+%3A+%22The+Three+Investigators%22%2C%0A++++%22volume%22+%3A+%221%22%0A++%7D%2C%0A++%22releaseDate%22+%3A+%221964%22%2C%0A++%22author%22+%3A+%22Robert+Arthur%22%2C%0A++%22characters%22+%3A+%5B+%7B%0A++++%22name%22+%3A+%22Jupiter+Jones%22%2C%0A++++%22role%22+%3A+%22Investigator%22%0A++%7D%2C+%7B%0A++++%22name%22+%3A+%22Peter+Crenshaw%22%2C%0A++++%22role%22+%3A+%22Investigator%22%0A++%7D%2C+%7B%0A++++%22name%22+%3A+%22Bob+Andrews%22%2C%0A++++%22role%22+%3A+%22Research+%26+Archive%22%0A++%7D+%5D%0A%7D)
Answer -[See here](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-json%0A%7Cfix%28transformationFile%29%0A%7Cencode-yaml%0A%7Cprint%0A%3B&transformation=retain%28%22title%22%2C%22isPartOf.series%22%2C%22characters%5B%5D.3.name%22%29&data=%7B%0A++%22title%22+%3A+%22The+Secret+of+Terror+Castle%22%2C%0A++%22isPartOf%22+%3A+%7B%0A++++%22series%22+%3A+%22The+Three+Investigators%22%2C%0A++++%22volume%22+%3A+%221%22%0A++%7D%2C%0A++%22releaseDate%22+%3A+%221964%22%2C%0A++%22author%22+%3A+%22Robert+Arthur%22%2C%0A++%22characters%22+%3A+%5B+%7B%0A++++%22name%22+%3A+%22Jupiter+Jones%22%2C%0A++++%22role%22+%3A+%22Investigator%22%0A++%7D%2C+%7B%0A++++%22name%22+%3A+%22Peter+Crenshaw%22%2C%0A++++%22role%22+%3A+%22Investigator%22%0A++%7D%2C+%7B%0A++++%22name%22+%3A+%22Bob+Andrews%22%2C%0A++++%22role%22+%3A+%22Research+%26+Archive%22%0A++%7D+%5D%0A%7D) +[See here](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-json%0A%7Cfix%28transformationFile%29%0A%7Cencode-yaml%0A%7Cprint%0A%3B&transformation=retain%28%22title%22%2C%22isPartOf.series%22%2C%22characters%5B%5D.3%22%29&data=%7B%0A++%22title%22+%3A+%22The+Secret+of+Terror+Castle%22%2C%0A++%22isPartOf%22+%3A+%7B%0A++++%22series%22+%3A+%22The+Three+Investigators%22%2C%0A++++%22volume%22+%3A+%221%22%0A++%7D%2C%0A++%22releaseDate%22+%3A+%221964%22%2C%0A++%22author%22+%3A+%22Robert+Arthur%22%2C%0A++%22characters%22+%3A+%5B+%7B%0A++++%22name%22+%3A+%22Jupiter+Jones%22%2C%0A++++%22role%22+%3A+%22Investigator%22%0A++%7D%2C+%7B%0A++++%22name%22+%3A+%22Peter+Crenshaw%22%2C%0A++++%22role%22+%3A+%22Investigator%22%0A++%7D%2C+%7B%0A++++%22name%22+%3A+%22Bob+Andrews%22%2C%0A++++%22role%22+%3A+%22Research+%26+Archive%22%0A++%7D+%5D%0A%7D)
[Again append the last names to the specific character Justus Jonas, Peter Shaw and Bob Andrews. Also add a field to each character "type":"Person"`](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-yaml%0A%7Cfix%28transformationFile%29%0A%7Cencode-json%28prettyPrinting%3D%22true%22%29%0A%7Cprint%0A%3B&transformation=&data=---%0Acharacters%3A+%0A++-+name%3A+Justus%0A++++role%3A+Investigator%0A++-+name%3A+Peter%0A++++role%3A+Investigator%0A++-+name%3A+Bob%0A++++role%3A+Research+%26+Archive%0A) -
Answer -[See here](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-yaml%0A%7Cfix%28transformationFile%29%0A%7Cencode-json%28prettyPrinting%3D%22true%22%29%0A%7Cprint%0A%3B&transformation=append%28%22characters%5B%5D.1.name%22%2C%22+Jonas%22%29%0Aappend%28%22characters%5B%5D.2.name%22%2C%22+Shaw%22%29%0Aappend%28%22characters%5B%5D.3.name%22%2C%22+Andrews%22%29%0Aadd_field%28%22characters%5B%5D.%2A.type%22%2C%22+Andrews%22%29&data=---%0Acharacters%3A+%0A++-+name%3A+Justus%0A++++role%3A+Investigator%0A++-+name%3A+Peter%0A++++role%3A+Investigator%0A++-+name%3A+Bob%0A++++role%3A+Research+%26+Archive%0A) +[See here](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-yaml%0A%7Cfix%28transformationFile%29%0A%7Cencode-json%28prettyPrinting%3D%22true%22%29%0A%7Cprint%0A%3B&transformation=append%28%22characters%5B%5D.1.name%22%2C%22+Jonas%22%29%0Aappend%28%22characters%5B%5D.2.name%22%2C%22+Shaw%22%29%0Aappend%28%22characters%5B%5D.3.name%22%2C%22+Andrews%22%29%0Aadd_field%28%22characters%5B%5D.%2A.type%22%2C%22+Person%22%29&data=---%0Acharacters%3A+%0A++-+name%3A+Justus%0A++++role%3A+Investigator%0A++-+name%3A+Peter%0A++++role%3A+Investigator%0A++-+name%3A+Bob%0A++++role%3A+Research+%26+Archive%0A)
-In this post we learned the JSON Path syntax and how it can be used to point to parts of a JSON data set want to manipulate. We explained the Fix path using a YAML transformation as example, because this is easier to read. - -Especially when working with complex bibliographic data one has to get to know the paths so that you do not have to guess what a path to a certain element is: +In this post we learned the FixPath syntax and how it can be used to point to parts of a YAML or JSON data set we want to manipulate. -There exists multiple ways to find out the path-names of records: +Especially when working with complex bibliographic data one has to get to know the paths so that you do not have to guess what a path to a certain element is. -e.g.: -[Here a way to show pathways in combination with values.](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-lines%0A%7C+decode-pica%0A%7C+fix%28%22nothing%28%29%22%2C+repeatedFieldsToEntities+%3D+%22true%22%29%0A%7C+flatten%0A%7C+encode-literals%0A%7C+print%0A%3B&data=001@+%1Fa5%1F01-2%1E001A+%1F01100%3A15-10-94%1E001B+%1F09999%3A12-06-06%1Ft16%3A10%3A17.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aag%1E003@+%1F0482147350%1E006U+%1F094%2CP05%1E007E+%1F0U+70.16407%1E007I+%1FSo%1F074057548%1E011@+%1Fa1970%1E017A+%1Farh%1E021A+%1FaDie+@Berufsfreiheit+der+Arbeitnehmer+und+ihre+Ausgestaltung+in+vo%CC%88lkerrechtlichen+Vertra%CC%88gen%1FdEine+Grundrechtsbetrachtg%1E028A+%1F9106884905%1F7Tn3%1FAgnd%1F0106884905%1FaProjahn%1FdHorst+D.%1E033A+%1FpWu%CC%88rzburg%1E034D+%1FaXXXVIII%2C+165+S.%1E034I+%1Fa8%1E037C+%1FaWu%CC%88rzburg%2C+Jur.+F.%2C+Diss.+v.+7.+Aug.+1970%1E%0A001@+%1F01%1Fa5%1E001A+%1F01140%3A08-12-99%1E001B+%1F09999%3A05-01-08%1Ft22%3A57%3A29.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aa%1E003@+%1F0958090564%1E004A+%1Ffkart.+%3A+DM+9.70%2C+EUR+4.94%2C+sfr+8.00%2C+S+68.00%1E006U+%1F000%2CB05%2C0285%1E007I+%1FSo%1F076088278%1E011@+%1Fa1999%1E017A+%1Farb%1Fasi%1E019@+%1FaXA-AT%1E021A+%1FaZukunft+Bildung%1FhPolitische+Akademie.+%5BHrsg.+von+Gu%CC%88nther+R.+Burkert-Dottolo+und+Bernhard+Moser%5D%1E028C+%1F9130681849%1F7Tp1%1FVpiz%1FAgnd%1F0130681849%1FE1952%1FaBurkert%1FdGu%CC%88nther+R.%1FBHrsg.%1E033A+%1FpWien%1FnPolit.+Akad.%1E034D+%1Fa79+S.%1E034I+%1Fa24+cm%1E036F+%1Fx299+12%1F9551720077%1FgAdn%1F7Tb1%1FAgnd%1F01040469-7%1FaPolitische+Akademie%1FgWien%1FYPA-Information%1FhPolitische+Akademie%2C+WB%1FpWien%1FJPolitische+Akad.%2C+WB%1Fl99%2C2%1E036F/01+%1Fx12%1F9025841467%1FgAdvz%1Fi2142105-5%1FYAktuelle+Fragen+der+Politik%1FhPolitische+Akademie%1FpWien%1FJPolitische+Akad.+der+O%CC%88VP%1FlBd.+2%1E045E+%1Fa22%1Fd18%1Fm370%1E047A+%1FSFE%1Fata%1E%0A001@+%1Fa5%1F01%1E001A+%1F01140%3A19-02-03%1E001B+%1F09999%3A19-06-11%1Ft01%3A20%3A13.000%1E001D+%1F09999%3A26-04-03%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aal%1E003@+%1F0361809549%1E004A+%1FfHlw.%1E006U+%1F000%2CL01%1E006U+%1F004%2CP01-s-41%1E006U+%1F004%2CP01-f-21%1E007G+%1FaDNB%1F0361809549%1E007I+%1FSo%1F072658383%1E007M+%1F04413/0275%1E011@+%1Fa1925%1E019@+%1FaXA-DXDE%1FaXA-DE%1E021A+%1FaHundert+Jahre+Buchdrucker-Innung+Hamburg%1FdWesen+u.+Werden+d.+Vereinigungen+Hamburger+Buchdruckereibesitzer+1825-1925+%3B+Gedenkschrift+zur+100.+Wiederkehr+d.+Gru%CC%88ndungstages%2C+verf.+im+Auftr.+d.+Vorstandes+d.+Buchdrucker-Innung+%28Freie+Innung%29+zu+Hamburg%1FhFriedrich+Voeltzer%1E028A+%1F9101386281%1F7Tp1%1FVpiz%1FAgnd%1F0101386281%1FE1895%1FaVo%CC%88ltzer%1FdFriedrich%1E033A+%1FpHamburg%1FnBuchdrucker-Innung+%28Freie+Innung%29%1E033A+%1FpHamburg%1Fn%5BVerlagsbuchh.+Broschek+%26+Co.%5D%1E034D+%1Fa44+S.%1E034I+%1Fa4%1E%0A001@+%1Fa5%1F01-3%1E001A+%1F01240%3A01-08-95%1E001B+%1F09999%3A24-09-10%1Ft17%3A42%3A20.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Af%1E003@+%1F0945184085%1E004A+%1F03-89007-044-2%1FfGewebe+%3A+DM+198.00%2C+sfr+198.00%2C+S+1386.00%1E006T+%1F095%2CN35%2C0856%1E006U+%1F095%2CA48%2C1186%1E006U+%1F010%2CP01%1E007I+%1FSo%1F061975997%1E011@+%1Fa1995%1E017A+%1Fara%1E021A+%1Fx213%1F9550711899%1FYNeues+Handbuch+der+Musikwissenschaft%1Fhhrsg.+von+Carl+Dahlhaus.+Fortgef.+von+Hermann+Danuser%1FpLaaber%1FJLaaber-Verl.%1FS48%1F03-89007-030-2%1FgAc%1E021B+%1FlBd.+13.%1FaRegister%1Fhzsgest.+von+Hans-Joachim+Hinrichsen%1E028C+%1F9121445453%1F7Tp3%1FVpiz%1FAgnd%1F0121445453%1FE1952%1FaHinrichsen%1FdHans-Joachim%1E034D+%1FaVIII%2C+408+S.%1E045V+%1F9090001001%1E047A+%1FSFE%1Fagb/fm%1E%0A001@+%1F01-2%1Fa5%1E001A+%1F01239%3A18-08-11%1E001B+%1F09999%3A05-09-11%1Ft23%3A31%3A44.000%1E001D+%1F01240%3A30-08-11%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Af%1E003@+%1F01014417392%1E004A+%1Ffkart.%1E006U+%1F011%2CA37%1E007G+%1FaDNB%1F01014417392%1E007I+%1FSo%1F0752937239%1E010@+%1Fager%1E011@+%1Fa2011%1E017A+%1Fara%1Fasf%1E021A+%1Fxtr%1F91014809657%1F7Tp3%1FVpiz%1FAgnd%1F01034622773%1FE1958%1FaLu%CC%88beck%1FdMonika%1FYPersonalwirtschaft+mit+DATEV%1FhMonika+Lu%CC%88beck+%3B+Helmut+Lu%CC%88beck%1FpBodenheim%1FpWien%1FJHerdt%1FRXA-DE%1FS650%1FgAc%1E021B+%1FlTrainerbd.%1E032@+%1Fg11%1Fa1.+Ausg.%1E034D+%1Fa129+S.%1E034M+%1FaIll.%1E047A+%1FSFE%1Famar%1E047A+%1FSERW%1Fasal%1E047I+%1Fu%24%1Fc04%1FdDNB%1Fe1%1E) +There exists multiple ways to find out the path-names of records. Two examples: -[Here is a way to collect and count all paths in all records by using the `list-fix-paths`-command.](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-lines%0A%7C+decode-pica%0A%7C+list-fix-paths%0A%7C+print%0A%3B&data=001@+%1Fa5%1F01-2%1E001A+%1F01100%3A15-10-94%1E001B+%1F09999%3A12-06-06%1Ft16%3A10%3A17.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aag%1E003@+%1F0482147350%1E006U+%1F094%2CP05%1E007E+%1F0U+70.16407%1E007I+%1FSo%1F074057548%1E011@+%1Fa1970%1E017A+%1Farh%1E021A+%1FaDie+@Berufsfreiheit+der+Arbeitnehmer+und+ihre+Ausgestaltung+in+vo%CC%88lkerrechtlichen+Vertra%CC%88gen%1FdEine+Grundrechtsbetrachtg%1E028A+%1F9106884905%1F7Tn3%1FAgnd%1F0106884905%1FaProjahn%1FdHorst+D.%1E033A+%1FpWu%CC%88rzburg%1E034D+%1FaXXXVIII%2C+165+S.%1E034I+%1Fa8%1E037C+%1FaWu%CC%88rzburg%2C+Jur.+F.%2C+Diss.+v.+7.+Aug.+1970%1E%0A001@+%1F01%1Fa5%1E001A+%1F01140%3A08-12-99%1E001B+%1F09999%3A05-01-08%1Ft22%3A57%3A29.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aa%1E003@+%1F0958090564%1E004A+%1Ffkart.+%3A+DM+9.70%2C+EUR+4.94%2C+sfr+8.00%2C+S+68.00%1E006U+%1F000%2CB05%2C0285%1E007I+%1FSo%1F076088278%1E011@+%1Fa1999%1E017A+%1Farb%1Fasi%1E019@+%1FaXA-AT%1E021A+%1FaZukunft+Bildung%1FhPolitische+Akademie.+%5BHrsg.+von+Gu%CC%88nther+R.+Burkert-Dottolo+und+Bernhard+Moser%5D%1E028C+%1F9130681849%1F7Tp1%1FVpiz%1FAgnd%1F0130681849%1FE1952%1FaBurkert%1FdGu%CC%88nther+R.%1FBHrsg.%1E033A+%1FpWien%1FnPolit.+Akad.%1E034D+%1Fa79+S.%1E034I+%1Fa24+cm%1E036F+%1Fx299+12%1F9551720077%1FgAdn%1F7Tb1%1FAgnd%1F01040469-7%1FaPolitische+Akademie%1FgWien%1FYPA-Information%1FhPolitische+Akademie%2C+WB%1FpWien%1FJPolitische+Akad.%2C+WB%1Fl99%2C2%1E036F/01+%1Fx12%1F9025841467%1FgAdvz%1Fi2142105-5%1FYAktuelle+Fragen+der+Politik%1FhPolitische+Akademie%1FpWien%1FJPolitische+Akad.+der+O%CC%88VP%1FlBd.+2%1E045E+%1Fa22%1Fd18%1Fm370%1E047A+%1FSFE%1Fata%1E%0A001@+%1Fa5%1F01%1E001A+%1F01140%3A19-02-03%1E001B+%1F09999%3A19-06-11%1Ft01%3A20%3A13.000%1E001D+%1F09999%3A26-04-03%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aal%1E003@+%1F0361809549%1E004A+%1FfHlw.%1E006U+%1F000%2CL01%1E006U+%1F004%2CP01-s-41%1E006U+%1F004%2CP01-f-21%1E007G+%1FaDNB%1F0361809549%1E007I+%1FSo%1F072658383%1E007M+%1F04413/0275%1E011@+%1Fa1925%1E019@+%1FaXA-DXDE%1FaXA-DE%1E021A+%1FaHundert+Jahre+Buchdrucker-Innung+Hamburg%1FdWesen+u.+Werden+d.+Vereinigungen+Hamburger+Buchdruckereibesitzer+1825-1925+%3B+Gedenkschrift+zur+100.+Wiederkehr+d.+Gru%CC%88ndungstages%2C+verf.+im+Auftr.+d.+Vorstandes+d.+Buchdrucker-Innung+%28Freie+Innung%29+zu+Hamburg%1FhFriedrich+Voeltzer%1E028A+%1F9101386281%1F7Tp1%1FVpiz%1FAgnd%1F0101386281%1FE1895%1FaVo%CC%88ltzer%1FdFriedrich%1E033A+%1FpHamburg%1FnBuchdrucker-Innung+%28Freie+Innung%29%1E033A+%1FpHamburg%1Fn%5BVerlagsbuchh.+Broschek+%26+Co.%5D%1E034D+%1Fa44+S.%1E034I+%1Fa4%1E%0A001@+%1Fa5%1F01-3%1E001A+%1F01240%3A01-08-95%1E001B+%1F09999%3A24-09-10%1Ft17%3A42%3A20.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Af%1E003@+%1F0945184085%1E004A+%1F03-89007-044-2%1FfGewebe+%3A+DM+198.00%2C+sfr+198.00%2C+S+1386.00%1E006T+%1F095%2CN35%2C0856%1E006U+%1F095%2CA48%2C1186%1E006U+%1F010%2CP01%1E007I+%1FSo%1F061975997%1E011@+%1Fa1995%1E017A+%1Fara%1E021A+%1Fx213%1F9550711899%1FYNeues+Handbuch+der+Musikwissenschaft%1Fhhrsg.+von+Carl+Dahlhaus.+Fortgef.+von+Hermann+Danuser%1FpLaaber%1FJLaaber-Verl.%1FS48%1F03-89007-030-2%1FgAc%1E021B+%1FlBd.+13.%1FaRegister%1Fhzsgest.+von+Hans-Joachim+Hinrichsen%1E028C+%1F9121445453%1F7Tp3%1FVpiz%1FAgnd%1F0121445453%1FE1952%1FaHinrichsen%1FdHans-Joachim%1E034D+%1FaVIII%2C+408+S.%1E045V+%1F9090001001%1E047A+%1FSFE%1Fagb/fm%1E%0A001@+%1F01-2%1Fa5%1E001A+%1F01239%3A18-08-11%1E001B+%1F09999%3A05-09-11%1Ft23%3A31%3A44.000%1E001D+%1F01240%3A30-08-11%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Af%1E003@+%1F01014417392%1E004A+%1Ffkart.%1E006U+%1F011%2CA37%1E007G+%1FaDNB%1F01014417392%1E007I+%1FSo%1F0752937239%1E010@+%1Fager%1E011@+%1Fa2011%1E017A+%1Fara%1Fasf%1E021A+%1Fxtr%1F91014809657%1F7Tp3%1FVpiz%1FAgnd%1F01034622773%1FE1958%1FaLu%CC%88beck%1FdMonika%1FYPersonalwirtschaft+mit+DATEV%1FhMonika+Lu%CC%88beck+%3B+Helmut+Lu%CC%88beck%1FpBodenheim%1FpWien%1FJHerdt%1FRXA-DE%1FS650%1FgAc%1E021B+%1FlTrainerbd.%1E032@+%1Fg11%1Fa1.+Ausg.%1E034D+%1Fa129+S.%1E034M+%1FaIll.%1E047A+%1FSFE%1Famar%1E047A+%1FSERW%1Fasal%1E047I+%1Fu%24%1Fc04%1FdDNB%1Fe1%1E) +1) [Here a way to show pathways in combination with values.](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-lines%0A%7C+decode-pica%0A%7C+fix%28%22nothing%28%29%22%2C+repeatedFieldsToEntities+%3D+%22true%22%29%0A%7C+flatten%0A%7C+encode-literals%0A%7C+print%0A%3B&data=001@+%1Fa5%1F01-2%1E001A+%1F01100%3A15-10-94%1E001B+%1F09999%3A12-06-06%1Ft16%3A10%3A17.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aag%1E003@+%1F0482147350%1E006U+%1F094%2CP05%1E007E+%1F0U+70.16407%1E007I+%1FSo%1F074057548%1E011@+%1Fa1970%1E017A+%1Farh%1E021A+%1FaDie+@Berufsfreiheit+der+Arbeitnehmer+und+ihre+Ausgestaltung+in+vo%CC%88lkerrechtlichen+Vertra%CC%88gen%1FdEine+Grundrechtsbetrachtg%1E028A+%1F9106884905%1F7Tn3%1FAgnd%1F0106884905%1FaProjahn%1FdHorst+D.%1E033A+%1FpWu%CC%88rzburg%1E034D+%1FaXXXVIII%2C+165+S.%1E034I+%1Fa8%1E037C+%1FaWu%CC%88rzburg%2C+Jur.+F.%2C+Diss.+v.+7.+Aug.+1970%1E%0A001@+%1F01%1Fa5%1E001A+%1F01140%3A08-12-99%1E001B+%1F09999%3A05-01-08%1Ft22%3A57%3A29.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aa%1E003@+%1F0958090564%1E004A+%1Ffkart.+%3A+DM+9.70%2C+EUR+4.94%2C+sfr+8.00%2C+S+68.00%1E006U+%1F000%2CB05%2C0285%1E007I+%1FSo%1F076088278%1E011@+%1Fa1999%1E017A+%1Farb%1Fasi%1E019@+%1FaXA-AT%1E021A+%1FaZukunft+Bildung%1FhPolitische+Akademie.+%5BHrsg.+von+Gu%CC%88nther+R.+Burkert-Dottolo+und+Bernhard+Moser%5D%1E028C+%1F9130681849%1F7Tp1%1FVpiz%1FAgnd%1F0130681849%1FE1952%1FaBurkert%1FdGu%CC%88nther+R.%1FBHrsg.%1E033A+%1FpWien%1FnPolit.+Akad.%1E034D+%1Fa79+S.%1E034I+%1Fa24+cm%1E036F+%1Fx299+12%1F9551720077%1FgAdn%1F7Tb1%1FAgnd%1F01040469-7%1FaPolitische+Akademie%1FgWien%1FYPA-Information%1FhPolitische+Akademie%2C+WB%1FpWien%1FJPolitische+Akad.%2C+WB%1Fl99%2C2%1E036F/01+%1Fx12%1F9025841467%1FgAdvz%1Fi2142105-5%1FYAktuelle+Fragen+der+Politik%1FhPolitische+Akademie%1FpWien%1FJPolitische+Akad.+der+O%CC%88VP%1FlBd.+2%1E045E+%1Fa22%1Fd18%1Fm370%1E047A+%1FSFE%1Fata%1E%0A001@+%1Fa5%1F01%1E001A+%1F01140%3A19-02-03%1E001B+%1F09999%3A19-06-11%1Ft01%3A20%3A13.000%1E001D+%1F09999%3A26-04-03%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aal%1E003@+%1F0361809549%1E004A+%1FfHlw.%1E006U+%1F000%2CL01%1E006U+%1F004%2CP01-s-41%1E006U+%1F004%2CP01-f-21%1E007G+%1FaDNB%1F0361809549%1E007I+%1FSo%1F072658383%1E007M+%1F04413/0275%1E011@+%1Fa1925%1E019@+%1FaXA-DXDE%1FaXA-DE%1E021A+%1FaHundert+Jahre+Buchdrucker-Innung+Hamburg%1FdWesen+u.+Werden+d.+Vereinigungen+Hamburger+Buchdruckereibesitzer+1825-1925+%3B+Gedenkschrift+zur+100.+Wiederkehr+d.+Gru%CC%88ndungstages%2C+verf.+im+Auftr.+d.+Vorstandes+d.+Buchdrucker-Innung+%28Freie+Innung%29+zu+Hamburg%1FhFriedrich+Voeltzer%1E028A+%1F9101386281%1F7Tp1%1FVpiz%1FAgnd%1F0101386281%1FE1895%1FaVo%CC%88ltzer%1FdFriedrich%1E033A+%1FpHamburg%1FnBuchdrucker-Innung+%28Freie+Innung%29%1E033A+%1FpHamburg%1Fn%5BVerlagsbuchh.+Broschek+%26+Co.%5D%1E034D+%1Fa44+S.%1E034I+%1Fa4%1E%0A001@+%1Fa5%1F01-3%1E001A+%1F01240%3A01-08-95%1E001B+%1F09999%3A24-09-10%1Ft17%3A42%3A20.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Af%1E003@+%1F0945184085%1E004A+%1F03-89007-044-2%1FfGewebe+%3A+DM+198.00%2C+sfr+198.00%2C+S+1386.00%1E006T+%1F095%2CN35%2C0856%1E006U+%1F095%2CA48%2C1186%1E006U+%1F010%2CP01%1E007I+%1FSo%1F061975997%1E011@+%1Fa1995%1E017A+%1Fara%1E021A+%1Fx213%1F9550711899%1FYNeues+Handbuch+der+Musikwissenschaft%1Fhhrsg.+von+Carl+Dahlhaus.+Fortgef.+von+Hermann+Danuser%1FpLaaber%1FJLaaber-Verl.%1FS48%1F03-89007-030-2%1FgAc%1E021B+%1FlBd.+13.%1FaRegister%1Fhzsgest.+von+Hans-Joachim+Hinrichsen%1E028C+%1F9121445453%1F7Tp3%1FVpiz%1FAgnd%1F0121445453%1FE1952%1FaHinrichsen%1FdHans-Joachim%1E034D+%1FaVIII%2C+408+S.%1E045V+%1F9090001001%1E047A+%1FSFE%1Fagb/fm%1E%0A001@+%1F01-2%1Fa5%1E001A+%1F01239%3A18-08-11%1E001B+%1F09999%3A05-09-11%1Ft23%3A31%3A44.000%1E001D+%1F01240%3A30-08-11%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Af%1E003@+%1F01014417392%1E004A+%1Ffkart.%1E006U+%1F011%2CA37%1E007G+%1FaDNB%1F01014417392%1E007I+%1FSo%1F0752937239%1E010@+%1Fager%1E011@+%1Fa2011%1E017A+%1Fara%1Fasf%1E021A+%1Fxtr%1F91014809657%1F7Tp3%1FVpiz%1FAgnd%1F01034622773%1FE1958%1FaLu%CC%88beck%1FdMonika%1FYPersonalwirtschaft+mit+DATEV%1FhMonika+Lu%CC%88beck+%3B+Helmut+Lu%CC%88beck%1FpBodenheim%1FpWien%1FJHerdt%1FRXA-DE%1FS650%1FgAc%1E021B+%1FlTrainerbd.%1E032@+%1Fg11%1Fa1.+Ausg.%1E034D+%1Fa129+S.%1E034M+%1FaIll.%1E047A+%1FSFE%1Famar%1E047A+%1FSERW%1Fasal%1E047I+%1Fu%24%1Fc04%1FdDNB%1Fe1%1E) -Other ways are also possible, too. +2) [Here is a way to collect and count all paths in all records by using the `list-fix-paths` command.](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-lines%0A%7C+decode-pica%0A%7C+list-fix-paths%0A%7C+print%0A%3B&data=001@+%1Fa5%1F01-2%1E001A+%1F01100%3A15-10-94%1E001B+%1F09999%3A12-06-06%1Ft16%3A10%3A17.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aag%1E003@+%1F0482147350%1E006U+%1F094%2CP05%1E007E+%1F0U+70.16407%1E007I+%1FSo%1F074057548%1E011@+%1Fa1970%1E017A+%1Farh%1E021A+%1FaDie+@Berufsfreiheit+der+Arbeitnehmer+und+ihre+Ausgestaltung+in+vo%CC%88lkerrechtlichen+Vertra%CC%88gen%1FdEine+Grundrechtsbetrachtg%1E028A+%1F9106884905%1F7Tn3%1FAgnd%1F0106884905%1FaProjahn%1FdHorst+D.%1E033A+%1FpWu%CC%88rzburg%1E034D+%1FaXXXVIII%2C+165+S.%1E034I+%1Fa8%1E037C+%1FaWu%CC%88rzburg%2C+Jur.+F.%2C+Diss.+v.+7.+Aug.+1970%1E%0A001@+%1F01%1Fa5%1E001A+%1F01140%3A08-12-99%1E001B+%1F09999%3A05-01-08%1Ft22%3A57%3A29.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aa%1E003@+%1F0958090564%1E004A+%1Ffkart.+%3A+DM+9.70%2C+EUR+4.94%2C+sfr+8.00%2C+S+68.00%1E006U+%1F000%2CB05%2C0285%1E007I+%1FSo%1F076088278%1E011@+%1Fa1999%1E017A+%1Farb%1Fasi%1E019@+%1FaXA-AT%1E021A+%1FaZukunft+Bildung%1FhPolitische+Akademie.+%5BHrsg.+von+Gu%CC%88nther+R.+Burkert-Dottolo+und+Bernhard+Moser%5D%1E028C+%1F9130681849%1F7Tp1%1FVpiz%1FAgnd%1F0130681849%1FE1952%1FaBurkert%1FdGu%CC%88nther+R.%1FBHrsg.%1E033A+%1FpWien%1FnPolit.+Akad.%1E034D+%1Fa79+S.%1E034I+%1Fa24+cm%1E036F+%1Fx299+12%1F9551720077%1FgAdn%1F7Tb1%1FAgnd%1F01040469-7%1FaPolitische+Akademie%1FgWien%1FYPA-Information%1FhPolitische+Akademie%2C+WB%1FpWien%1FJPolitische+Akad.%2C+WB%1Fl99%2C2%1E036F/01+%1Fx12%1F9025841467%1FgAdvz%1Fi2142105-5%1FYAktuelle+Fragen+der+Politik%1FhPolitische+Akademie%1FpWien%1FJPolitische+Akad.+der+O%CC%88VP%1FlBd.+2%1E045E+%1Fa22%1Fd18%1Fm370%1E047A+%1FSFE%1Fata%1E%0A001@+%1Fa5%1F01%1E001A+%1F01140%3A19-02-03%1E001B+%1F09999%3A19-06-11%1Ft01%3A20%3A13.000%1E001D+%1F09999%3A26-04-03%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Aal%1E003@+%1F0361809549%1E004A+%1FfHlw.%1E006U+%1F000%2CL01%1E006U+%1F004%2CP01-s-41%1E006U+%1F004%2CP01-f-21%1E007G+%1FaDNB%1F0361809549%1E007I+%1FSo%1F072658383%1E007M+%1F04413/0275%1E011@+%1Fa1925%1E019@+%1FaXA-DXDE%1FaXA-DE%1E021A+%1FaHundert+Jahre+Buchdrucker-Innung+Hamburg%1FdWesen+u.+Werden+d.+Vereinigungen+Hamburger+Buchdruckereibesitzer+1825-1925+%3B+Gedenkschrift+zur+100.+Wiederkehr+d.+Gru%CC%88ndungstages%2C+verf.+im+Auftr.+d.+Vorstandes+d.+Buchdrucker-Innung+%28Freie+Innung%29+zu+Hamburg%1FhFriedrich+Voeltzer%1E028A+%1F9101386281%1F7Tp1%1FVpiz%1FAgnd%1F0101386281%1FE1895%1FaVo%CC%88ltzer%1FdFriedrich%1E033A+%1FpHamburg%1FnBuchdrucker-Innung+%28Freie+Innung%29%1E033A+%1FpHamburg%1Fn%5BVerlagsbuchh.+Broschek+%26+Co.%5D%1E034D+%1Fa44+S.%1E034I+%1Fa4%1E%0A001@+%1Fa5%1F01-3%1E001A+%1F01240%3A01-08-95%1E001B+%1F09999%3A24-09-10%1Ft17%3A42%3A20.000%1E001D+%1F09999%3A99-99-99%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Af%1E003@+%1F0945184085%1E004A+%1F03-89007-044-2%1FfGewebe+%3A+DM+198.00%2C+sfr+198.00%2C+S+1386.00%1E006T+%1F095%2CN35%2C0856%1E006U+%1F095%2CA48%2C1186%1E006U+%1F010%2CP01%1E007I+%1FSo%1F061975997%1E011@+%1Fa1995%1E017A+%1Fara%1E021A+%1Fx213%1F9550711899%1FYNeues+Handbuch+der+Musikwissenschaft%1Fhhrsg.+von+Carl+Dahlhaus.+Fortgef.+von+Hermann+Danuser%1FpLaaber%1FJLaaber-Verl.%1FS48%1F03-89007-030-2%1FgAc%1E021B+%1FlBd.+13.%1FaRegister%1Fhzsgest.+von+Hans-Joachim+Hinrichsen%1E028C+%1F9121445453%1F7Tp3%1FVpiz%1FAgnd%1F0121445453%1FE1952%1FaHinrichsen%1FdHans-Joachim%1E034D+%1FaVIII%2C+408+S.%1E045V+%1F9090001001%1E047A+%1FSFE%1Fagb/fm%1E%0A001@+%1F01-2%1Fa5%1E001A+%1F01239%3A18-08-11%1E001B+%1F09999%3A05-09-11%1Ft23%3A31%3A44.000%1E001D+%1F01240%3A30-08-11%1E001U+%1F0utf8%1E001X+%1F00%1E002@+%1F0Af%1E003@+%1F01014417392%1E004A+%1Ffkart.%1E006U+%1F011%2CA37%1E007G+%1FaDNB%1F01014417392%1E007I+%1FSo%1F0752937239%1E010@+%1Fager%1E011@+%1Fa2011%1E017A+%1Fara%1Fasf%1E021A+%1Fxtr%1F91014809657%1F7Tp3%1FVpiz%1FAgnd%1F01034622773%1FE1958%1FaLu%CC%88beck%1FdMonika%1FYPersonalwirtschaft+mit+DATEV%1FhMonika+Lu%CC%88beck+%3B+Helmut+Lu%CC%88beck%1FpBodenheim%1FpWien%1FJHerdt%1FRXA-DE%1FS650%1FgAc%1E021B+%1FlTrainerbd.%1E032@+%1Fg11%1Fa1.+Ausg.%1E034D+%1Fa129+S.%1E034M+%1FaIll.%1E047A+%1FSFE%1Famar%1E047A+%1FSERW%1Fasal%1E047I+%1Fu%24%1Fc04%1FdDNB%1Fe1%1E) ## Bonus: XML in MF and their paths From 896ca6e16d04048fb13ca1d93a6ae350b6ac7f55 Mon Sep 17 00:00:00 2001 From: Phu Tu Date: Thu, 13 Mar 2025 13:19:43 +0100 Subject: [PATCH 06/10] Update 01_Getting_Metafacture.md Co-authored-by: TobiasNx <61879957+TobiasNx@users.noreply.github.com> --- 01_Getting_Metafacture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/01_Getting_Metafacture.md b/01_Getting_Metafacture.md index d317809..5e20080 100644 --- a/01_Getting_Metafacture.md +++ b/01_Getting_Metafacture.md @@ -17,7 +17,7 @@ It is useful to test, share and export Metafacture workflows. Starting with [Chapter 6](./06_MetafactureCLI.md) we can switch from using Playground to running Metafacture on our own hardware. But the examples are still provided in the Playground. -To run Metafacture on your local machine you need a Linux/Unix Bash Shell (part of every Linux, MacOS and Windows >=10) with Metafacture Core installed. In this course we are not teaching you how to use the command line. For that see: **TODO** +To run Metafacture on your local machine you need a Linux/Unix Bash Shell (part of every Linux, MacOS and Windows >=10) with Metafacture Core installed. In this course we are not teaching you how to use the command line. For that see: [Chapter 6](./06_MetafactureCLI.md) **Next lesson**: [02 Introduction into Metafacture Flux](./02_Introduction_into_Metafacture-Flux.md) From 86eb9462214e46a4dd75fc18a92612dfc5192936 Mon Sep 17 00:00:00 2001 From: Phu Tu Date: Thu, 13 Mar 2025 13:20:07 +0100 Subject: [PATCH 07/10] Update 04_Fix-Path.md Co-authored-by: TobiasNx <61879957+TobiasNx@users.noreply.github.com> --- 04_Fix-Path.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/04_Fix-Path.md b/04_Fix-Path.md index d7d261e..17acff7 100644 --- a/04_Fix-Path.md +++ b/04_Fix-Path.md @@ -46,7 +46,7 @@ source_records: - "promise:bwb_daily_pallets_2023-05-10:W8-BRV-242" ``` -Metafacture Fix is using FixPath to access and selectively extract data out of YAML and JSON documents. It is JSONPath like but not identical. It also uses the dot notation but there are some differences with the path structure of arrays and repeated fields. +Metafacture Fix is using FixPath to access and selectively extract data out of (semi) structured documents. It is JSONPath like but not identical. It also uses the dot notation but there are some differences with the path structure of arrays and repeated fields. You can point to every part of the YAML file using a dot-notation. For simple top level fields the path is just the name of the field: From 2f2be7bc688468e4214eebe642233a37550fec32 Mon Sep 17 00:00:00 2001 From: Phu Tu Date: Thu, 13 Mar 2025 15:35:20 +0100 Subject: [PATCH 08/10] Remove code marker from names --- 03_Introduction_into_Metafacture-Fix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/03_Introduction_into_Metafacture-Fix.md b/03_Introduction_into_Metafacture-Fix.md index ccf9ad1..f5ac10f 100644 --- a/03_Introduction_into_Metafacture-Fix.md +++ b/03_Introduction_into_Metafacture-Fix.md @@ -3,7 +3,7 @@ In the last session we learned about Flux moduls. Flux moduls can do a lot of things. They configure the "high-level" transformation pipeline. -But the main transformation of incoming data at record, elemenet and value level is usually done by the transformation moduls `Fix` or `Morph` as one step in the pipeline. +But the main transformation of incoming data at record, elemenet and value level is usually done by the transformation moduls Fix or Morph as one step in the pipeline. By transformation we mean things like: From 2d383b17754bb6fcbb9595439f4123114a7f71ba Mon Sep 17 00:00:00 2001 From: Phu Tu Date: Thu, 13 Mar 2025 15:35:36 +0100 Subject: [PATCH 09/10] Revision for lesson 5 --- 05-More-Fix-Concepts.md | 69 ++++++++++++++++++----------------------- 1 file changed, 31 insertions(+), 38 deletions(-) diff --git a/05-More-Fix-Concepts.md b/05-More-Fix-Concepts.md index 6c18837..978798d 100644 --- a/05-More-Fix-Concepts.md +++ b/05-More-Fix-Concepts.md @@ -1,21 +1,20 @@ # Lesson 5: More Fix concepts -We already learned about simple Fixes (or Fix Functions) but there are three additional concepts in Fix selector, conditionals and binds. -These Fix concepts were introduced by Catmandu (see [functions](https://librecat.org/Catmandu/#functions), [selector](https://librecat.org/Catmandu/#selectors), [conditionals](https://librecat.org/Catmandu/#conditionals) and [binds](https://librecat.org/Catmandu/#binds)). But be aware Metafacture Fix does not support all of the specific functions, selectors, conditionals and binds from Catmandu. Check the documentation for an full overview of the supported [fix functions](https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#functions). +We already learned about simple Fixes aka *[Fix functions](https://metafacture.github.io/metafacture-documentation/docs/fix/Fix-functions.html)* but there are three additional concepts in Fix: selector, conditionals and binds. + +These Fix concepts were introduced by Catmandu (see [functions](https://librecat.org/Catmandu/#functions), [selector](https://librecat.org/Catmandu/#selectors), [conditionals](https://librecat.org/Catmandu/#conditionals) and [binds](https://librecat.org/Catmandu/#binds)). But be aware that Metafacture Fix does not support all of the specific functions, selectors, conditionals and binds from Catmandu. Check the documentation for a full overview of the supported [Fix functions](https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#functions). ## Additional concepts -The following code snippet shows examples of eachs of these concepts: +The following code snippet shows examples of each of these concepts: ```PERL -# Simple fix function - +# Simple Fix function add_field("hello", "world") remove_field("my.deep.nested.junk") copy_field("stats", "output.$append") # Conditionals - if exists("error") set_field("is_valid", "no") log("error") @@ -26,27 +25,26 @@ else set_field("is_valid", "yes") end -# Binds - Loops - -do list(path: "foo", "var": "$i") - add_field("$i.bar", "baz") -end - # Selector if exists("error") reject() end + +# Binds - Loops +do list(path: "foo", "var": "$i") + add_field("$i.bar", "baz") +end ``` -We already got to know simple fixes aka *[Functions](https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#functions)*. They are used to add, change, remove or otherwise manipulate elements. Simple fixes were introduced in [Lesson 3](./03_Introduction_into_Metafacture-Fix.md). +Fix functions are used to add, change, remove or otherwise manipulate elements. They were introduced in [Lesson 3](./03_Introduction_into_Metafacture-Fix.html). The other three concepts help when you intend to use more complex transformations: -*[Conditionals](https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#conditionals)* are used to control the processing of fix functions. The included fix functions are not process with every workflow but only under certain conditions. +*[Conditionals](https://metafacture.github.io/metafacture-documentation/docs/fix/Fix-functions.html#conditionals)* are used to control the processing of fix functions. The included fix functions are not process with every workflow but only under certain conditions. -*[Selectors](https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#selectors)* can be used to filter the records you want. +*[Selectors](https://metafacture.github.io/metafacture-documentation/docs/fix/Fix-functions.html#selectors)* can be used to filter the records you want. -*[Binds](https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#binds)* are wrappers for one or more fixes. They give extra control functionality for fixes such as loops. All binds have the same syntax: +*[Binds](https://metafacture.github.io/metafacture-documentation/docs/fix/Fix-functions.html#binds)* are wrappers for one or more fixes. They give extra control functionality for fixes such as loops. All binds have the same syntax: ```PERL do Bind(params,…) @@ -59,7 +57,7 @@ end Conditionals are a common concept in programming and scripting languages. They control processes, in our scenario: transformations, with regard to specific requirements. -e.g. [we have records some of them are of type book](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=add_field%28%22type%22%2C%22BibliographicResource%22%29&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22): +For example, [we have records some of them are of type book](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=add_field%28%22type%22%2C%22BibliographicResource%22%29&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22): ```YAML --- @@ -95,13 +93,13 @@ director: "Hayo Freitag" language: "ger" ``` -If we want to add an element e.g. type for every record. Then we simply use the fix function `add_field("type","BibliographicResource")` +If we want to add an element `type` for **every** record then we simply use the Fix function `add_field("type","BibliographicResource")` -But if you want to add more specific type depending on the type, then we use contitionals: +But if you want to add more specific type depending on the medium then we use contitionals. -e.g.: +### if -only `add_field("type","BibliographicResource")` [if the medium contains Book](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=if+all_contain%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22BibliographicResource%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22): +only `add_field("type","BibliographicResource")` [if medium **contains** Book](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=if+all_contain%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22BibliographicResource%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22): ```PERL if all_contain("medium","Book") @@ -109,24 +107,17 @@ if all_contain("medium","Book") end ``` -or [only if medium is exact](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=if+all_equal%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22BibliographicResource%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22): +or [if medium is **exact** Book](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=if+all_equal%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22BibliographicResource%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22): ```PERL if all_equal("medium","Book") add_field("type","BibliographicResource") end -``` +``` +### unless -Conditionals usually have the same syntax pattern: - -```PERL -if conditional("Parameter1","Parameter2") - fix-functions -end -``` - -[If you want to add transformations for all that do not meet the condition with `unless`:](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=unless+all_equal%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22Other%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22) +Use `unless` [to add transformations for all that do **not** meet the condition:](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=unless+all_equal%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22Other%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22) ```PERL unless all_equal("medium","Book") @@ -134,6 +125,8 @@ unless all_equal("medium","Book") end ``` +### if...else + [You can also use conditionals `if` they meet the requirements and handle all others differently with `else`:](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=if+all_equal%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22BibliographicResource%22%29%0Aelse%0A++++add_field%28%22type%22%2C%22Other%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22) ```PERL @@ -144,7 +137,9 @@ else end ``` -[You can also use conditionals `if` they meet the requirements and handle all others differently with an additionl conditionals with `elsif` and the rest with `else`:](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=if+all_equal%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22BibliographicResource%22%29%0Aelsif+all_contain%28%22medium%22%2C%22Audio%22%29%0A++++add_field%28%22type%22%2C%22AudioResource%22%29%0Aelsif+all_match%28%22medium%22%2C%22.%2AMovie.%2A%22%29%0A++++add_field%28%22type%22%2C%22AudioVisualResource%22%29%0Aelse%0A++++add_field%28%22type%22%2C%22Other%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22) +### if...elseif...else + +[You can also use conditionals `if` they meet the requirements and handle all others differently with an additional conditionals with `elsif` and the rest with `else`:](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=if+all_equal%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22BibliographicResource%22%29%0Aelsif+all_contain%28%22medium%22%2C%22Audio%22%29%0A++++add_field%28%22type%22%2C%22AudioResource%22%29%0Aelsif+all_match%28%22medium%22%2C%22.%2AMovie.%2A%22%29%0A++++add_field%28%22type%22%2C%22AudioVisualResource%22%29%0Aelse%0A++++add_field%28%22type%22%2C%22Other%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22) ```PERL if all_equal("medium","Book") @@ -158,9 +153,7 @@ else end ``` -Metafacture supports lots of conditionals, find a list of all of them [here](https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#conditionals). - -Hint: Some conditionals have variations with `all_`, `any_` or `none` while they behave in the same way if you process them on simple string-elements. They also can be used with arrays/lists then the conditional has different out-come depending on the fact that all (`all_`) or at least one (`any_`) value of an array matches the requierement. `none` checks if the conditionally does not match. +Metafacture supports lots of conditionals, find a list of all of them [here](https://metafacture.github.io/metafacture-documentation/docs/fix/Fix-functions.html#conditionals). ## Selectors @@ -176,11 +169,11 @@ end Selectors work in combination with conditionals to define the conditions that you want to kick out. -For the supported selectors see: https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#selectors +For the supported selectors see: https://metafacture.github.io/metafacture-documentation/docs/fix/Fix-functions.html#selectors ## Binds -As mentioned above [Binds](https://github.com/metafacture/metafacture-documentation/blob/master/Fix-function-and-Cookbook.md#binds)* are wrappers for one or more fixes. They give extra control functionality for fixes such as loops. All binds have the same syntax: +As mentioned above [Binds](https://metafacture.github.io/metafacture-documentation/docs/fix/Fix-functions.html#binds) are wrappers for one or more fixes. They give extra control functionality for fixes such as loops. All binds have the same syntax: ```PERL do Bind(params,…) From 6488b280ebf0b502e1380cd38877ff4f8ada6e8c Mon Sep 17 00:00:00 2001 From: TobiasNx Date: Thu, 20 Mar 2025 11:03:10 +0100 Subject: [PATCH 10/10] Adjustments after phus review --- 02_Introduction_into_Metafacture-Flux.md | 18 ++++++++--------- 03_Introduction_into_Metafacture-Fix.md | 25 +++++++++++++----------- 04_Fix-Path.md | 8 ++++---- 05-More-Fix-Concepts.md | 12 ++++++------ 4 files changed, 33 insertions(+), 30 deletions(-) diff --git a/02_Introduction_into_Metafacture-Flux.md b/02_Introduction_into_Metafacture-Flux.md index bc68079..79f87cc 100644 --- a/02_Introduction_into_Metafacture-Flux.md +++ b/02_Introduction_into_Metafacture-Flux.md @@ -1,6 +1,6 @@ # Lesson 2: Introduction into Metafacture Flux -To perform data processing with Metafacture transformation workflows are configured with **Metafacture Flux**, a domain-specific scripting language (DSL). +To perform data processing with Metafacture transformation workflows are configured with **Metafacture Flux**, a [domain-specific scripting language (DSL)](https://en.wikipedia.org/wiki/Domain-specific_language). With Metafacture Flux we combine different modules for reading, opening, transforming, and writing data sets. In this lesson we will learn about Metafacture Flux, what Flux workflows are and how to combine different Flux modules to create a workflow in order to process datasets. @@ -9,9 +9,9 @@ In this lesson we will learn about Metafacture Flux, what Flux workflows are and To process data Metafacture can be used with the command line, as JAVA library or you can use the Metafacture Playground. -For this introduction we will start with the Playground since it allows a quick start without additional installing. The [Metafacture Playground](https://metafacture.org/playground) is a web interface to test and share Metafacture workflows. The commandline handling will be subject in [lesson 6](./06_MetafactureCLI.md) +For this introduction we will start with the Playground since it allows a quick start without additional installing. The [Metafacture Playground](https://metafacture.org/playground) is a web interface to test and share Metafacture workflows. The commandline handling will be subject in [lesson 6](./06_MetafactureCLI.md) -In this tutorial we are going to process *structured information*. We call data structured when it organised in such a way that is easy processable by computers. Literary text documents like *War and Peace* are structured only in words and sentences, but a computer doesn’t know which words are part of the title or which words contain names. We had to tell the computer that. Today we will download a weather report in a structured format called JSON and inspect it with Metafacture. +In this tutorial we are going to process *structured information*. We call data structured when it organised in such a way that is easy processable by computers. Literary text documents like *War and Peace* are structured only in words and sentences, but a computer doesn’t know which words are part of the title or which words contain names. We had to tell the computer that. Today we will download a book record in a structured format called JSON and inspect it with Metafacture. ## Flux Workflows @@ -56,7 +56,7 @@ But the result is the same if you process the flux. Often you want to process data stored in a file. -The playground has an input area called `ìnputFile-content`. In this text area you can insert data that you have usually stored in a file. The variable `inputFile` can be used at the beginning of the workflow and it refers to the input file. +The playground has an input area called `ìnputFile-content`. In this text area you can insert data that you have usually stored in a file. The variable `inputFile` can be used at the beginning of the workflow and it refers to the input file represented by the `ìnputFile-content`-area. e.g. @@ -65,7 +65,7 @@ e.g. So lets use `inputFile` instead of `INPUT` and copy the value of the text string in the Data field above the Flux. -Data: +Data for `inputFile-content`: `Hello, friend. I'am Metafacture!` @@ -81,7 +81,7 @@ Oops... There seems to be unusual output. Its a file path. Why? Because the variable `inputFile` refers to a file (path). To read the content of the file we need to handle the incoming file path differently. -(You will learn how to process files on your computer in lesson 06 when we show how to run metafacture on the command line.) +(You will learn how to process files on your computer in lesson 06 when we show how to run metafacture on the command line on your computer.) We need to add two additional Metafacture commands: `open-file` and `as-lines` @@ -215,13 +215,13 @@ last_modified: This is better readable, right? -But we cannot only open the data we have in our `ìnputFile-content` field, we also can open stuff on the web: +But we cannot only open the data we have in our `inputFile-content` field, we also can open stuff on the web: Instead of using `inputFile` lets read the book data which is provided by the URL from above: Clear your playground and copy the following Flux workflow: -``` +```default "https://openlibrary.org/books/OL2838758M.json" | open-http | as-lines @@ -259,7 +259,7 @@ Now take some time and play around a little bit more and use some other modules. Click to see the new workflow -``` +```default "https://openlibrary.org/books/OL2838758M.json" | open-http | as-lines diff --git a/03_Introduction_into_Metafacture-Fix.md b/03_Introduction_into_Metafacture-Fix.md index f5ac10f..260456a 100644 --- a/03_Introduction_into_Metafacture-Fix.md +++ b/03_Introduction_into_Metafacture-Fix.md @@ -3,7 +3,7 @@ In the last session we learned about Flux moduls. Flux moduls can do a lot of things. They configure the "high-level" transformation pipeline. -But the main transformation of incoming data at record, elemenet and value level is usually done by the transformation moduls Fix or Morph as one step in the pipeline. +But the main transformation of incoming data at record, elemenet and value level is usually done by the transformation moduls [Fix](https://metafacture.github.io/metafacture-documentation/docs/flux/flux-commands.html#fix) or [Morph](https://metafacture.github.io/metafacture-documentation/docs/flux/flux-commands.html#morph) as one step in the pipeline. By transformation we mean things like: @@ -13,7 +13,7 @@ By transformation we mean things like: But not changing serialization that is part of encoding and decoding. -In this tutorial we focus on Fix. If you want to learn about Morph have a look at https://slides.lobid.org/metafacture-2020/#/ +In this tutorial we focus on [Fix](https://metafacture.github.io/metafacture-documentation/docs/flux/flux-commands.html#fix). If you want to learn about Morph have a look [at this presentation](https://slides.lobid.org/metafacture-2020/#/) and the [great documentation by Swiss Bib](https://sschuepbach.github.io/metamorph-hacks/). ## Metafacture Fix and Fix Functions @@ -22,7 +22,7 @@ So let's dive into Metafacture Fix and get back to the [Playground](https://meta Clear it if needed and paste the following Flux in the Flux-File area. -``` +```default "https://openlibrary.org/books/OL2838758M.json" | open-http | as-lines @@ -47,12 +47,12 @@ and single quotes in the fix functions. As we did here: `fix ("retain('title')") Now let us additionally keep the info that is given in the element `"publish_date"` and the subfield `"key"` in `'type'` by adding `'publish_date', 'type.key'` to `retain`: -``` +```default "https://openlibrary.org/books/OL2838758M.json" | open-http | as-lines | decode-json -| fix ("retain('title', 'publish_date', 'type.key')") +| fix ("retain('title', 'publish_date', 'notes.value', 'type.key')") | encode-yaml | print ; @@ -64,11 +64,10 @@ You should now see something like this: --- title: "Ordinary vices" publish_date: "1984" -type: - key: "/type/edition" +notes: + value: "Bibliography: p. 251-260.\nIncludes index." ``` -**TODO**: In diesem Beispiel macht es im Ergebnis keinen Unterscheid, ob man `type` oder `type.key` adressiert. Vielleicht ein anderes Beispiel wählen wie z. B. `identifiers`, wo mehrere Unterfelder vorhanden sind? When manipulating data you often need to create many fixes to process a data file in the format and structure you need. With a text editor you can write all fix functions in a singe separate Fix file. @@ -82,7 +81,7 @@ Like this. Fix: ```PERL -retain("title", "publish_date", "type.key") +retain("title", "publish_date", "notes.value", "type.key") ``` Using a separate Fix file is recommended if you need to write many Fix functions. It will keep the Flux workflow clear and legible. @@ -98,7 +97,7 @@ Also change the `retain` function so that you keep the new element `"pub_type"` ``` move_field("type.key","pub_type") -retain("title", "publish_date", "pub_type") +retain("title", "publish_date", "notes.value", "pub_type") ``` The output should be something like this: @@ -108,6 +107,8 @@ The output should be something like this: title: "Ordinary vices" publish_date: "1984" pub_type: "/type/edition" +notes: + value: "Bibliography: p. 251-260.\nIncludes index." ``` With `move_field` we moved and renamed an existing element. @@ -124,6 +125,8 @@ If you execute your last workflow with the "Process" button again, you should no title: "Ordinary vices" publish_date: "1984" pub_type: "edition" +notes: + value: "Bibliography: p. 251-260.\nIncludes index." ``` We cleaned up the value of `"pub_type"` element for better readability. @@ -142,7 +145,7 @@ Comments in Fix start with a hash mark `#`, while in Flux they start with `//`. Example: -``` +```PERL # Make type.key a top level element. move_field("type.key","pub_type") diff --git a/04_Fix-Path.md b/04_Fix-Path.md index 17acff7..528a385 100644 --- a/04_Fix-Path.md +++ b/04_Fix-Path.md @@ -1,12 +1,12 @@ # Lesson 4: FixPath and more complex transformations in Fix -Over the last lessons we learned how to construct a Metafacture workflow, how to use the Playground and how Metafacture Flux and Fix can be used to parse structured information. We saw how you can use Flux to transform the JSON format into the YAML format which is easier to read and contains the same information. We also learned how to retrieve information out of the JSON file using a Fix function like `retain("title", "publish_date", "type.key")`. +Over the last lessons we learned how to construct a Metafacture workflow, how to use the Playground and how Metafacture Flux and Fix can be used to parse structured information. We saw how you can use Flux to transform the JSON format into the YAML format which is easier to read and contains the same information. We also learned how to retrieve information out of the JSON file using a Fix function like `retain("title", "publish_date", "notes.value", "type.key")`. In this lesson we will go deeper into Metafacture Fix and describe how to pluck data out of structured information. First, let's fetch of a new book with the Metafacture Playground: -``` +```default "https://openlibrary.org/books/OL27333998M.json" | open-http | as-lines @@ -115,7 +115,7 @@ If you want to refer to all creators then you can use the `*` sign as a wildcard [See here](https://metafacture.org/playground/?flux=inputFile%0A%7Copen-file%0A%7Cas-records%0A%7Cdecode-yaml%0A%7Cfix%28transformationFile%29%0A%7Cencode-json%28prettyPrinting%3D%22true%22%29%0A%7Cprint%0A%3B&transformation=append%28%22creator.1%22%2C%22+Jonas%22%29%0Aappend%28%22creator.2%22%2C%22+Shaw%22%29%0Aappend%28%22creator.3%22%2C%22+Andrews%22%29%0Aprepend%28%22creator.%2A%22%2C%22Investigator+%22%29&data=---%0Acreator%3A+Justus%0Acreator%3A+Peter%0Acreator%3A+Bob%0A) -### Working with arrays +### Working with JSON and YAML arrays In JSON or YAML element repetion is possible but unusual. Instead of repeating elements an element can have a list or array of values. @@ -166,7 +166,7 @@ So, the path of the `red` would be: `my.colors[].2` And the path for `Peter` would be `characters[].2.name` -Also if you want to generate an array in the target format, then you need to add `[]` at the end of an list element like `newArray[]`. +Also if you want to generate an array in the target format JSON or YAML, then you need to add `[]` at the end of an list element like `newArray[]`. ## Excercise: diff --git a/05-More-Fix-Concepts.md b/05-More-Fix-Concepts.md index 978798d..20d48fd 100644 --- a/05-More-Fix-Concepts.md +++ b/05-More-Fix-Concepts.md @@ -127,19 +127,19 @@ end ### if...else -[You can also use conditionals `if` they meet the requirements and handle all others differently with `else`:](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=if+all_equal%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22BibliographicResource%22%29%0Aelse%0A++++add_field%28%22type%22%2C%22Other%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22) +[You can add an `else`-block to any `if` conditional if you want to process fixes only if the contition is `falsy`:](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=if+all_equal%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22BibliographicResource%22%29%0Aelse%0A++++add_field%28%22type%22%2C%22Other%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22) ```PERL if all_equal("medium","Book") add_field("type","BibliographicResource") -else +else # if previous condition is false. add_field("type","Other") end ``` -### if...elseif...else +### if...elsif(...else) -[You can also use conditionals `if` they meet the requirements and handle all others differently with an additional conditionals with `elsif` and the rest with `else`:](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=if+all_equal%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22BibliographicResource%22%29%0Aelsif+all_contain%28%22medium%22%2C%22Audio%22%29%0A++++add_field%28%22type%22%2C%22AudioResource%22%29%0Aelsif+all_match%28%22medium%22%2C%22.%2AMovie.%2A%22%29%0A++++add_field%28%22type%22%2C%22AudioVisualResource%22%29%0Aelse%0A++++add_field%28%22type%22%2C%22Other%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22) +[You can also use additional `elsif`-blocks in as part of an `if`-conditional if you want to process data if the previous contitional is `falsy` but add a condition when the defined transformations should be processed:](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=if+all_equal%28%22medium%22%2C%22Book%22%29%0A++++add_field%28%22type%22%2C%22BibliographicResource%22%29%0Aelsif+all_contain%28%22medium%22%2C%22Audio%22%29%0A++++add_field%28%22type%22%2C%22AudioResource%22%29%0Aelsif+all_match%28%22medium%22%2C%22.%2AMovie.%2A%22%29%0A++++add_field%28%22type%22%2C%22AudioVisualResource%22%29%0Aelse%0A++++add_field%28%22type%22%2C%22Other%22%29%0Aend&data=---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22eBook%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22Die+13%C2%BD+Leben+des+K%C3%A4pt%E2%80%99n+Blaub%C3%A4r%22%0Amedium%3A+%22Book%22%0Aauthor%3A+%22Walter+Moers%22%0Alanguage%3A+%22ger%22%0A%0A---%0Aname%3A+%22The+13+1/2+lives+of+Captain+Bluebear%22%0Amedium%3A+%22Audio+Book%22%0Aauthor%3A+%22Walter+Moers%22%0Anarrator%3A+%22Bronson+Pinchot%22%0Alanguage%3A+%22eng%22%0A%0A---%0Aname%3A+%22K%C3%A4pt%27n+Blaub%C3%A4r+-+Der+Film%22%0Amedium%3A+%22Movie%22%0Aauthor%3A+%22Walter+Moers%22%0Adirector%3A+%22Hayo+Freitag%22%0Alanguage%3A+%22ger%22) ```PERL if all_equal("medium","Book") @@ -244,8 +244,8 @@ end [See this example here in the playground.](https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-yaml%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=do+list%28path%3A%22colours%5B%5D%22%2C%22var%22%3A%22%24i%22%29%0A++++if+any_equal%28%22%24i%22%2C%22green%22%29%0A++++++++add_array%28%22result%5B%5D%22%29+%23+To+create+a+new+array+named+result%0A++++++++upcase%28%22%24i%22%29%0A++++++++append%28%22%24i%22%2C%22+is+a+nice+color%22%29%0A++++++++copy_field%28%22%24i%22%2C%22result%5B%5D.%24append%22%29%0A++++end%0Aend&data=---%0Acolours%3A%0A+-+red%0A+-+yellow%0A+-+green) -TO BE CONTINUED ... +For the supported binds see: https://metafacture.github.io/metafacture-documentation/docs/fix/Fix-functions.html#selectors -TODO: Add more Excercises. +TODO: Add excercises. Next lesson: [06 Metafacture CLI](./06_MetafactureCLI.md)