diff --git a/content/en/docs/refguide/modeling/integration/data-transformers/_index.md b/content/en/docs/refguide/modeling/integration/data-transformers/_index.md new file mode 100644 index 00000000000..ace83eaf30c --- /dev/null +++ b/content/en/docs/refguide/modeling/integration/data-transformers/_index.md @@ -0,0 +1,130 @@ +--- +title: "Data Transformers" +url: /refguide/data-transformers/ +weight: 50 +description: "Describes Data Transformers in Mendix Studio Pro." +--- + +## Introduction + +Data Transformer can be used for transforming data of certain structure into another structure, basically a message-to-message transformation within Mendix Studio Pro. With this feature, you can pre-process an incoming message (for example, from an API response, MQTT message, etc.) before an Import Mapping. Additionally, you can also use it to transform a message before passing it on to a downstream system that expects the data in a certain structure. + +{{% alert color="info" %}} +This feature is in beta and at the moment we only support JSON-to-JSON transformation with JSLT, a JSON transformation language. +{{% /alert %}} + +### Example + +Consider an API that returns customer data with many fields, but you only need a few specific fields for your Mendix app: + +**Input JSON from API:** + +```json +{ + "customer_id": "12345", + "first_name": "John", + "last_name": "Smith", + "email": "john.smith@example.com", + "phone": "+1-555-0123", + "address": "123 Main St", + "city": "New York", + "country": "USA", + "account_status": "active", + "credit_limit": 5000, + "internal_notes": "VIP customer", + "created_date": "2024-01-15" +} +``` + +**JSLT Transformation:** + +```jslt +{ + "id": .customer_id, + "fullName": .first_name + " " + .last_name, + "email": .email, + "location": .city + ", " + .country +} +``` + +**Output JSON:** + +```json +{ + "id": "12345", + "fullName": "John Smith", + "email": "john.smith@example.com", + "location": "New York, USA" +} +``` + +In this example, the Data Transformer extracts only the needed fields, combines the first and last name into a single field, and creates a location string from the city and country. This simplified output can then be easily mapped to your Mendix entities. + +## Limitations + +At the moment we only support JSON-to-JSON transformation with JSLT, a JSON transformation language. + +## Prerequisites + +* [Studio Pro 11.11](https://marketplace.mendix.com/link/studiopro/11.11.0) and above + +## Add the Data Transformer Document + +You can add the Data Transformer document to your app, by following these steps: + +1. Right-click the module you want to add the Data Transformer document to. +2. Select **Add other** > **Data Transformer**. +3. Name the data transformer. + +{{< figure src="/attachments/refguide/modeling/integration/data-transformers/add-data-transformer.png" alt="Add Data Transformer dialog" >}} + +4. In the **Input JSON** editor you can paste a JSON snippet that you would like to transform. +5. Define the transformation in **JSLT transformation** editor. +6. Click the **Test Transformation** button below the JSLT transformation editor to preview the transformation result in the **Output JSON**. + +{{< figure src="/attachments/refguide/modeling/integration/data-transformers/define-transformation.png" alt="Data Transformer interface showing Input JSON, JSLT transformation, and Output JSON editors" >}} + +## Use the Data Transformer in a Microflow + +To perform a transformation in a microflow, complete the following steps: + +1. Drag the **Transform JSON** activity into a microflow, preferably after a REST call or anything that provides input for the transformation. + +{{< figure src="/attachments/refguide/modeling/integration/data-transformers/transform-json-dialog.png" alt="Add Transform JSON activity" >}} + +2. Double-click the activity and click **Select** to choose an existing Data Transformer document or create a new one. +3. Click on the dropdown **Variable (String)** and select the input string variable from the list. +4. Specify the name of the output in the **Variable name** text field. +5. Click **OK**. + +{{< figure src="/attachments/refguide/modeling/integration/data-transformers/configure-transformer-activity.png" alt="Configure Transform JSON activity dialog" >}} + +## Using the output of Data Transformer + +You can use your transformed JSON snippet in the following ways: + +* Create a new JSON structure for Import Mapping +* Pass the transformed JSON to downstream systems +* Use it as input for further processing in your microflow + + +## Use Cases + +Use the Data Transformer document to do the following: + +* [Filtering out unused fields](/refguide/data-transformer-how-tos/#filtering-out-unused-fields) +* [Simplifying nested structures](/refguide/data-transformer-how-tos/#simplifying-nested-structures) +* [Normalising objects to arrays (working with dynamic keys)](/refguide/data-transformer-how-tos/#normalising-objects-to-arrays-working-with-dynamic-keys) +* [Zipping metadata with data](/refguide/data-transformer-how-tos/#zipping-metadata-with-data) +* [Flattening Bill of Materials (BOM)](/refguide/data-transformer-how-tos/#flattening-bill-of-materials-bom) +* [Extracting information from a string](/refguide/data-transformer-how-tos/#extracting-information-from-a-string) +* [Working with SPARQL query results](/refguide/data-transformer-how-tos/#working-with-sparql-query-results) + +For detailed examples, see [Data Transformer How-Tos](/refguide/data-transformer-how-tos/). + +## JSLT + +You can find more information about JSLT below: + +* A short [introduction](https://github.com/schibsted/jslt/blob/master/README.md#jslt) and [tutorial](https://github.com/schibsted/jslt/blob/master/tutorial.md#jslt-tutorial) on how to use JSLT +* A complete list of [functions available in JSLT](https://github.com/schibsted/jslt/blob/master/functions.md#jslt-functions) \ No newline at end of file diff --git a/content/en/docs/refguide/modeling/integration/data-transformers/data-transformer-how-tos.md b/content/en/docs/refguide/modeling/integration/data-transformers/data-transformer-how-tos.md new file mode 100644 index 00000000000..56545b8e614 --- /dev/null +++ b/content/en/docs/refguide/modeling/integration/data-transformers/data-transformer-how-tos.md @@ -0,0 +1,506 @@ +--- +title: "Data Transformer How-Tos" +url: /refguide/data-transformer-how-tos/ +weight: 60 +description: "Provides practical examples of common JSLT transformation patterns for Data Transformers." +--- + +## Introduction + +How-tos separate page + +These how-tos provide practical, example-driven walkthroughs of common JSLT transformation patterns. Each guide focuses on a specific use-case you may encounter when working with real-world API responses, such as restructuring nested data, renaming fields, or combining metadata with data. Rather than covering every detail of the JSLT language, the guides are designed to be hands-on and immediately applicable, showing a concrete input, the expected output, and the transformation that connects the two. + +## Filtering out unused fields + +It is common for an API to return payloads that contain more fields than your Mendix app or a downstream system need. Rather than passing the entire payload along, this transformation selects only the fields that are relevant, effectively dropping everything else. This keeps the output clean, reduces payload size, and avoids exposing unnecessary data. + +### Example + +**Input:** + +```json +{ + "orderId": "ORD-4521", + "customerId": "CUST-001", + "orderDate": "2024-05-10T09:15:00Z", + "shippingAddress": "456 Elm Street, Berlin, Germany", + "totalAmount": 149.99, + "currency": "EUR", + "status": "shipped" +} +``` + +**JSLT:** + +```jslt +{ + "orderId": .orderId, + "customerId": .customerId, + "status": .status +} +``` + +**Output:** + +```json +{ + "orderId": "ORD-4521", + "customerId": "CUST-001", + "status": "shipped" +} +``` + +### Explanation + +In JSLT, the output object is always constructed explicitly: only the fields you name in the query will appear in the output, just like SQL or OQL. This means that dropping fields requires no special syntax; any field from the input that is simply not referenced in the JSLT is automatically excluded from the result. In this example, fields such as orderDate and shippingAddress are dropped by simply not being referenced in the query. The three relevant fields are carried over directly from the input. + +Read more about constructing JSLT queries: https://github.com/schibsted/jslt/blob/master/tutorial.md#dot-accessors + +## Simplifying nested structures + +Sometimes a JSON structure contains nested sub-objects that group related fields together, but you just need a simpler, flat representation. This transformation moves fields from nested sub-objects up to the top level, merging them into a single flat object. + +### Example + +**Input:** + +```json +{ + "id": "USR-001", + "name": "Jane Doe", + "email": "jane.doe@example.com", + "profileImage": { + "url": "https://example.com/images/jane.jpg", + "altText": "Profile photo of Jane Doe" + }, + "address": { + "street": "123 Main St", + "city": "Munich", + "country": "Germany" + } +} +``` + +**JSLT:** + +```jslt +{ + "id": .id, + "name": .name, + "email": .email, + "profileImageUrl": .profileImage.url, + "profileImageAltText": .profileImage.altText, + "addressStreet": .address.street, + "addressCity": .address.city, + "addressCountry": .address.country +} +``` + +**Output:** + +```json +{ + "id": "USR-001", + "name": "Jane Doe", + "email": "jane.doe@example.com", + "profileImageUrl": "https://example.com/images/jane.jpg", + "profileImageAltText": "Profile photo of Jane Doe", + "addressStreet": "123 Main St", + "addressCity": "Munich", + "addressCountry": "Germany" +} +``` + +### Explanation + +The transformation is straightforward. Each field in the output is explicitly mapped from the input using its full path. Fields that were previously nested inside a sub-object are accessed using dot notation (e.g. .profileImage.url) and assigned to a new top-level key that reflects their origin (e.g. profileImageUrl). The result is a flat object where all fields sit at the same level. + +Read more about dot accessors: https://github.com/schibsted/jslt/blob/master/tutorial.md#dot-accessors + +## Normalising objects to arrays (working with dynamic keys) + +Some APIs return collections of records as a keyed object, where each key acts as a unique identifier for that record. Mendix works more naturally with lists of objects, so this transformation converts that keyed structure into a flat, normalised array that can be directly mapped to a Mendix entity list. + +### Example + +**Input:** + +```json +{ + "data": { + "Tag1": { + "TagName": "TQ-02-FIT-WA123123", + "Value": 0 + }, + "Tag2": { + "TagName": "TQ-02-WIT-WA123123", + "Value": 41.7807087 + }, + "Tag3": { + "TagName": "TQ-03-FIT-WA123123", + "Value": 45.812341 + }, + "Tag4": { + "TagName": "TQ-04-FIT-WA123123", + "Value": 0 + } + } +} +``` + +**JSLT:** + +```jslt +[for (.data) + { + "TagId": .key, + "TagName": .value.TagName, + "Value": .value.Value + } +] +``` + +**Output:** + +```json +[ { + "TagId" : "Tag1", + "TagName" : "TQ-02-FIT-WA123123", + "Value" : 0 +}, { + "TagId" : "Tag2", + "TagName" : "TQ-02-WIT-WA123123", + "Value" : 41.7807087 +}, { + "TagId" : "Tag3", + "TagName" : "TQ-03-FIT-WA123123", + "Value" : 45.812341 +}, { + "TagId" : "Tag4", + "TagName" : "TQ-04-FIT-WA123123", + "Value" : 0 +} ] +``` + +### Explanation + +When you use a for expression to iterate over an object, JSLT converts each key-value pair into: + +```json +{ "key": "", "value": } +``` + +So for (.data) iterates over each entry, exposing .key (e.g., "Tag1") and .value (e.g., { "TagName": "...", "Value": ... }). We then construct a new flat object per entry, promoting .key into its own "TagId" field alongside the fields from .value. + +Read more about for expression and constructing lists in JSLT: https://github.com/schibsted/jslt/blob/master/tutorial.md#for-expressions + +## Zipping metadata with data + +Some APIs return data and its metadata separately: the metadata describes the structure (e.g. column names), while the data is returned as raw arrays. This is the case with, for example, Snowflake SQL REST APIs. To make the data meaningful and easy to consume, the two need to be combined so that each value is associated with its corresponding column name. + +In this example, a Snowflake SQL REST API response (with input fields irrelevant to this use-case omitted) contains employee records returned as arrays, alongside a separate metadata block that defines the column names. The transformation zips these together to produce a list of objects. + +### Example + +**Input:** + +```json +{ + "resultSetMetaData": { + "numRows": 4, + "rowType": [ + { "name": "EMPLOYEE_ID", "type": "fixed" }, + { "name": "FULL_NAME", "type": "text" }, + { "name": "DEPARTMENT", "type": "text" }, + { "name": "SALARY", "type": "fixed" } + ] + }, + "data": [ + ["1001", "Anna Müller", "Engineering", "72000"], + ["1002", "James Carter", "Finance", "65000"], + ["1003", "Sofia Rossi", "Engineering", "78000"], + ["1004", "Liam O'Brien", "HR", "58000"] + ] +} +``` + +**JSLT:** + +```jslt +let cols = .resultSetMetaData.rowType + +{ + "data": [for (.data) + let row = . + {for (zip($cols, $row)) + .[0].name : .[1] + } + ] +} +``` + +**Output:** + +```json +{ + "data" : [ { + "EMPLOYEE_ID" : "1001", + "FULL_NAME" : "Anna Müller", + "DEPARTMENT" : "Engineering", + "SALARY" : "72000" + }, { + "EMPLOYEE_ID" : "1002", + "FULL_NAME" : "James Carter", + "DEPARTMENT" : "Finance", + "SALARY" : "65000" + }, { + "EMPLOYEE_ID" : "1003", + "FULL_NAME" : "Sofia Rossi", + "DEPARTMENT" : "Engineering", + "SALARY" : "78000" + }, { + "EMPLOYEE_ID" : "1004", + "FULL_NAME" : "Liam O'Brien", + "DEPARTMENT" : "HR", + "SALARY" : "58000" + } ] +} +``` + +### Explanation + +The transformation starts by storing the column definitions in a variable $cols for later use inside the loops. It then iterates over each row in .data, capturing the current row as $row. For each row, zip($cols, $row) pairs every column definition with its corresponding value by position, producing two-element arrays like: + +```json +[ + [{"name": "EMPLOYEE_ID", "type": "fixed"}, "1001"], + [{"name": "FULL_NAME", "type": "text"}, "Anna Müller"], + ... +] +``` + +These pairs are then fed into an object for expression, which builds the output object by using the column name (.[0].name) as the key and the row value (.[1]) as the value. + +Read more about declaring variables: https://github.com/schibsted/jslt/blob/master/tutorial.md#variables + +Read more about zip and other functions: https://github.com/schibsted/jslt/blob/master/functions.md#ziparray1-array2---array + +## Flattening Bill of Materials (BOM) + +A Bill of Materials (BOM) is naturally represented as a tree structure, where each assembly can contain child sub-assemblies, which can themselves contain further children. Flattening such a structure into a simple list is sometimes needed when feeding data into downstream systems that expect a flat, tabular format. This transformation also helps in simplifying the Import Mapping process of the BOM to Mendix entities. + +In this example, a Bicycle BOM is represented as a nested tree. The transformation flattens it into a list of assemblies, where each entry records its own ID and name, its direct parent (parentSubAssembly), and its direct children (childrenSubAssemblies). + +### Example + +**Input:** + +```json +{ + "name": "Bicycle", + "rootSubAssemblies": [ + { + "attributes": { + "baseID": "UID-001", + "name": "Back Wheel", + "children": [ + { + "attributes": { + "baseID": "UID-002", + "name": "Rubber Tire", + "children": [] + } + }, + { + "attributes": { + "baseID": "UID-003", + "name": "Rim", + "children": [ + { + "attributes": { + "baseID": "UID-004", + "name": "Spoke", + "children": [] + } + } + ] + } + } + ] + } + } + ] +} +``` + +**JSLT:** + +```jslt +def flatten-assemblies(assemblies, parentAssembly) + [for ($assemblies) + let parentWithoutChildren = if ($parentAssembly) + $parentAssembly.attributes.baseID + else null + let childrenWithoutGrandchildren = [for (.attributes.children) + .attributes.baseID + ] + let currentAssembly = { + "baseID": .attributes.baseID, + "name": .attributes.name, + "parentSubassembly": $parentWithoutChildren, + "childrenSubassemblies": $childrenWithoutGrandchildren + } + let childAssemblies = if (.attributes.children and size(.attributes.children) > 0) + flatten-assemblies(.attributes.children, .) + else + [] + [$currentAssembly] + $childAssemblies + ] | flatten(.) + +{ + "name": .name, + "flattenedAssemblies": flatten-assemblies(.rootSubAssemblies, null) +} +``` + +**Output:** + +```json +{ + "name" : "Bicycle", + "flattenedAssemblies" : [ { + "baseID" : "UID-001", + "name" : "Back Wheel", + "parentSubassembly" : null, + "childrenSubassemblies" : [ "UID-002", "UID-003" ] + }, { + "baseID" : "UID-002", + "name" : "Rubber Tire", + "parentSubassembly" : "UID-001", + "childrenSubassemblies" : [ ] + }, { + "baseID" : "UID-003", + "name" : "Rim", + "parentSubassembly" : "UID-001", + "childrenSubassemblies" : [ "UID-004" ] + }, { + "baseID" : "UID-004", + "name" : "Spoke", + "parentSubassembly" : "UID-003", + "childrenSubassemblies" : [ ] + } ] +} +``` + +### Explanation + +The transformation defines a recursive function flatten-assemblies that takes a list of assemblies and the parent assembly of that list. For each assembly it processes, it first resolves the parent's baseID (or null if there is no parent) and collects the baseID of each direct child, without descending further. It then constructs a flat object for the current assembly containing its ID, name, parent reference, and list of child IDs. If the current assembly has children, the function calls itself recursively on those children, passing the current assembly as the new parent. The results for the current assembly and all its descendants are concatenated into a single array, and flatten is applied at the end of each recursive level to collapse the nested arrays into a flat list. + +The root of the transformation kicks this off by calling flatten-assemblies on rootSubAssemblies with null as the initial parent, producing a fully flattened list that preserves the parent-child relationships without any nesting. + +Read more about declaring functions in JSLT: https://github.com/schibsted/jslt/blob/master/tutorial.md#function-declarations + +## Extracting information from a string + +Sometimes multiple pieces of information are encoded within a single structured string, such as a file path, an identifier, or a URL, and you need to extract a specific piece of that information for use downstream or in your own Mendix app. JSLT's string functions allow you to extract each component into its own dedicated field. This makes the data easier to consume, filter, and process without placing any additional burden on the downstream step. In this example, a file path string is broken down into its individual components: the root folder, department, year, and file name, each mapped to a dedicated output field. + +### Example + +**Input:** + +```json +{ + "filePath": "reports/finance/2024/annual-report.pdf" +} +``` + +**JSLT:** + +```jslt +{ + "folder": split(.filePath, "/")[0], + "department": split(.filePath, "/")[1], + "year": split(.filePath, "/")[2], + "fileName": split(.filePath, "/")[3] +} +``` + +**Output:** + +```json +{ + "folder": "reports", + "department": "finance", + "year": "2024", + "fileName": "annual-report.pdf" +} +``` + +### Explanation + +The split function breaks the file path string into an array of segments using / as the separator, producing the following array: + +```json +["reports", "finance", "2024", "annual-report.pdf"] +``` + +Each segment is then accessed by its index: [0] for the first element, [1] for the second, and so on. This allows each component of the path to be mapped to a clearly named output field. + +For other useful built-in functions, refer to: https://github.com/schibsted/jslt/blob/master/functions.md#jslt-functions + +## Working with SPARQL query results + +SPARQL is a query language for RDF data, commonly used with knowledge graphs and semantic web APIs. Its query results follow a standard JSON format where the column names (called variables) are declared separately in a head block, and the actual result rows are returned as bindings, a list of objects where each key maps to a typed value wrapper rather than a plain value. This structure is precise and interoperable, but verbose. Transforming it into a simple flat list of objects makes it far easier to work with in Import Mappings. + +In this example, a SPARQL query returns customer records. The transformation extracts the variable names from the head block and uses them to map each binding into a plain object, pulling the value field out of each typed wrapper. + +### Example + +**Input:** + +```json +{ + "head": { "vars": ["customer", "customerId", "customerName"] }, + "results": { + "bindings": [ + { + "customer": { "type": "uri", "value": "http://.../Customer/0000000" }, + "customerId": { "type": "literal", "value": "CUST001" }, + "customerName": { "type": "literal", "value": "Global Tech Solutions Inc." } + } + ] + } +} +``` + +**JSLT:** + +```jslt +let vars = .head.vars + +([for (.results.bindings) + let binding = . + {for ($vars) + . : fallback(get-key($binding, .).value, "") + } +]) +``` + +**Output:** + +```json +[ + { + "customer" : "http://.../Customer/0000000", + "customerId" : "CUST001", + "customerName" : "Global Tech Solutions Inc." + } +] +``` + +### Explanation + +The variable names are captured into vars at the root level before any looping begins. The transformation then iterates over each binding in the results. Because .will be rebound inside the inner loop, the current binding is saved into binding immediately. The inner for loop iterates over the variable names, using each variable name as both the key and the lookup argument — get-key($binding, .) retrieves the typed value wrapper for that variable from the saved binding, and .value extracts the plain value from it. fallback ensures that if a variable is missing from a binding, an empty string is used instead of null. The result is a clean, flat list of objects with no type wrappers that you can easily use as source for Import Mapping. + +Read more about get-key, fallback, and other functions: https://github.com/schibsted/jslt/blob/master/functions.md#get-keyobject-key-fallback---value diff --git a/static/attachments/refguide/modeling/integration/data-transformers/add-data-transformer.png b/static/attachments/refguide/modeling/integration/data-transformers/add-data-transformer.png new file mode 100644 index 00000000000..e39e9ecb81a Binary files /dev/null and b/static/attachments/refguide/modeling/integration/data-transformers/add-data-transformer.png differ diff --git a/static/attachments/refguide/modeling/integration/data-transformers/configure-transformer-activity.png b/static/attachments/refguide/modeling/integration/data-transformers/configure-transformer-activity.png new file mode 100644 index 00000000000..0627c535569 Binary files /dev/null and b/static/attachments/refguide/modeling/integration/data-transformers/configure-transformer-activity.png differ diff --git a/static/attachments/refguide/modeling/integration/data-transformers/define-transformation.png b/static/attachments/refguide/modeling/integration/data-transformers/define-transformation.png new file mode 100644 index 00000000000..7aeb1fdb5f2 Binary files /dev/null and b/static/attachments/refguide/modeling/integration/data-transformers/define-transformation.png differ diff --git a/static/attachments/refguide/modeling/integration/data-transformers/transform-json-dialog.png b/static/attachments/refguide/modeling/integration/data-transformers/transform-json-dialog.png new file mode 100644 index 00000000000..6a99017b691 Binary files /dev/null and b/static/attachments/refguide/modeling/integration/data-transformers/transform-json-dialog.png differ