Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: provide core task for easily manipulating JSON structured data #3148

Open
fhussonnois opened this issue Feb 26, 2024 · 0 comments
Open
Labels
enhancement New feature or request
Milestone

Comments

@fhussonnois
Copy link
Member

fhussonnois commented Feb 26, 2024

Feature description

Motivation

Kestra relies on JSON structured data (e.g., Amazon ION) for exchanging outputs among flow tasks. Frequently, data processing entails applying transformations to JSON records (e.g., formatting, selection, aggregation).
Presently, achieving these transformations entails using the FileTransform task, which necessitates scripting in Javascript, Python, or Groovy. However, this approach isn't optimal for basic transformations and is not very user-friendly

Proposition

Introduce a dedicated task in Kestra for JSON manipulation, leveraging solutions like JSONata. This would empower users to perform data transformations efficiently using a powerful querying language, enhancing workflow simplicity and flexibility.

Flow Example

For example, using the following Flow:

id: example
namespace: example
tasks:
- id: transformJson
  task: io.kestra.plugin.task.JSONataTransform
  # can be either a Kestra URI or a STRING
  from: {{ previousTask.outputs.uri }}
  expr: |
     {
        "order_id": order_id,
        "customer_name": customer_name,
        "total_price": $sum(items.(quantity * price_per_unit))
     }

with the given JSON input:

{
  "order_id": "ABC123",
  "customer_name": "John Doe",
  "items": [
    {
      "product_id": "001",
      "name": "Apple",
      "quantity": 5,
      "price_per_unit": 0.5
    },
    {
      "product_id": "002",
      "name": "Banana",
      "quantity": 3,
      "price_per_unit": 0.3
    },
    {
      "product_id": "003",
      "name": "Orange",
      "quantity": 2,
      "price_per_unit": 0.4
    }
  ]
}

will output:

{
  "order_id": "ABC123",
  "customer_name": "John Doe",
  "total_price": 4.2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Backlog
Development

No branches or pull requests

2 participants