Skip to content

Conversation

@topher-lo
Copy link
Contributor

@topher-lo topher-lo commented May 15, 2025

What changed

  • Added support to raise error, raise error with check (i.e. jsonpaths that lead to null values), and drop nulls in core.transform.reshape action
  • check is super powerful because it turns a list of jsonpaths / dot notation to the fields with nulls e.g:
Null values encountered in output. Found null values in the following fields:

 - metadata.updated_by
 - items[0].nested.values[1]
 - items[1].nested

It even supports lists indexing and fields with special characters e.g. items[0]."this is some".field

Why do we need this

Reshape is often used to organize / transform data from upstream (i.e. from an alert) into a flatter / more readable object. This is then used in downstream tasks (e.g. post notification to Slack, open Jira ticket).

We want to raise an error with informative message (for fast and easy debugging) if there any any nulls.

QA

Added unit tests and created simple PoC workflow, which expects ip but receives ip_address field instead. We pass the error message to Claude to explain.

Screenshot 2025-05-15 at 3 42 32 PM Screenshot 2025-05-15 at 3 48 42 PM

@topher-lo topher-lo added the enhancement New feature or request label May 15, 2025
@topher-lo topher-lo changed the title feat(integrations): Check is any nulls in reshape output feat(integrations): Check if any nulls in reshape output May 15, 2025
@topher-lo topher-lo closed this Jun 7, 2025
@topher-lo topher-lo deleted the feat/reshape-error-if-none branch June 17, 2025 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants