v1.5.0

Highlights

Matrix Connector

Added the matrix connector. This allows running multiple writes in parallel based on variables. e.g. run once each for a list of categories.

# Create files for a list of categories
# a file for Tools.xlsx, PPE.xlsx and Electrical.xlsx will be created
write:
  - matrix:
      variables:
        category: ["Tools", "PPE", "Electrical"]
      write:
        - file:
            name: ${category}.xlsx
            where: category = ?
            where_params:
              - ${category}

select.element

This supports python syntax for selecting from lists and dicts. e.g. column_name["key"][0].

wrangles:
  - select.element:
      input: input_column["key"][0]
      output: output_column

select.group_by

Easily aggregate and group data.

wrangles:
  - select.group_by:
      by:
        - category1
        - category2
      sum: sum_me
      min: min_me
      max: max_me

Features

• Added convert to and from YAML.
• Allow setting a default value for split.dictionary.
• Added the memory connector - save data in memory for communication between successive wrangles/recipes.
• Support recipe in recipe syntax, in addition to externally referenced recipes such as from files.
• Enable running a recipe from a model ID.

Bugs / Misc

• Fixed an issue with the schema for convert.to_json.
• Set default ensure_ascii = False for convert.to_json to prevent issues with double encoding.
• Fixed a bug that where can't use a column not included in the write.
• Recipes read from a file now assume UTF-8 encoding for all operating systems.
• Display the name of the wrangle in error messages for wrangles.