v1.13.0

Extract.ai enhancements

• Run saved extract.ai wrangles by using their model_id. This functionality is currently available with extract.custom. Extract.custom will continue to work, and the main difference is that extract.ai will use the definition but run locally, while extract.custom will run remotely on the server. It is recommended to use extract.ai due to the greater flexibility running locally, including extending the timeout beyond 30s.

• Alternative simpler syntax. Basic definitions can be provided as just descriptions, without needing to define type etc.

  wrangles:
    - extract.ai:
         output:
            length_column: Any lengths found in the data such as cm, m, ft, etc.
            # <column name>: <description of desired output>

• Added a python functional version. This can receive an input of a single value or a list. If provided a list, the prompt will be applied to each element individually. The output can be defined as a string, a dict containing a single JSON schema definition or a dict with multiple JSON schema definitions per key.

  wrangles.extract.ai(
    "yellow square",
    api_key="<api_key>",
    output="The names of any colors found in the input"
  )
  # --> "yellow"

• Support models that don't allow the temperature parameter, such as o1 and o3.

• Fixed a bug where the wrangle would fail on unhandled exceptions.

Others

• select.group_by: Fixed a bug where not using an aggregation caused the columns in by to split the characters with periods.
• create.embeddings: Added precision parameter with options float16 or float32 to set the numeric precision of the output.
• convert.to_yaml: Fixed a typo in schema docs for sort_keys.
• round: Improved error handling.
• compare.text: Fixed a bug where exact matches gave a 0 score.
• try: Added a retries parameter to set the number of times to attempt to retry.
• copy: Added an option to express columns as a dictionary, as for rename.
• format.remove_duplicates: Added ignore_case option.
• classify: Added include_confidence true/false parameter.
• create.hash: New wrangle to create a hash from an input. Supports md5, sha1, sha256, and sha512.
• select.length: New wrangle to get the length of the values in a column. This will use python's length behaviour depending on the data type e.g. a list will be the number of elements, a string will be the number of characters.
• extract.regex: Added support for output_patterns to use capture groups to structure the output. e.g. \1 \2
• extract.custom: Set default batch size for extract.ai variants to 20.
• Drop tests for python 3.8 now that it is EoL.
• Added tests and support for python 3.13.
• Fixed issues installing on MacOS using ARM due to dependencies. Python 3.9 is not currently support on Mac ARM CPUs.
• Fixed a bug where variables were not passed through correctly for accordion, batch, and rename
• Allowing passing wildcards as a list for the output of split wrangles. Only one column is supported if a wildcard is used.
• Use CSafeLoader and CSafeDumper for improved performance when reading/writing YAML on systems that support it, else fall back to the default methods.
• batch:
  • Log errors that are suppressed when using the on_error handler.
  • Add a timeout parameter to set a time limit for the batch.
• Fixed a bug where optional columns failed for some connectors.
• Added sqlite connector with read/write/run for a sqlite3 database file.
• Allow embeddings function to support single inputs in addition to a list. If a single input is provided, a single embedding array will be returned. If a list is provided, a list will be returned as before.
• Various schema updates.