0.8.44 | Dungeons and Dragons Release

Configure trigger in code

In addition to configuring triggers in UI, Mage also supports configuring triggers in code now. Create a triggers.yaml file under your pipeline folder and enter the triggers config. The triggers will automatically be synced to DB and trigger UI.

Doc: https://docs.mage.ai/guides/triggers/configure-triggers-in-code

Centralize server logger and add verbosity control

Shout out to Dhia Eddine Gharsallaoui for his contribution of centralizing the server loggings and adding verbosity control. User can control the verbosity level of the server logging by setting the SERVER_VERBOSITY environment variables. For example, you can set SERVER_VERBOSITY environment variable to ERROR to only print out errors.

Doc: https://docs.mage.ai/production/observability/logging#server-logging

Customize resource for Kubernetes executor

User can customize the resource when using the Kubernetes executor now by adding the executor_config to the block config in pipeline’s metadata.yaml.

Doc: https://docs.mage.ai/production/configuring-production-settings/compute-resource#kubernetes-executor

Data integration pipelines

• Google sheets source: Fix loading sample data from Google Sheets
• Postgres source: Allow customizing the publication name for logical replication
• Google search console source: Support email field in google_search_console config
• BigQuery destination: Limit the number of subqueries in BigQuery query
• Show more descriptive error (instead of {}) when a stream that was previously selected may have been deleted or renamed. If a previously selected stream was deleted or renamed, it will still appear in the SelectStreams modal but will automatically be deselected and indicate that the stream is no longer available in red font. User needs to click "Confirm" to remove the deleted stream from the schema.

Terminal improvements

• Use named terminals instead of creating a unique terminal every time Mage connects to the terminal websocket.
• Update terminal for windows. Use cmd shell command for windows instead of bash. Allow users to overwrite the shell command with the SHELL_COMMAND environment variable.
• Support copy and pasting multiple commands in terminal at once.
• When changing the path in the terminal, don’t permanently change the path globally for all other processes.
• Show correct logs in terminal when installing requirements.txt.

DBT improvements

• Interpolate environment variables and secrets in DBT profile

Git improvements

• Update git to support multiple users

Postgres exporter improvements

• Support reordering columns when exporting a dataframe to Postgres
• Support specifying unique constraints when exporting the dataframe

with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,
            if_exists='append',
            allow_reserved_words=True,
            unique_conflict_method='UPDATE',
            unique_constraints=['col'],
        )

Other bug fixes & polish

• Fix chart loading errors.
• Allow pipeline runs to be canceled from UI.
• Fix raw SQL block trying to export upstream python block.
• Don’t require metadata for dynamic blocks.
• When editing a file in the file editor, disable keyboard shortcuts for notebook pipeline blocks.
• Increase autosave interval from 5 to 10 seconds.
• Improve vertical navigation fixed scrolling.
• Allow users to force delete block files. When attempting to delete a block file with downstream dependencies, users can now override the safeguards in place and choose to delete the block regardless.

View full Changelog