Skip to content

0.8.75 | Guardians Release
Compare
Choose a tag to compare
@thomaschung408 thomaschung408 released this 09 May 20:11
· 2034 commits to master since this release
0.8.75
44028b0
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Image

Polars integration

Support using Polars DataFrame in Mage blocks.

Untitled

Opsgenie integration

Shout out to Sergio Santiago for his contribution of integrating Opsgenie as an alerting option in Mage.

Doc: https://docs.mage.ai/production/observability/alerting-opsgenie

Untitled

Data integration

Speed up exporting data to BigQuery destination

Add support for using batch load jobs instead of the query API in BigQuery destination. You can enable it by setting use_batch_load to true in BigQuery destination config.

When loading ~150MB data to BigQuery, using batch loading reduces the time from 1 hour to around 2 minutes (30x the speed).

Doc: https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/bigquery/README.md

Microsoft SQL Server destination improvements

  • Support ALTER command to add new columns
  • Support MERGE command with multiple unique columns (use AND to connect the columns)
  • Add MSSQL config fields to io_config.yaml
  • Support multiple keys in MSSQL destination

Other improvements

  • Fix parsing int timestamp in intercom source.
  • Remove the “Execute” button from transformer block in data integration pipelines.
  • Support using ssh tunnel for MySQL source with private key content

Git integration improvements

Update disable notebook edit mode to allow certain operations

Add another value to DISABLE_NOTEBOOK_EDIT_ACCESS environment variable to allow users to create secrets, variables, and run blocks.

The available values are

  • 0: this is the same as omitting the variable
  • 1: no edit/execute access to the notebook within Mage. Users will not be able to use the notebook to edit pipeline content, execute blocks, create secrets, or create variables.
  • 2: no edit access for pipelines. Users will not be able to edit pipeline/block metadata or content.

Doc: https://docs.mage.ai/production/configuring-production-settings/overview#read-only-access

Update Admin abilities

  • Allow admins to view and update existing users' usernames/emails/roles/passwords (except for owners and other admins).
  • Admins can only view Viewers/Editors and adjust their roles between those two.
  • Admins cannot create or delete users (only owners can).
  • Admins cannot make other users owners or admins (only owners can).

Retry block runs from specific block

For standard python pipelines, retry block runs from a selected block. The selected block and all downstream blocks will be re-ran after clicking the Retry from selected block button.

Untitled

Other bug fixes & polish

  • Fix terminal user authentication. Update terminal authentication to happen on message.

  • Fix a potential authentication issue for the Google Cloud PubSub publisher client

  • Dependency graph improvements

    • Update dependency graph connection depending on port side
    • Show all ports for data loader and exporter blocks in dependency graph

    Untitled

  • DBT

    • Support DBT alias and schema model config
    • Fix limit property in DBT block PUT request payload.

  • Retry pipeline run

    • Fix bug: Individual pipeline run retries does not work on sqlite.
    • Allow bulk retry runs when DISABLE_NOTEBOOK_EDIT_ACCESS enabled
    • Fix bug: Retried pipeline runs and errors don’t appear in Backfill detail page.

  • Fix bug: When Mage fails to fetch a pipeline due to a backend exception, it doesn't show the actual error. It uses "undefined" in the pipeline url instead, which makes it hard to debug the issue.

  • Improve job scheduling: If jobs with QUEUED status are not in queue, re-enqueue them.

  • Pass imagePullSecrets to k8s job when using k8s as the executor.

  • Fix streaming pipeline cancellation.

  • Fix the version of google-cloud-run package.

  • Fix query permissions for block resource

  • Catch sqlalchemy.exc.InternalError in server and roll back transaction.

View full Changelog

Assets 2