Skip to content

Releases: mage-ai/mage-ai

0.9.70 | Fallout ☢️

26 Apr 23:53
@wangxiaoyou1993 wangxiaoyou1993
0.9.70
1e805ef
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.70 | Fallout ☢️ Latest
Latest

What's Changed

🎉 Exciting New Features

💥 Add "Last run failed" filter to Pipeline Runs table

This feature adds a new filter option to the Pipeline Runs table, allowing users to easily view and filter pipeline runs that have failed in their most recent execution. This makes it simpler to identify and troubleshoot problematic pipelines.
pipeline runs page

by @johnson-mage in #4926

🛠️ Add workspace update features

This feature allows users to update the workspace settings (environment variables) and update the workspace main container config from the UI. Being able to update the container config will allow users to update the workspace image as well.


by @dy46 in #4951

🔄 Add upsert to MySQL, BQ, and Snowflake

Add ability to upsert data in MySQL, BigQuery, and Snowflake IO classes.
It can be achieved by specifying the following params in the export method:

            unique_conflict_method='UPDATE',
            unique_constraints=['col'],

by @dy46 in #4769

⚠️ Support alerts and limited retries in streaming pipeline

Streaming pipeline used to be retried automatically and infinitely.
With this feature, when retry_config is set in streaming pipeline's metadata.yaml at pipeline level, only retry the streaming pipeline with limited times and send alerts when pipeline fails.
Example retry config

retry_config:
  # Number of retry times
  retries: 3
  # Initial delay (in seconds) before retry. If exponential_backoff is true,
  # the delay time is multiplied by 2 for the next retry
  delay: 5
  # Maximum time between the first attempt and the last retry
  max_delay: 60
  # Whether to use exponential backoff retry
  exponential_backoff: true

by @wangxiaoyou1993 in #4902

🔑 Add setting to update user roles on login

This feature adds the UPDATE_ROLES_ON_LOGIN setting which will tell Mage to update the roles every time a user logs in using LDAP or OAuth (that supports roles mapping). If LDAP or OAuth returns no roles, the roles for the user will not be updated.

by @dy46 in #4905

🐛 Bug Fixes

💅 Enhancements & Polish

Read more

Contributors

lqc, tommydangerous, and 10 other contributors
Assets 2

0.9.68 | Invincible 🦸‍♂️

28 Mar 15:35
@wangxiaoyou1993 wangxiaoyou1993
0.9.68
8ddf586
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.68 | Invincible 🦸‍♂️

What's Changed

🎉 Exciting New Features

🐍 Custom python source in Streaming pipeline

This feature allows users to use custom Python source code in their Streaming pipelines, which provides more flexibility and control for advanced users who need to incorporate custom logic or functionality into their streaming pipelines. Here is the code template:

from mage_ai.streaming.sources.base_python import BasePythonSource
from typing import Callable

if 'streaming_source' not in globals():
    from mage_ai.data_preparation.decorators import streaming_source


@streaming_source
class CustomSource(BasePythonSource):
    def init_client(self):
        """
        Implement the logic of initializing the client.
        """

    def batch_read(self, handler: Callable):
        """
        Batch read the messages from the source and use handler to process the messages.
        """
        while True:
            records = []
            # Implement the logic of fetching the records
            if len(records) > 0:
                handler(records)

by @wangxiaoyou1993 in #4763

📦 OracleDB exporter

Add OracleDB exporter template to standard batch pipeline.
image

by @matrixstone in #4785

🔧 Server logging format template overriding

This feature enables users to override the default server logging format template in Mage. This provides more flexibility in customizing the logging output, which can be useful for troubleshooting, monitoring, or integrating with external logging systems.
Here is an example setting:

export SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'

by @afal007 in #4772

🔄 Update pipeline name before pipeline creation

Allow user to edit the pipeline name and add a description/tags before creating a new pipeline. Previously, a randomized name was always used for new pipelines, and then users had to go to the pipeline's settings to update the name.
image

by @johnson-mage in #4790

🐛 Bug Fixes

💅 Enhancements & Polish

✨ New Contributors

Full Changelog: 0.9.66...0.9.68

Contributors

matrixstone, afal007, and 4 other contributors
Assets 2

0.9.66 | Shogun 🎌

14 Mar 00:08
@wangxiaoyou1993 wangxiaoyou1993
0.9.66
b3f8e23
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.66 | Shogun 🎌

What's Changed

🎉 Exciting New Features

🗑️ Delete backfills

  • Add delete button to backfills table that allows user to delete backfills individually.
  • Prevent user from setting a backfill interval unit value less than 1, which could cause issues loading the backfills.
    image

by @johnson-mage in #4726

🪪 Support AzureDevOps using Entra ID/AD authentication

If you have an Azure DevOps organization connected to your Active Directory/Microsoft Entra ID tenant, you can enable DevOps in the version control app.

  • First, you will need to enable the Active Directory integration in order for Mage to authenticate your user.
  • Once the Active Directory integration is enabled, you simply need to add your DevOps organization as an environment variable, and you should be able to authenticate with DevOps in the version control app
    • export AZURE_DEVOPS_ORGANIZATION=<organization_name>

by @dy46 in #4721

📅  Scheduler status endpoint

Add /api/status endpoint to scheduler when running scheduler in standalone mode.
This endpoint can be used to check the liveness of the scheduler.


by @wangxiaoyou1993 in #4736

🔐 Read authentication settings from AWS secrets manager

Add support for reading settings from AWS secrets manager. The existing authentication settings have been moved to using the new Settings class.

In order to set the settings backend, the user will need to add a section to the project metadata. Here is an example to set the setting backend to AWS secrets manager:

settings_backend:
  backend_type: aws_secrets_manager
  prefix: default_repo/settings/

Read the doc for details.

by @dy46 in #4674

💅 Enhancements & Polish

🐛 Bug Fixes

✨ New Contributors

Full Changelog: 0.9.65...0.9.66

Contributors

Jawahars, ckfear, and 7 other contributors
Assets 2

0.9.65 | Demon Slayer 🐗 ⚔️

28 Feb 23:59
@wangxiaoyou1993 wangxiaoyou1993
0.9.65
563e3b0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.65 | Demon Slayer 🐗 ⚔️

What's Changed

🎉 Exciting New Features

🔔 Telegram notification


Support for setting up Telegram notification channel for alerts is now available.
Check out the doc for details

by @luthfir-rahman in #4632

🔄 Runtime variables and concurrency config in Backfills

Backfills support configuring global variables override and concurrency limit is now available.


by @dy46 in #4618

☁️ Export raw files to S3 and GCS

Using S3 and GCS data exporter templates, you can now directly upload raw files in addition to dataframes.

    bucket_name = 'test-bucket'
    object_key = 'test_file_copy.csv'

    S3.with_config(ConfigFileLoader(config_path, config_profile)).export(
        '/home/src/test_file.csv',
        bucket_name,
        object_key,
    )
    GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
        '/home/src/test_file.csv',
        bucket_name,
        object_key,
    )

by @wangxiaoyou1993 in #4607

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

Full Changelog: 0.9.64...0.9.65

Contributors

hugabora, carillpower, and 7 other contributors
Assets 2

0.9.64 | Maestro 🎵

16 Feb 02:43
@wangxiaoyou1993 wangxiaoyou1993
0.9.64
dcb9b0d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.64 | Maestro 🎵

What's Changed

🎉 Exciting New Features

🔭 OpenTelemetry integration

  • Added OpenTelemetry Libraries: Integrated OpenTelemetry Python libraries to instrument the application for telemetry data collection.
    • opentelemetry-api: Provides the API for tracing and metrics.
    • opentelemetry-sdk: The core SDK for OpenTelemetry in Python.
    • opentelemetry-instrumentation: General instrumentation library.
    • opentelemetry-instrumentation-sqlalchemy: Specific for instrumenting SQLAlchemy operations.
    • opentelemetry-instrumentation-tornado: For monitoring Tornado-based HTTP server activities.
  • Environment Variables: Added new environment variables (OTEL_EXPORTER_OTLP_HTTP_ENDPOINT and OTEL_EXPORTER_OTLP_ENDPOINT) for configuring the OpenTelemetry Collector endpoints.
  • Database and HTTP Server Instrumentation: Configured instrumentation for both SQLAlchemy database operations and Tornado HTTP server activities to gather traces and metrics.
  • Check out this doc for more details

by @mfreeman451 in #4476

🦊 Add GitLab authentication to version control

Gitlab

Support authenticating with GitLab in version control app.
The feature can be enabled by configuring the following environment variables:

  • GITLAB_HOST
  • GITLAB_CLIENT_ID
  • GITLAB_CLIENT_SECRET

by @dy46 in #4534

🔔 Discord notification integration

Discord

Support setting up Discord notification channel for alerts.
Check out the doc for details

by @luthfir-rahman in #4478

🔑 Add OIDC generic provider

This feature allows users to hook up their OIDC server (e.g. Keycloak ) and use it to authenticate with Mage.
It can be enabled by configuring the following environment variables:

  • OIDC_CLIENT_ID
  • OIDC_CLIENT_SECRET
  • OIDC_DISCOVERY_URL

by @dy46 in #4563

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

Full Changelog: 0.9.63...0.9.64

Contributors

mfreeman451, Luishfs, and 4 other contributors
Assets 2

0.9.63 | Halo 👾

08 Feb 04:06
@wangxiaoyou1993 wangxiaoyou1993
0.9.63
ba3714d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.63 | Halo 👾

What's Changed

🎉 Exciting New Features

🦆 MotherDuck Support


This one's for all the ducklings out there! In addition to supporting DuckDB, Mage now supports MotherDuck destinations!

By specifying a MOTHERDUCK_TOKEN and adding a md: prefix to your DuckDB database, you can read/write from/to MotherDuck locations! Check it out and get started here.

by @wangxiaoyou1993 in #4533

🤖 Support Thick mode in OracleDB


We like our OracleDB connections like we like our pancakes, THICK 🥞. By default, Mage's Oracle client runs in a "Thin" mode which connects directly to Oracle Database— this mode does not need Oracle Client libraries. However, some additional functionality is available when they're used.

Now, you can use the "Thick" mode in Mage to connect to OracleDB using the Oracle Client libraries!

Check out our docs to get started or read more about the differences between "Thin" and "Thick" modes here.

by @matrixstone in #4421

🗄️ Show unused block files in file browser

This is one of our most requested features and we're excited to announce that it's finally here! 🎉

You can now see which files are not being used in your pipeline! This is a great way to clean up your projects and remove any unnecessary files. Check out the gif below to see it in action!


Head over to our docs to learn more!

by @johnson-mage in #4449

🤐 Import functionality for pipeline zips

Like to share? So do we! You can now import pipelines (via .zip files) in your Mage projects! We're optimistic that this simple improvement will make it easier to share your pipelines or even borrow pipelines from your friends!


by @johnson-mage in #4453

🐛 Bug Fixes

💅 Enhancements & Polish

  • MSSQL data integration source - add support for DATETIMEOFFSET type by @hugabora in #4499
  • Fix: handle unprivileged user for postgres by @jdbranham in #4357
  • Optimize pipeline schedule queries by @dy46 in #4188
  • Remove pipeline's updated_at attribute by @johnson-mage in #4521
  • Create and view workspaces in different namespaces by @dy46 in #4513
  • Select status for pipeline runs that time out by @dy46 in #4519

New Contributors

Full Changelog: 0.9.62...0.9.63

Contributors

tommydangerous, hugabora, and 8 other contributors
Assets 2

0.9.62 | The Beekeeper 🐝

01 Feb 00:56
@wangxiaoyou1993 wangxiaoyou1993
0.9.62
a9cacb0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.62 | The Beekeeper 🐝

What's Changed

🎉 Exciting New Features

🧵 [Mage IO] Weaviate Integration

🤔 Building AI apps with Mage? Perfect! Now you can use Weaviate as a data source. Weaviate is an open-source, AI-native vector database that helps developers create intuitive and reliable AI applications. With Mage, you can now read from and write to Weaviate databases! Read more in our docs here.

by @matrixstone in #4158

🔍 [Mage IO] Alogia Integration

Like search? Us too! That's why we've added support for Alogia as a data source in Mage. Alogia is a powerful search engine that helps you build fast and accurate search experiences. With Mage, you can now read from and write to Alogia! Read more in our docs here.

by @matrixstone in #4198

💥 Dynamic SQL Blocks

Another big improvement to dynamic blocks this week SQL Dynamic Blocks! That's right, you can now create dynamic outputs from SQL blocks in Mage. Previously, blocks had to be Python for dynamic outputs, but no more! This is a big step forward in making Mage more flexible and powerful. Give it a shot today. 🎉

by @tommydangerous in #4430

🌊 [Kafka Streaming Sources] Offset & Partitions

For our streaming aficionados, we have a few new Kafka features! You can now specify the offset and partitions for Kafka streaming sources. Offsets can be one of: Beginning, End, Int, & Timestamp. This allows users to set specific positions inside a topic to consume data.

by @Luishfs in #4226

⛴️ Configure Kubernetes Affinity & Tolerations

Last, but certainly not least, we've got some nitty Kubernetes configuration updates! You can now specify affinity and tolerations in your Kubernetes settings. ⚓

Node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The rules are defined using custom labels on nodes and label selectors specified in pods. Node affinity allows a pod to specify an affinity (or anti-affinity) towards a group of nodes it can be placed on.

Tolerations are applied to pods and indicate that the pod can be scheduled on nodes with specific taints.

These configurations should help our Kubernetes power users make the most of Mage! 🧙

by @wangxiaoyou1993 in #4407

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

Full Changelog: 0.9.60...0.9.62

Contributors

tommydangerous, matrixstone, and 6 other contributors
Assets 2

0.9.60 | Yusuke Urameshi

19 Jan 17:45
@wangxiaoyou1993 wangxiaoyou1993
0.9.60
27beeeb
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.60 | Yusuke Urameshi

What's Changed

🎉 Exciting New Features

🌊 [Streaming] Google Cloud Storage Destination

🎉 Google Cloud users rejoice! Streaming pipelines just even got better— Mage now supports Google Cloud Storage as a streaming destination! Check out the docs here and get started today!

by @wangxiaoyou1993 in #4340

👷‍♂️ Overwrite SQL types

For anyone with a data warehouse, listen up! (We assume that's most of you 😅)

You can now specify custom column types when exporting to SQL destinations. This is useful when you want to export a dataframe with a column that has a type that is not supported by the default mapping. You can read more about overwriting types here.

Here's an example of an exporter that overwrites column types for a PostgreSQL destination:

@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
    schema_name = 'your_schema_name'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    overwrite_types = {'column_name': 'VARCHAR(255)'}

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
            allow_reserved_words=True,
            unique_conflict_method='UPDATE',
            unique_constraints=['col'],
            overwrite_types=overwrite_types,
        )

This feature is currently supported for PostgreSQL, Redshift, Trino, MSSQL, MySQL, Clickhouse, and BigQuery.

by @Luishfs in #3474

👨‍💻 [Command Center] Version Control & Files

The Mage Command Center can now be used for version control commands! You can both configure git and run your favorite version control commands directly from the Mage UI. Additionally, you can manage your files via nav and Mage's new file explorer (🧙 Arcane Library)!

As a reminder, to enable the command center, you can do so by going to Settings (the wizard icon) and click the Command Center toggle. It can be accessed via ⌘ + . (Mac) or Win + . (Windows).

by @tommydangerous in #4273

👾 [Command Center] Terminal App

Mage LEGEND @tommydangerous is back at it again— he's implemented a full terminal app into the command center. For those of you with this beta feature enabled, you'll now have a terminal at your fingertips at all times.

Check out the following video for sample usage:

CleanShot.2024-01-14.at.10.09.43.mp4

As a reminder, to enable the command center, you can do so by going to Settings (the wizard icon) and click the Command Center toggle. It can be accessed via ⌘ + . (Mac) or Win + . (Windows).

by @tommydangerous in #4365

JSON Logging

A huge shoutout to @dy46 for adding JSON logging to Mage! This will make it easier to parse logs and integrate with other logging tools. Just specify SERVER_LOGGING_FORMAT=json to change the output to something like the following:

by @dy46 in #4336

🐛 Bug Fixes

💅 Enhancements & Polish

Full Changelog: 0.9.59...0.9.60

Contributors

tommydangerous, Luishfs, and 3 other contributors
Assets 2

0.9.59 | 🐲 🐉 Year of the Dragon

10 Jan 20:43
@wangxiaoyou1993 wangxiaoyou1993
0.9.59
3dc128e
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.59 | 🐲 🐉 Year of the Dragon

What's Changed

🎉 Exciting New Features

Note: many new features this week are in beta. You can enable them by navigating to your Mage settings and toggling the beta features there.

🎮 Multi-project Platform [BETA]

We've reworked our support for mulitple projects with a new multi-project platform! @tommydangerous is back at it again with this huge feature release, enabling nested projects, custom code paths, cross-project triggering, a split pipeline scheduler, and much more!

CleanShot.2024-01-08.at.02.33.54.mp4

If you'd like to try out the multi-project platform, you can check out this repo for a sample structure. Head over to your Mage settings to enable

🚀 Command Center [BETA]

Another new & exciting feature this week— the Mage Command Center. The command center is a floating search bar that can invoke actions like opening files & pages, perform actions within Mage, interacting with the page, and much more!

CleanShot.2024-01-08.at.03.15.15.mp4

Enable the Command Center in settings and give it a spin today!

🪣 Bitbucket Version Control

Shout out to @dy46 for continuing to crush the version control integrations!

You can now use Bitbucket as a version control provider! This is a great option for teams that use Bitbucket for their code repositories. To get started, navigate to the Mage Version Control app and select Bitbucket as your provider. You'll be prompted to authenticate with Bitbucket, and then you'll be able to select your repositories. Read more here.

by @dy46 in #4176

◔ Qdrant integration

Mage now supports Qdrant, an open-source vector search engine. Qdrant is a great tool for similarity search, and it can be used for a variety of use cases, including product recommendations, image search, and more. With this update, you can load/export data from Qdrant sources in your batch pipelines! Read more here.

by @matrixstone in #4081

🧱 dbt DX v2 + dbt Upgrade [BETA]

This release contains a huge dbt overhaul 🤯

Alongside a much-awaited upgrade to dbt 1.7, the dbt developer experience has been completely rebuilt. @tommydangerous has been hard at work crafting a dbt experience that is more intuitive, powerful, and flexible!

Here's a quick demo:

CleanShot.2024-01-08.at.03.08.21.mp4

🐛 Bug Fixes

SQL blocks

Data integration

Trigger and scheduling

dbt

File browser

Dynamic blocks

Git

Other

Read more

Contributors

aliavni, eofs, and 9 other contributors
Assets 2

0.9.50 | Wonka 🎩

13 Dec 17:00
@mattppal mattppal
0.9.50
b0a3287
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.50 | Wonka 🎩

What's Changed

🎉 Exciting New Features

🌊 [Data Integration] Dremio Source


🥳 Dremio users, rejoice! Mage now supports Dremio as a data integration source, meaning you can now build data integration pipelines pulling from data lakes and more!

by @Luishfs in #4072

🏃‍♂️ Manually run pipeline once in same trigger

This update, courtesy of our frontend engineer, Johnson, is a big one!


First, he added a new button to the Trigger Detail page for running pipelines once in the same trigger! 👀

Next, some quality of life improvements:

  • For @once triggers, the trigger does not need to be active status before running the pipeline once using the Run@once button in the Trigger Detail page. Many users have told us this is confusing... No more!
  • The trigger will automatically be updated to active status. However, if the trigger is NOT an @once trigger (e.g. a recurring interval or API trigger), the trigger must be set to active status before manually running the pipeline once from the Trigger Detail page.

Finally, Johnson renamed the "start/pause" trigger on Trigger Detail page to "enable/disable" to be a bit more clear and communicative. Again, we've heard this is a bit misleading, so we did something about it! 🗣️

by @johnson-mage in #4133

[Streaming] ActiveMQ Sink


Shruti continues her epic tear of contributing magical ✨ pipelines. In this PR, she's added an ActiveMQ streaming sink to Mage. Apache ActiveMQ is an open source message broker written in Java... and now you can write data there via Mage! 💫

by @shrutimantri in #4141

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

Full Changelog: 0.9.48...0.9.50

Contributors

Abiodun, tommydangerous, and 4 other contributors
Assets 2