Roadmap & Community Meetup

[joshua-stauffer]

Great Expectations is back!

Great Expectations has a new home: dbt/Fivetran has taken stewardship of the library and is investing in its future. Over the last few years, OSS development slowed as GX Labs focused on GX Cloud, their commercial offering. With dbt/Fivetran's backing, GX Core now has dedicated, full-time maintainership, and the roadmap is community-focused.

We want to hear from you!

The future of Great Expectations is one in which the community plays an essential part in steering the product.

So if you use Great Expectations, we want to hear from you, either: contributing to this discussion, and/or joining the community feedback session later this week.

Tip

The GX Community Meetup is back!

Register here

WHEN: Thursday, August 20, 11:00 ET / 17:00 CEST
WHO: Josh Stauffer (Great Expectations maintainer)
WHAT: Jump on a Zoom call with us to:

• catch up on these changes,
• see a demo of upcoming features, and
• weigh in on the roadmap.

Can't make it? The session will be recorded and shared back here on this discussion.

What's new?

Since the project transferred to Fivetran/dbt, there have been three minor releases, with the latest, 1.20.0, released on August 7. Below is a highlight of the work that's shipped.

Ready for external contributions!

Contribution docs have been thoroughly revamped. Check out CONTRIBUTING.md to learn more. We welcome any and all thoughtful pull requests and issues!

Stabilization

The first priority on the roadmap is hardening existing capabilities. The GX community has documented many gaps over the years, some of which were simply not prioritized.

Reviewing all open issues, as well as many that were closed as "won't fix", highlighted some improvement opportunities, such as:

• expanded test suite (in progress)
• shoring up some "experimental" data sources so that they can be officially supported and tested

Native agent skills!

As part of the 1.0.0 release, Data Assistants and Profilers were removed, with the promise that they'd return in the future. That time is now — GX 1.21.0 ships with agent skills, packaged workflows that let users configure Data Sources and Expectations with their LLM of choice.

Soon we'll ship support for automatically generating Expectations from your data.

Coming soon

The return of Custom Expectations

GX v1 removed the capability to write custom Expectations, in part because the underlying metrics layer has significant complexity. Future work will use the new metrics API to reintroduce them, allowing users to freely define their own metrics and Expectations as plugins.

And more!

There are many other parts of the library that we're looking forward to improving, including:

• Data Docs
• Validation Results
• broader Data Source support, and
• more supported Expectations.

[AlmeidaJoao]

This update made my day. :).