Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release v0.15.0 (#1509) #1510

Merged
merged 72 commits into from
May 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
5711a6d
dbt_runner: add retry command to dbt runner
haritamar Dec 14, 2023
07d64a9
update package version (#1439)
NoyaArie Feb 29, 2024
87d47a9
Added singular tests to groups API.
elongl Jan 15, 2024
5c7d93c
Added 'original_path' to 'test_results[].metadata'.
elongl Mar 3, 2024
0117224
Test fixes.
elongl Mar 3, 2024
f29b2aa
Merge pull request #1372 from elementary-data/ele-2361-show-singular-…
elongl Mar 3, 2024
90cee26
Fix disable samples for alerts
IDoneShaveIt Mar 4, 2024
43b7fb5
Merge pull request #1443 from elementary-data/ele-2617-disable-sample…
IDoneShaveIt Mar 4, 2024
1993304
docs-roni-changes (#1445)
ronisaban2024 Mar 6, 2024
13ebbb3
Update README.md (#1446)
Maayan-s Mar 6, 2024
9f47652
Update README.md (#1448)
Maayan-s Mar 6, 2024
0842287
Fixed update_sent_alerts macro
svdimchenko Mar 6, 2024
031aa90
Ele 2626 support singular tests in test results runs (#1449)
NoyaArie Mar 6, 2024
792ddc6
Merge branch 'master' into fix/athena-update-alerts-v2
svdimchenko Mar 6, 2024
c314e0c
change original path to optional (#1454)
NoyaArie Mar 12, 2024
7cfc8a6
Ele 2677 add normalized path to tests (#1458)
NoyaArie Mar 13, 2024
17e2a84
Merge branch 'master' into fix/athena-update-alerts-v2
svdimchenko Mar 14, 2024
3fd8723
Merge pull request #1451 from svdimchenko/fix/athena-update-alerts-v2
IDoneShaveIt Mar 18, 2024
dbf177a
Remove poetry.lock from the package
IDoneShaveIt Mar 18, 2024
ab22381
Added poetry.lock to gitignore
IDoneShaveIt Mar 18, 2024
af4fcd7
Merge pull request #1464 from elementary-data/ele-2721-update-python-…
IDoneShaveIt Mar 18, 2024
fcdd4ac
ELE-2722 - support popuate_data argument to control alerts data popul…
IDoneShaveIt Mar 18, 2024
44796ab
Merge pull request #1466 from elementary-data/ele-2722-support-popula…
IDoneShaveIt Mar 18, 2024
ec7f163
Add @svdimchenko, @ronisaban2024, @dongchris, @sanromeo, @AvivZikelDY…
Maayan-s Mar 18, 2024
477f701
ELE-2725 - remove unused elementary database and schema from data mon…
IDoneShaveIt Mar 18, 2024
7f048c6
Merge pull request #1467 from elementary-data/ele-2725-remove-unused-…
IDoneShaveIt Mar 18, 2024
d8b391b
Update dbt-package for change in result rows
noaKurman Mar 13, 2024
f401eb2
Merge pull request #1468 from elementary-data/ele-2648-change-result-…
noaKurman Mar 19, 2024
39588ff
ELE-2726 - support days back for alerts population - default is 1
IDoneShaveIt Mar 19, 2024
03b340c
Merge branch 'master' of github.com:elementary-data/elementary into e…
IDoneShaveIt Mar 19, 2024
2db425d
Better time comparison to support future index
IDoneShaveIt Mar 19, 2024
9214155
Added logs for debugging
IDoneShaveIt Mar 19, 2024
299d0ea
Removed logs for debugging
IDoneShaveIt Mar 19, 2024
b39851b
Merge pull request #1470 from elementary-data/ele-2726-default-popula…
IDoneShaveIt Mar 19, 2024
7ab85a8
Updated table of contents (#1475)
ronisaban2024 Mar 24, 2024
604433b
Ele 2785 remove retry package (#1478)
NoyaArie Mar 25, 2024
73cd9e9
Merge branch 'master' into ele-2171-support-dbt_retry-in-dbt-runner
haritamar Mar 25, 2024
ca74ed8
Merge pull request #1331 from elementary-data/ele-2171-support-dbt_re…
haritamar Mar 26, 2024
6f0c70b
ELE-2918 - add user_select and static_select to Slack message builder
IDoneShaveIt Apr 11, 2024
d643573
Remove pre release installation of dbt from test-warehouse workflow
IDoneShaveIt Apr 11, 2024
b5b4821
Merge pull request #1486 from elementary-data/ele-2918-extend-slack-m…
ellakz Apr 11, 2024
6eab02f
Remove unused code of test management
IDoneShaveIt Apr 11, 2024
3e70570
Updated dbt package.
elongl Apr 11, 2024
c22a061
Merge pull request #1487 from elementary-data/ele-2934-fix-normalized…
elongl Apr 11, 2024
11866f5
Add actions block to Slack message builder
IDoneShaveIt Apr 11, 2024
831fe48
Refactor Slack integration and Slack Alert message builder to make th…
IDoneShaveIt Apr 14, 2024
09becbc
remove test subtype from most alerts (#1485)
ellakz Apr 15, 2024
a52ba6b
ELE-489 Support dbt-trino connector (#1378)
Tomme Apr 15, 2024
353f735
Added test run execution time to report json
ofek1weiss Apr 24, 2024
6b2a48f
Filter exposures with no upstream tables in lineage.
elongl Apr 25, 2024
c3d7a7d
Filtering upstream by models and sources.
elongl Apr 25, 2024
cb29018
Merge pull request #1501 from elementary-data/ele-2990-filter-exposur…
elongl Apr 25, 2024
8b52c66
Filter loops in exposures.
elongl Apr 25, 2024
a56693e
Merge pull request #1502 from elementary-data/ele-2990-filter-exposur…
elongl Apr 25, 2024
c647675
Empty lists are excluded.
elongl Apr 28, 2024
9befab8
Merge pull request #1503 from elementary-data/ele-2990-filter-exposur…
elongl Apr 28, 2024
6cd025c
changed execution time to be based on dbt_run_results
ofek1weiss Apr 28, 2024
72c7e08
Merge branch 'master' into ele-2917-be-pass-test-duration-to-test-runs
ofek1weiss Apr 28, 2024
6eba38f
Merge pull request #1500 from elementary-data/ele-2917-be-pass-test-d…
ofek1weiss Apr 28, 2024
975ac55
added last and median exec times to test runs
ofek1weiss May 1, 2024
bf27335
Merge pull request #1504 from elementary-data/ele-3000-add-test-media…
ofek1weiss May 1, 2024
ec415cb
ELE-2933: Column anomalies group_by - CLI (#1499)
dapollak May 2, 2024
38fdf25
release/0.15.0 (#1507)
dapollak May 2, 2024
d7b1604
release v0.15.0 (#1509)
dapollak May 2, 2024
f5509e1
Fix notifcation
noaKurman May 5, 2024
056e9a5
Merge pull request #1512 from elementary-data/ele-3021-fix-notification
noaKurman May 5, 2024
d6d8b10
bugfix to BQ alerts (#1516)
haritamar May 8, 2024
7cc9b23
Ele 3035 docs to master (#1518)
haritamar May 8, 2024
e3d767a
remove refs to information_schema_columns (#1519)
haritamar May 8, 2024
7b17648
Ele 3040 update cli to use new package version (#1521)
haritamar May 9, 2024
de1c02d
release v0.15.1 (#1522)
github-actions[bot] May 9, 2024
2e0a0a6
ELE-3044: New dimensions features docs (#1520)
dapollak May 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test-github-action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ jobs:
always() &&
! cancelled() &&
! contains(needs.test.result, 'success') &&
! contains(needs.test.result, 'cancelled')
! contains(needs.test.result, 'cancelled') &&
${{ github.event_name == 'schedule' }}
uses: ./.github/workflows/notify_slack.yml
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-warehouse.yml
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ jobs:

- name: Install dbt
run: >
pip install --pre
pip install
"dbt-core${{ inputs.dbt-version && format('=={0}', inputs.dbt-version) }}"
"dbt-${{ (inputs.warehouse-type == 'databricks_catalog' && 'databricks') || inputs.warehouse-type }}${{ inputs.dbt-version && format('<={0}', inputs.dbt-version) }}"

Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ var/
*.mypy_cache/
logs/
.venv
poetry.lock
package-lock.yml

# PyInstaller
# Usually these files are written by a python script from a template
Expand Down
90 changes: 50 additions & 40 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,6 @@
<h2 align="center">
dbt native data observability for analytics & data engineers
</h2>
<h4 align="center">
Monitor your data quality, operation and performance directly from your dbt project.
</h4>

<p align="center">
<a href="https://www.elementary-data.com/community"><img src="https://img.shields.io/badge/join-Slack-ff69b4"/></a>
Expand All @@ -25,77 +22,85 @@ Monitor your data quality, operation and performance directly from your dbt proj

## What is Elementary?

Elementary is an open-source data observability solution, built for dbt users. Setup in minutes, gain immediate visibility, detect data issues, send actionable alerts, and understand impact and root cause.
Elementary is a dbt-native data observability solution for data and analytics engineers.
Set up in minutes, gain immediate visibility, detect data issues, send actionable alerts, and understand impact and root cause.
Available as self-hosted or Cloud service with premium features.

#### Table of Contents

- [Features](#features)
- [Self-hosted vs. Cloud service](#self-hosted-vs-cloud)
- [Get Started](#get-started)
- [Community & Support](#community--support)
- [Contribution](#contributions)

<kbd align="center">
<a href="https://storage.googleapis.com/elementary_static/elementary_demo.html"><img align="center" style="max-width:300px;" src="static/report_ui.gif"> </a>
</kbd>

<br>

## Quick start
## Features

Step 1 - [Install Elementary dbt package](https://docs.elementary-data.com/quickstart)
<img src="static/happy_fire.ico" width="16"/> **Anomaly detection tests** - Collect data quality metrics and detect anomalies, as native dbt tests.

Step 2 - [Install Elementary CLI](https://docs.elementary-data.com/quickstart-cli)
<img src="static/happy_fire.ico" width="16"/> **Automated monitors** - Out-of-the-box cloud monitors to detect freshness, volume and schema issues.

## Features
<img src="static/happy_fire.ico" width="16"/> **End-to-End Data Lineage** - Enriched with the latest test results, for impact and root cause analysis of data issues. Elementary Cloud offers **Column Level Lineage** and **BI integrations**.

<img src="static/happy_fire.ico" width="16"/> **Data observability report** - Generate a data observability report, host it or share with your team.
<img src="static/happy_fire.ico" width="16"/> **Data quality dashboard** - Single interface for all your data monitoring and test results.

<img src="static/happy_fire.ico" width="16"/> **Anomaly detection dbt tests** - Collect data quality metrics and detect anomalies, as native dbt tests.
<img src="static/happy_fire.ico" width="16"/> **Models performance** - Monitor models and jobs run results and performance over time.

<img src="static/happy_fire.ico" width="16"/> **Test results** - Enriched with details for fast triage of issues.
<img src="static/happy_fire.ico" width="16"/> **Configuration-as-code** - Elementary configuration is managed in your dbt code.

<img src="static/happy_fire.ico" width="16"/> **Models performance** - Visibility of execution times, easy detection of degradation and bottlenecks.
<img src="static/happy_fire.ico" width="16"/> **Alerts** - Actionable alerts including custom channels and tagging of owners.

<img src="static/happy_fire.ico" width="16"/> **Data lineage** - Enriched with test results, easy to navigate and filter.
<img src="static/happy_fire.ico" width="16"/> **Data catalog** - Explore your datasets information - descriptions, columns, datasets health, etc.

<img src="static/happy_fire.ico" width="16"/> **dbt artifacts uploader** - Save metadata and run results as part of your dbt runs.

<img src="static/happy_fire.ico" width="16"/> **Slack alerts** - Actionable alerts, including custom channels and tagging of owners and subscribers.

##

Join [Slack](https://www.elementary-data.com/community) to learn more on Elementary.

Our full documentation is [available here](https://docs.elementary-data.com/).

## How it works?
Join [Slack](https://www.elementary-data.com/community) to learn more about Elementary.

Elementary [dbt package](https://github.com/elementary-data/dbt-data-reliability) creates tables of metadata and test results in your data warehouse, as part of your dbt runs. The CLI tool reads the data from these tables, and is used to generate the UI and alerts.
## Self-hosted vs. Cloud

<img align="center" style="max-width:300px;" src="static/how_elementary_works.png">
### Elementary Cloud

## Community & Support
Ideal for teams monitoring mission-critical data pipelines, requiring guaranteed uptime and reliability, short-time-to-value, advanced features, collaboration, and professional support.
The solution is secure by design, and requires no access to your data from cloud.

For additional information and help, you can use one of these channels:
To learn more, [book a demo](https://cal.com/maayansa/elementary-intro-github) or see our [pricing page](https://www.elementary-data.com/pricing).

- [Slack](https://www.elementary-data.com/community) \(Live chat with the team, support, discussions, etc.\)
- [GitHub issues](https://github.com/elementary-data/elementary/issues) \(Bug reports, feature requests)
- [Twitter](https://twitter.com/ElementaryData) \(Updates on new releases and stuff)
### Self-hosted

## **Integrations**
Elementary Community is an open-source CLI tool you can deploy and orchestrate to send Slack alerts and self-host the Elementary report. It is best for data and analytics engineers that require basic observability capabilities or for evaluating features without vendor approval. Our community can provide great support on [Slack](https://www.elementary-data.com/community) if needed.

- [x] **dbt core (>=1.0.0)** ![](static/dbt-16.png)
- [x] **dbt cloud** ![](static/dbt-16.png)
## Get Started

**Data warehouses:**
### Elementary Cloud

- [x] **Snowflake** ![](static/snowflake-16.png)
- [x] **BigQuery** ![](static/bigquery-16.svg)
- [x] **Redshift** ![](static/redshift-16.png)
- [x] **Databricks SQL** ![](static/databricks-16.png)
- [x] **Postgres** ![](static/postgres-16.png)
1 - [Create an account](https://elementary-data.frontegg.com/oauth/account/sign-up)

**Operations:**
2 - [Install Elementary dbt package](https://docs.elementary-data.com/cloud/onboarding/quickstart-dbt-package)

- [x] **Slack** ![](static/slack-16.png)
- [x] **GitHub Actions** ![](static/github-actions-16.png)
- [x] **Amazon S3** ![](static/s3-16.svg)
- [x] **Google Cloud Storage** ![](static/gcs-16.png)
3 - [Connect Elementary Cloud](https://docs.elementary-data.com/cloud/onboarding/signup)

Ask us for integrations on [Slack](https://www.elementary-data.com/community) or as a [GitHub issue](https://github.com/elementary-data/elementary-lineage/issues/new).
### Elementary Community

1 - [Install Elementary dbt package](https://docs.elementary-data.com/quickstart)

2 - [Install Elementary CLI](https://docs.elementary-data.com/quickstart-cli)

## Community & Support

For additional information and help, you can use one of these channels:

- [Slack community](https://www.elementary-data.com/community) \(Release announcements, community support, discussions, etc.\)
- [GitHub issues](https://github.com/elementary-data/elementary/issues) \(Bug reports, feature requests)

## **Contributions**

Expand Down Expand Up @@ -196,6 +201,11 @@ Check out the [contributions guide](https://docs.elementary-data.com/general/con
<a href="https://github.com/suelai"><img src="https://avatars.githubusercontent.com/u/24508576?v=4" width="50" height="50" alt=""/></a>
<a href="https://github.com/aibazhang"><img src="https://avatars.githubusercontent.com/u/29817697?v=4" width="50" height="50" alt=""/></a>
<a href="https://github.com/samirS97"><img src="https://avatars.githubusercontent.com/u/51163297?v=4" width="50" height="50" alt=""/></a>
<a href="https://github.com/svdimchenko"><img src="https://avatars.githubusercontent.com/u/39801237?v=4" width="50" height="50" alt=""/></a>
<a href="https://github.com/ronisaban2024"><img src="https://avatars.githubusercontent.com/u/162031979?v=4" width="50" height="50" alt=""/></a>
<a href="https://github.com/dongchris"><img src="https://avatars.githubusercontent.com/u/86695140?v=4" width="50" height="50" alt=""/></a>
<a href="https://github.com/sanromeo"><img src="https://avatars.githubusercontent.com/u/44975602?v=4" width="50" height="50" alt=""/></a>
<a href="https://github.com/AvivZikelDY"><img src="https://avatars.githubusercontent.com/u/111749106?v=4" width="50" height="50" alt=""/></a>

<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->
Expand Down
8 changes: 8 additions & 0 deletions docs/_snippets/alerts/alerts-configuration.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
<CardGroup cols={2}>
<Card title="Alerts rules" icon="cloud" iconType="solid" href="/cloud/guides/alert-rules">
Use Alert Rules to distribute your alerts to the right channels.
</Card>
<Card title="Alerts configuration in code" icon="code" iconType="solid" href="/cloud/guides/alerts-configuration">
Alert content and properties configuration as code in your project YML files.
</Card>
</CardGroup>
11 changes: 4 additions & 7 deletions docs/_snippets/alerts/alerts-introduction.mdx
Original file line number Diff line number Diff line change
@@ -1,18 +1,15 @@
Elementary can send alerts about:

- Failures and/or of dbt tests
- Failures and/or Elementary tests
- Failures and/or warnings of dbt tests
- Failures and/or warnings Elementary tests
- Model runs failures
- Source freshness issues

The alerts can be sent to Slack or Microsoft Teams, and we are working on adding more channels.
Please [reach out to us](https://elementary-data.com/community) if you have a specific destination in mind.
- Source freshness failures

You can enrich your alerts by adding properties to tests and models in your `.yml` files.
The supported attributes are: owner, subscribers, description, tags.

You can configure and customize your alerts by configuring:
custom channel, suppression interval, alert fields (for test alerts only), alert grouping, alert filters.
Custom channel, suppression interval, alert filters, etc.

<Frame>
<div className="dark:bg-white rounded-md p-4">
Expand Down
Loading
Loading