v0.8.42

v0.8.42

Highlights

User Experience

• Improved Search Experience - preview cards now display usage and freshness information
• Update to Schema History - incorporated Community feedback to remove “Blame” terminology
• Improved UI-Based Ingestion - easily configure metadata ingestion from Snowflake, BigQuery, Looker, and Tableau with an easy-to-follow form; YAML is still supported!

Developer/Community Experience

• Python 3.6 is no longer supported for ingestion – we expect this to impact fewer than 1% of DataHub users (based on PyPi download stats). Please upgrade to Python 3.7 or newer
• Update to GitHub Issue management - issues will be marked as “Inactive” after 30 days of no activity and will be automatically closed following an additional 30 days of inactivity
• We’ve updated our Slack Guidelines! Read them here

Metadata Ingestion

• You can now test your Snowflake connection via the CLI and UI-based Ingestion to ensure you have proper access levels required for general ingestion, profiling, and usage. We will be expanding this functionality to other cloud-based ingestion sources in upcoming cycles.
• Hard delete will now discover and remove soft deleted entities
• Resolved issue of assertion error with dbt stateful ingestion

Full Commit Log

What's Changed

• feat(quickstart,docs): updates for v0.8.41 by @anshbansal in #5409
• fix(ingest): ensure upgrade checks run async by @shirshanka in #5383
• fix(ingest): pass transport options to usage history looker api calls by @mayurinehate in #5417
• feat(quickstart): moving to official confluent images for m1 by @shirshanka in #5416
• fix(documentation) Fix erratic cursor in documentation editor bug by @chriscollins3456 in #5411
• feat(ui): Supporting enriched search preview + misc improvements by @jjoyce0510 in #5419
• chore: remove unnecessary modules from codebase by @shirshanka in #5420
• fix(ingest): extract usage for dashboards allowed by pattern by @mayurinehate in #5424
• fix(docker): fix kafka-setup command to support same capabilities as … by @shirshanka in #5428
• fix(protobuf): ownership fixes by @leifker in #5425
• fix(ui): add dataset qualifiedName parameter to lineage query by @alexey-kravtsov in #5427
• fix(glossary) Fix dropdown where disabled buttons are still clickable by @chriscollins3456 in #5430
• docs(bigquery): add changelog and unittest for profiling limits by @MugdhaHardikar-GSLab in #5407
• fix(siblings): fixing lineage fetching for siblings & sources by @gabe-lyons in #5415
• fix(ui): Fixing unreleased search preview bugs by @jjoyce0510 in #5432
• feat(ui): Adding Statistics Summary to Dataset + Dashboard Profiles by @jjoyce0510 in #5440
• feat(ingest): add test source connection feature, structured report file by @shirshanka in #5442
• fix(ingest/glue): handle error when generating s3 tags for virtual view tables by @timcosta in #5398
• feat(ingest): model - adding a small extension to support communicati… by @shirshanka in #5429
• fix(bigquery-usage): fix dataset name for sharded table by @MugdhaHardikar-GSLab in #5412
• feat(ingestion) Add new endpoint to test an ingestion connection by @chriscollins3456 in #5438
• feat(cli,build): remove deprecated variables GMS_HOST/_PORT by @anshbansal in #5451
• fix(search): make filters by default an empty list if null by @aditya-radhakrishnan in #5454
• fix(hive): add column comment as a column description by @MugdhaHardikar-GSLab in #5449
• feat(groups): add native groups concept to DataHub by @aditya-radhakrishnan in #5443
• fix(ingest): fix serialization of report to handle nesting by @shirshanka in #5455
• fix(tableau): fix tableau db error, add more logs by @mayurinehate in #5423
• build(deps): bump terser from 5.9.0 to 5.14.2 in /docs-website by @dependabot in #5448
• feat(doc): spark-lineage - Adding spark lineage configuration doc for Amazon EMR by @treff7es in #5459
• feat(schema-history): remove blame language for the schema history feature by @aditya-radhakrishnan in #5457
• Search header: Menu icon alignment by @Ankit-Keshari-Vituity in #5458
• build(deps): bump terser from 4.8.0 to 4.8.1 in /datahub-web-react by @dependabot in #5446
• feat(ingest): snowflake - basic test connection capability by @shirshanka in #5464
• fix(ingest/trino): Avoid exception if $properties table empty or not readable by @glinmac in #5447
• feat(ingest): preflight - Add way to check/upgrade brew package version in preflight if needed by @treff7es in #5435
• fix(build): add base image with gradle wrapper cached by @anshbansal in #5467
• doc(bigquery): groups grants by requirements by @sgomezvillamor in #5468
• fix(docs,build): remove base image not needed, cleanup docs by @anshbansal in #5469
• feat(ui): Partial support for Chart usage by @jjoyce0510 in #5473
• fix(ingest): bigquery: multiproject profiling fix by @treff7es in #5474
• fix(ingest): kafka - revert deps back to < 1.9.0 by @shirshanka in #5476
• feat(docker): support multiplatform image for datahub-upgrade by @shirshanka in #5477
• feat(quickstart): experimental support for backup restore for quickstart by @shirshanka in #5418
• feat(dbt): updating source lineage logic by @gabe-lyons in #5414
• Ingestion: Added form in Big Query type to edit the queries. by @Ankit-Keshari-Vituity in #5431
• docs: fix docsearch config by @hsheth2 in #5479
• Search Results: Added checkbox option to select multiple results at once. by @Ankit-Keshari-Vituity in #5422
• feat(delete): hard delete deletes soft deleted entities by @anshbansal in #5478
• fix(docs): add missing closing marker for note section by @shirshanka in #5480
• fix(build): intermittent failure in github actions by @anshbansal in #5452
• feat(model, ingest): add user email in dashboard user usage counts by @mayurinehate in #5471
• feat(ingest): add support for capability report in snowflake test connection by @mayurinehate in #5472
• feat(build): automatically mark issues as stale to close inactive issues by @anshbansal in #5482
• fix(ingest): loosen confluent-kafka dep requirement by @hsheth2 in #5489
• refactor(ingest): cleanup importlib.import_module calls by @hsheth2 in #5490
• build(ingest): make gradle build less chatty by @hsheth2 in #5491
• fix(ingest): Fixing dbt trino datatypes by @aezomz in #5379
• refactor(ci): use custom action for checking codegen status by @hsheth2 in #5493
• feat(spark-lineage): Support ssl cert disable functionality by @MugdhaHardikar-GSLab in #5488
• docs(auth): fix link to point to new doc by @anshbansal in #5501
• docs(updating-datahub): add note for breaking change in looker usage … by @mayurinehate in #5499
• fix(ingest): cleanup unused flake8 noqa statements by @hsheth2 in #5492
• refactor(ci): refactor Docker build-and-push workflows by @hsheth2 in #5494
• docs(slack) Update to Slack guidelines by @maggiehays in #5504
• feat(cli): delete - make delete easier to use by @anshbansal in #5485
• fix(ingest): use temp dir for file generated during test by @hsheth2 in #5505
• feat(ui) Show Glossary and Domains header links to everyone by @chriscollins3456 in #5506
• Home Page: Flickering Issue on search input field by @Ankit-Keshari-Vituity in #5503
• fix(ingest): respect rest emitter timeout setting by @hsheth2 in #5508
• Search Page: Flickering Issue on search input field by @Ankit-Keshari-Vituity in #5515
• Ingestion: Added form to Looker and Tableau by @Ankit-Keshari-Vituity in #5487
• feat(identity): update azure and okta connectors to emit Origin aspects by @aditya-radhakrishnan in #5495
• feat(ui): Adding Search Select feature(frontend only) by @jjoyce0510 in #5507
• test(ingest): limit GMS retries in test by @hsheth2 in #5509
• fix(ingest)- airflow: update subdag check for compatibility with older Airflow versions by @vcs9 in #5523
• fix(ingest): fix unbound variable bug in cli ingest list-runs by @hsheth2 in #5527
• fix(ui) Display Term Group name properly in Recently Viewed by @chriscollins3456 in #5528
• feat(ingestion) Add frontend connection test for Snowflake by @chriscollins3456 in #5520
• fix(glossary) Fix Glossary success messages and sort Glossary by @chriscollins3456 in #5533
• feat(apache-ranger): Apache Ranger Authorizer support in datahub-gms by @mohdsiddique in #4999
• feat(ingest): add deprecation warning for Python 3.6 by @hsheth2 in #5519
• docs(townhall) add past townhall agendas by @maggiehays in #5536
• feat(ingestion): add groups to ldap users by @alexey-kravtsov in #5470
• chore(issues): reduce time for issues to be marked stale and then closed by @aditya-radhakrishnan in #5537
• fix(ingestion) Set pipeline_name on UI recipes with forms by @chriscollins3456 in #5535
• fix(ui): Correctly hit /logOut on logout by @jjoyce0510 in #5538
• [analytics-tab] fix analytics tab config variable for gms by @salihcaan in #5529
• feat(ui): Support batch adding / remove tags from search lists. (Batch Actions part 2/7) by @jjoyce0510 in #5534
• fix(ingestionSource): improve error experience when ingestion source is in an inconsistent state by @RyanHolstien in #5522
• docs(schema history): Correct typo in schema history documentation by @skylersinclair in #5545
• fix(docker): Fixing dev docker and quickstart by @jjoyce0510 in #5550
• feat(ui): Support Batch adding and removing Glossary Terms (Batch Actions 3/7) by @jjoyce0510 in #5544
• feat(ci): test quickstart works by @anshbansal in #5518
• test(ingest): mark trino/hana tests as xfail due to flakes by @hsheth2 in #5549
• feat(ingestion/superset): add display_uri to config by @milimetric in #5408
• fix(quickstart): failure on a path not being present by @anshbansal in #5554
• fix(dbt): fix issue of assertion error when stateful ingestion is used with dbt tests by @mohdsiddique in #5540

New Contributors

• @timcosta made their first contribution in #5398
• @glinmac made their first contribution in #5447
• @skylersinclair made their first contribution in #5545
• @milimetric made their first contribution in #5408

Full Changelog: v0.8.41...v0.8.42