v0.13.2

Hotfix Release

Fixes MCL message deserialization bug when using internal schema registry and running specific upgrade jobs.

policyFields (enabled by default):
BOOTSTRAP_SYSTEM_UPDATE_POLICY_FIELDS_ENABLED:true

dataJobNodeCLL (disabled by default):
BOOTSTRAP_SYSTEM_UPDATE_DATA_JOB_NODE_CLL_ENABLED:false

Example Error:

Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id 1
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 13 out of bounds for length 2
        at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:460)
        at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:283)
        at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:188)
        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:161)
        at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:260)
        at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:248)

Recovery Directions:

If currently affected, please remove the topic prior to upgrading to v0.13.2 to remove the corrupted message. The default topic name is MetadataChangeLog_Versioned_v1 however if you've customized the topic name be sure to remove that topic.

If running kafka per the example Helm chart for prerequisites the following command will delete the topic.

kubectl exec -it prerequisites-kafka-broker-0 -c kafka -- kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic MetadataChangeLog_Versioned_v1

Full Changelog: v0.13.1...v0.13.2

v0.13.1

DataHub Release Notes

User Experience

• Capture and Manage Common Joins between Datasets: Users can now view and manage common join relationships between datasets, making it easier than ever to capture best practices and bespoke join logic. Watch the walkthrough here! 8325
  • Head's up: you'll need to enable the ER_MODEL_RELATIONSHIP_FEATURE_ENABLED env variable to use this feature!
• Enhanced UI Interactions: Users can now enjoy an improved markdown editor and filter policies by active/inactive statuses, resulting in a more intuitive and manageable interface. 9949, 9958
• Visual Context for Groups: You can now include picture links for groups in the UI, adding a richer visual context and enhancing the navigational experience. 9882
• Improved Error Visibility: The UI now displays error messages related to data size limitations, allowing for better troubleshooting and user experience. 10038

Developer Experience

• Enhanced Kafka Compatibility: Updated client version for Kafka setup ensures better compatibility and functionality for developers. 9962
• Optimized Docker Build: Docker setups now respect pip mirrors, optimizing the build process especially in restricted network environments. 9963
• Advanced Error Handling: New error handling for duplicate class names and improved fspath lint error management enhance the code reliability and quality. 9960, 9976
• Latest OpenSearch Image: Incorporation of OpenSearch image version 2.11.0 aligns with the latest stable releases, boosting performance and security. 9984

Metadata Ingestion

• NEW: Dagster Integration: You can now seamlessly ingest your Dagster Pipelines, Jobs, Ops, and lineage into DataHub. 10071
• Expanded Field Classification Support: This release introduces support for field-level classification during ingestion for Redshift, BigQuery, DynamoDB, and SQL Sources. 10013, 10031
• Enhanced Ingestion Capabilities: DataHub now offers stateful ingestion by default, optimizing routines for REST sinks and improving metadata accuracy across diverse sources like dbt and BigQuery. 9934, 10158, 10080
• Better Data Lineage: This release introduced support for Openlineage in service of the Spark Lineage Beta Plugin; additionally, we now support incremental Column-Level Lineage, improving the accuracy of detecting column-level relationships during ingestion.9870, 9967, 10090
• Schema Clarity: New descriptions support for JSON schema arrays and a mechanism to escape special characters in BigQuery table descriptions aid in clearer schema validation and ingestion processes. Databricks ingestion now supports Hive Metastore schemas with special characters. 9757, 9932, 10049

Version Upgrades

• Kafka client and OpenSearch image were updated to the latest versions.

Breaking Changes

This release introduces default settings for stateful ingestion and updates in handling dbt ingestion. For details on all breaking changes, view the full documentation here.

Contributors

MASSIVE shoutout to our contributors!

First-Time Contributors

akarsh991, alexs-101, AvaniSiddhapuraAPT, diegmonti, dushayntAW, filipe-caetano-ovo, HuanjieGuo, jayacryl, k7ragav, kopax-polyconseil, LePuppy, Nelvin73, pinakipb2, poorvi767, rae89, trialiya, valeral.

Repeat Contributors

ANich, shubhamjagtap639, sgomezvillamor, siladitya2, skrydal, sumitappt, Masterchen09, mayurinehate, ngamanda, gaurav2733, githendrik, jayasimhankv.

DataHub Maintainers

anshbansal, asikowitz, chriscollins3456, darnaut, david-leifker, eboneil, ethan-cartwright, gabe-lyons, hsheth2, pedro93, RyanHolstien, treff7es, yoonhyejin.

What's Changed

• bump(kafka-setup): client version bump by @david-leifker in #9962
• feat(ingest): throw codegen error on duplicate class names by @hsheth2 in #9960
• feat(docker): respect pip mirrors with uv by @hsheth2 in #9963
• Openlineage endpoint and Spark Lineage Beta Plugin by @treff7es in #9870
• fix(ingest/json-schema): adding support descriptions for array by @AvaniSiddhapuraAPT in #9757
• fix(ingest/redshift): fix bug in lineage v2 table renames by @hsheth2 in #9967
• feat(ingest): speed up to_obj() and validate() by @hsheth2 in #9969
• feat(ingest): fix fspath lint error by @hsheth2 in #9976
• docs: archive old version before 0.12.0 & fix broken links by @yoonhyejin in #9957
• fix(ui/markdown-editor): arrows change field when editing description… by @gaurav2733 in #9949
• feat(ui/policies): add filter for Active/Inactive/All on policy page by @gaurav2733 in #9958
• feat(ui): add option to add picture link for groups by @akarsh991 in #9882
• feat(ingest): add Looks subtype + stop reemitting browsePathV2 by @hsheth2 in #9978
• fix(ingest/bigquery): escape special characters for table descriptions by @AvaniSiddhapuraAPT in #9932
• feat(ui): add loading spin to access management table by @filipe-caetano-ovo in #9974
• fix(ingestion/fivetran): Fix fivetran get connector jobs bug by @shubhamjagtap639 in #9975
• feat(ingest/dbt): generate CLL for all node types by @hsheth2 in #9964
• chore(search): bump OpenSearch image version to 2.11.0 by @darnaut in #9984
• feat(ingest): enable stateful_ingestion by default for DataHub rest sink by @shubhamjagtap639 in #9934
• feat(ingestion/cli): Adding check option to validate allow/deny and path_specs by @treff7es in #9983
• fix(ingest): only import PathSpec when necessary by @hsheth2 in #9989
• feat(config): add configuration to reprocess UI sourced events by @RyanHolstien in #9988
• feat(pluginRegistry): add configuration to reduce runnable frequency by @RyanHolstien in #9990
• build(react): Fix typescript errors in test files by @sumitappt in #9982
• feat(docs): disable last update timestamps by @hsheth2 in #9987
• feat: add versioned content for 0.12.1 by @yoonhyejin in #9944
• doc: add version 0.13.0 by @yoonhyej...

v0.13.0

DataHub v0.13.0 Release Notes Summary

User Experience

• NEW - Asset Documentation Forms & UI-Editable Properties: Define specific documentation requirements via a Form, and empower your asset owners to capture their valuable knowledge via UI-Editable Properties. Watch the demo here!
• NEW - DataHub Incidents: Create, communicate, and data quality and observability incidents when they inevitably arise. Watch the demo here!
  UI Improvements: Editing secrets, handling forms, and rendering token pages and lineage diagrams have been improved for a smoother user interface experience.
• UI Improvements: Editing secrets, handling forms, and rendering token pages and lineage diagrams have been improved for a smoother user interface experience.

Developer Experience

• Security Upgrades: Core dependencies like shiro-core and FastAPI have been upgraded to fix vulnerabilities, ensuring a safer development environment.
• GraphQL/OpenAPI Enhancements: New GraphQL endpoints and better OpenAPI documentation provide more powerful tools for API interaction, making developers' jobs easier.
• Performance Tuning: Backend improvements for search operations and ingestion processes make the platform faster and more reliable.

Metadata Ingestion

• Platform Integrations: Enhanced support for dbt, Metabase, BigQuery, AWS Glue, Oracle, and Redshift allows for more comprehensive metadata capture, making integration with these platforms smoother.
• Ingestion Framework: The reliability of ingestion has been improved, with new capabilities like support for tags from Tableau datasources and compatibility with Airflow 2.5.0, facilitating a broader range of data synchronization tasks.
• Connector Improvements: Ingestion connectors for external data tools have been streamlined, ensuring easier integration and data synchronization.

Other Improvements and Fixes

• Enhanced internal testing frameworks with Cypress and pytest-random-order for ingestion tests.
• Simplified developer workflows with configurable Docker Compose project names in CLI.
• Addressed various ingestion-related bugs for platforms like Feast and Snowflake.
• Enhanced the UI codebase with TypeScript compilation linting and updated styles.
• Streamlined CI processes for pull requests and linting conditions.
• Version Upgrades: Upgraded pytest-docker, Pegasus, and SQLglot, among others, to improve stability and performance. Security vulnerabilities addressed by upgrading FastAPI, gitdb, and follow-redirects.

Notable Breaking Changes

• Updates to MySQL version for quickstarts and migration to Neo4j 5.x may impact existing setups.
• JDK17 build requirement and Docker Compose > 2.20 needed for building DataHub.
• Python 3.8+ requirement for the acryl-datahub CLI.
• Changes in Unity Catalog ingestion source configs and Redshift lineage generation.
• Deprecation of Spark 2.x and associated JDK8 build requirements.

For full details on breaking changes, please visit DataHub's update guide.

Acknowledgements

A huge thank you to all our contributors for making this release possible. Your hard work and dedication are greatly appreciated.

First-Time Contributors

7onn, Adityamalik123, atjones0011, BlueHorn07, diegoreico, dim-ops, fer-marino, Gerrit-K, gp1105739, ilpianista, ingthorb, KaYunKIM, Kunal-kankriya, muzzacode, nnnkkk7, pankajmahato-visa, rubiojr, ryaminal, scalvanese452, sleeperdeep, stevenayers.

Repeat Contributors

allizex, arunvasudevan, cburroughs, feldjay, gaurav2733, iprentic, KulykDmytro, kushagra-apptware, mayurinehate, nmbryant, noggi, purnimagarg1, rinzool, sgomezvillamor, shubhamjagtap639, siddiquebagwan-gslab, siladitya2, skrydal, sumitappt, TonyOuyangGit, wngus606, yangjiandan, Salman-Apptware.

DataHub Maintainers

anshbansal, asikowitz, chriscollins3456, darnaut, david-leifker, eboneil, ethan-cartwright, gabe-lyons, hsheth2, jjoyce0510, maggiehays, pedro93, RyanHolstien, shirshanka, sid-acryl, treff7es, yoonhyejin.

What's Changed

• fix(ingest/transformer): correct registration by @anshbansal in #9418
• docs(ingest/sql-queries): Rearrange sections by @asikowitz in #9426
• fix: Adjusting the view of the Column Stats by @Salman-Apptware in #9430
• feat(patch): support fine grained lineage patches by @RyanHolstien in #9408
• fix(CVE-2023-6378): update logback classic by @RyanHolstien in #9438
• feat: allow the sidebar size to be draggable by @Salman-Apptware in #9401
• fix(json-schema): do not send invalid URLs by @anshbansal in #9417
• fix(ingest/profiling) Fixing profile eligibility check by @treff7es in #9446
• fix(ingest): avoid git dependency in dbt by @hsheth2 in #9447
• feat(ingest): add retries for tableau by @hsheth2 in #9437
• docs(updating-datahub): update docs for v0.12.1 by @david-leifker in #9441
• feat: Allow specifying Data Product URN via UI by @Salman-Apptware in #9386
• Add button to copy urn of an Ownership Type by @Salman-Apptware in #9452
• docs(ingest/tableau): add token to sink config in sample recipe by @KaYunKIM in #9411
• feat(glossary): add ability to clone glossary term(name and documentation) from term profile menu by @allizex in #9445
• feat(ingestion): Add typeUrn handling to ownership transformers by @skrydal in #9370
• fix(ingest): reduce GraphQL Logs to warning for circuit breaker by @arunvasudevan in #9436
• fix: support Apollo caching for settings / Policies by @Salman-Apptware in #9442
• refactor | PRD-785 | datahub oss: migrate use of useGetAuthenticatedU… by @sumitappt in #9456
• refactor(ui): Minor improvements & refactoring by @jjoyce0510 in #9420
• feat(ingest): add ingest --no-progress option by @BlueHorn07 in #9300
• fix(powerbi): add access token refresh by @anshbansal in #9405
• fix | PRD-463 | Stop trying to ping the track endpoint on login home … by @sumitappt in #9462
• feat(ingest/unity): enable hive metastore ingestion by @mayurinehate in #9416
• feat(ingestion/transformer): create tag if not exist by @siddiquebagwan-gslab in #9076
• fix(ingest): make user_urn and group_urn generation consider user and… by @shirshanka in #9026
• feat(ingestion): Add test_connection methods for important sources by @shubhamjagtap639 in #9334
• docs: fix sample command for container logs by @nnnkkk7 in #9427
• fix(ingest): bump source configs json schema version by @hsheth2 in https://github...

DataHub v0.12.1

Release Highlights

New Features

SQLAlchemy Source Enhancements: Support for view lineage across all SQLAlchemy sources (PR #9039).
Airflow Integration: Retry callback and support for ExternalTaskSensor subclasses added (PR #8514).
Kafka Enhancements: Increased Kafka message size and enabled compression (PR #9038).
JSONSchema Ingestion: Enabled schema-aware JsonSchemaTranslator (PR #8971).
Search Bar Improvements: Added a flag to hide/display the autocomplete query (PR #9104).
SQL Parser Performance: Enhancements and asyncio fixes (PR #9119).
MongoDB Ingestion: Support for stateful ingestion and improved schema inference for lists (PR #9118, PR #9145).
Policy Engine Updates: Refactoring and enhancements, including support for 10k+ policies (PR #9163, PR #9177).
UI Enhancements: Numerous improvements including command-k icons in the search bar, updated Apollo cache, and auto-complete debounce in the search bar (PR #9194, PR #9193, PR #9205).
Fivetran Integration: Connector integration for Fivetran (PR #9018).
Neo4j Database Support: Connection to specific Neo4j databases now supported (PR #9179).
Chart Subtypes in UI: Support for chart subtypes (PR #9186).

Fixes and Improvements

BigQuery Fixes: Resolved issues with lineage filter query, and fixed extracting comments from complex types (PR #9114, PR #8950).
MongoDB Refactoring: Platform instance addition to MongoDB (PR #8663).
Kafka Setup: Adjusted truststore settings for PEM files (PR #8656).
REST API Authorization: Fixed rollback failure when authorization is enabled (PR #9092).
Java Exception Handling: Addressed java.util.ConcurrentModificationException (PR #9090).
UI and Documentation: Fixed filtering logic in UI, corrected documentation errors, and added feature guides (PR #9116, PR #9125, PR #9124, PR #9126, PR #9134, PR #9137, PR #9122, PR #9068).
SQL Server and Snowflake Ingestion: Updated queries and fixed missing view downstream call (PR #9127, PR #8966).
ClickHouse and DB2 Ingestion: Addressed column reflection regression and table properties handling (PR #9143, PR #9128).
Ingestion Improvements: Numerous fixes and enhancements across various ingestion sources (PR #9153, PR #9155, PR #9141, PR #9157, PR #9123).
CI and Build Process: Tweaked workflows, increased gradle retries, and addressed CI errors (PR #9052, PR #9091, PR #9160).
Security Updates: Addressed a zookeeper CVE and other security concerns (PR #9190).
UI Refactoring: Improved entity page loading indicators and renamed button texts (PR #9195, PR #9196).
Policy and Auth Enhancements: Refactored policy locking and added roles to policy engine validation logic (PR #9178).

Testing and Continuous Integration

API Testing: Added tests for managing secrets, access token privilege, and flaky tests fix (PR #9121, PR #9167, PR #9132, PR #9175).
Cypress Test Fixes: Addressed glossary navigation and download_lineage_results tests (PR #9175, PR #9132).
Cleanup and Refactoring
Ingestion Cleanup: Removed legacy memory_leak_detector and refactored ingestion sources (PR #9158, PR #9135, PR #9120, PR #9105).
PDL Refactoring: Refactored Assertion model enums (PR #9191).
Build and Deployment
Release Preparation: Updated files for the 0.12.0 release (PR #9130).

What's Changed

• feat(ingest): support view lineage for all sqlalchemy sources by @mayurinehate in #9039
• fix(ingest/bigquery): Fixing lineage filter query by @treff7es in #9114
• refactor(ingestion/mongodb): Add platform_instance to mongodb by @nicholas-fwang in #8663
• fix(kafka-setup): Don't set truststore pass for PEM files by @mmmeeedddsss in #8656
• fix(ingest): Fix roll back failure when REST_API_AUTHORIZATION_ENABLED is set to true by @TonyOuyangGit in #9092
• (fix): Avoid java.util.ConcurrentModificationException by @rtekal in #9090
• Fix(ingest/bigquery): fix extracting comments from complex types by @maaaikoool in #8950
• docs: add versions 0.12.0 by @yoonhyejin in #9125
• fix(ui) Fix filtering logic for everwhere generating OR filters by @chriscollins3456 in #9116
• build(release): Update files for 0.12.0 release by @pedro93 in #9130
• fix(ingest/sql-server): update queries to use escaped procedure name by @mayurinehate in #9127
• feat(airflow): retry callback, support ExternalTaskSensor subclasses by @richenc in #8514
• docs: fix saasonly flags for some pages by @yoonhyejin in #9124
• fix(ingest/snowflake): missing view downstream cll if platform instance is set by @mayurinehate in #8966
• feat: Add flag to hide/display the autocomplete query for search bar by @kushagra-apptware in #9104
• docs(timeline): correct markdown heading level by @mayurinehate in #9126
• docs(graphql) Correct mutation -> query for searchAcrossLineage examples by @eboneil in #9134
• feat(kafka): increase kafka message size and enable compression by @david-leifker in #9038
• feat(ingest/jsonschema) enable schema-aware JsonSchemaTranslator by @KulykDmytro in #8971
• fix(metadata-ingestion): adds default value to _resolved_domain_urn i… by @alexklavensnyt in #9115
• ci: tweak to only run relevant workflows by @anshbansal in #9052
• Fix for flaky download_lineage_results cypress test by @kkorchak in #9132
• docs: Update updating-datahub.md by @pedro93 in #9131
• fix(ingest/clickhouse): pin version to solve column reflection regression by @hsheth2 in #9143
• feat(ingest/looker): cleanup error handling by @hsheth2 in #9135
• feat(ingest): add entity_supports_aspect helper by @hsheth2 in #9120
• feat(sqlparser): support more update syntaxes + fix bug with subqueries by @hsheth2 in #9105
• docs: correct broken doc links by @sachinsaju in #9137
• feat(ingest): sql parser perf + asyncio fixes by @hsheth2 in #9119
• feat(quickstart): fix broker InconsistentClusterIdException issues by @hsheth2 in #9148
• fix(policies): remove non-existent policies, fix name by @anshbansal in #9150
• Fix for a test that passed on Oss and failed on Saas by @kkorchak in #9147
• docs(teradata): teradata doc external link 404 fix by @sachinsaju in #9152
• fix(datahub-client): Include relocation for snakeyaml dependency. by @jiateoh in #8911
• fix(ingest): cleanup large images in CI by @hsheth2 in #9153
• build: increase gradle retries by @hsheth2 in #9091
• feat(ingest): bump sqlglot parser by @hsheth2 in #9155
• feat(ingest/mongodb): support stateful ingestion by @TonyOuyangGit in #9118
• API test for managing secrets privilege by @kkorchak in #9121
• fix(ingest): handle exceptions in min, max, mean profiling by @mayurinehate in #9129
• feat: rename Assets tab to Owner Of by @kushagra-apptware in #9141
• fix(ingest/mongodb): fix schema inference for lists of values by @hsheth2 in #9145
• fix(ingest/db2): fix handling for table properties by @deepgarg-visa in #9128
• fix(ingest): fully support MCPs in urn_iter primitive by @hsheth2 in #9157
• fix(ingest/bigquery): use correct row count in null count profiling c… by @mayurinehate in #9123
• docs: add feature guides for subscriptions and notifications by @yoonhyejin in #9122
• docs: unify oidc guides using tabs by @yoonhyejin in #9068
• chore(ingest): remove legacy memory_leak_detector by @hsheth2 in #9158
• feat(ingest/looker): support emitting unused explores by @hsheth2 in #9159
• refactor(policy): refactor policy locking, no functional difference by @david-leifker in #9163
• API test for managing access token privilege by @kkorchak in #9167
• fix(mysql-setup): quote database name by @darnaut in #9169
• fix(health): fix health check ...

v0.12.1rc2

What's Changed

• fix(deprecation): bring frontend in-sync with model by @anshbansal in #9303
• fix: fix the settings height when there are not many items by @Salman-Apptware in #9294
• docs: update recommended CLI by @anshbansal in #9307
• feat(ui): bump frontend dependencies by @ngamanda in #8353
• fix(java) Fixes NPE ES service by @chriscollins3456 in #9311
• feat(config): Configurable bootstrap of ownership types by @skrydal in #9308
• feat: update the "json-schema" version from package.json to solve json-schema vulnerability by @kushagra-apptware in #9289

Full Changelog: v0.12.1...v0.12.1rc2

v0.12.0

v0.12.0 Release Highlights

User Experience

Nested Domains

Nested Domains are here! This provides flexibility in organizing your entities within Domains to match the unique organizational structure of your company.

DataHub Chrome Extension Improvements

The Acryl DataHub Chome extension now supports PowerBI! This is a super powerful way for your business users to gain DataHub-specific insights directly in the BI tools they use most. Additionally, we now support making edits back to DataHub Entities directly from the Chrome extension.

Access Management Tab for Datasets

Shoutout to @Ramendra761 from the PayPal Team for contributing a new Access Management tab in Dataset Entity pages! The aim of this feature is to enable users to view the required roles for accessing the Dataset, as defined by Roles and/or Policies in the organization’s Access Management System. It also introduces the ability to request access directly from the page.

Metadata Ingestion

Miscellaneous Improvements

• Sampling-Based Profiling: You can now configure sampling-based profiling to address query performance concerns in Snowflake and BigQuery
• Kafka Connect > Snowflake: We now support automatically defining lineage between the two platforms
• Athena: Support for complex and nested schemas

Column-Level Lineage

We are incubating CLL support for the following:

• Airflow plugin v2 now supports automatic extraction of CLL for certain operators, removing the need to annotate DAGs
• dbt
• Redshift
• PowerBI (support for Column-Level Lineage for M-Query)

Incubating Sources

• MLflow
• Teradata
• Unity Catalog Notebooks
• DynamoDB

Developer Experience

• Data Contracts: v0.12.0 introduces underlying models and CLI; UI support to follow
• We now support creating custom models without requiring a fork of the main DataHub project
• Updates to support OpenSearch 2.x and alternate Postgres db in postgres-setup

Other Notable Changes

• Session token configuration has changed, all previously created session tokens will be invalid and users will be prompted to log in. Expiration time has also been shortened which may result in more login prompts with the default settings.
  There should be no other interruption due to this change.

Breaking Changes

Find full details here

• #9044 - GraphQL APIs for adding ownership now expect either an ownershipTypeUrn referencing a customer ownership type or a (deprecated) type. Where before adding an ownership without a concrete type was allowed, this is no longer the case. For simplicity you can use the type parameter which will get translated to a custom ownership type internally if one exists for the type being added.
• #9010 - In Redshift source's config incremental_lineage is set default to off.
• #8810 - Removed support for SQLAlchemy 1.3.x. Only SQLAlchemy 1.4.x is supported now.
• #8942 - Removed urn:li:corpuser:datahub owner for the Measure, Dimension and Temporal tags emitted
  by Looker and LookML source connectors.
• #8853 - The Airflow plugin no longer supports Airflow 2.0.x or Python 3.7. See the docs for more details.
• #8853 - Introduced the Airflow plugin v2. If you're using Airflow 2.3+, the v2 plugin will be enabled by default, and so you'll need to switch your requirements to include pip install 'acryl-datahub-airflow-plugin[plugin-v2]'. To continue using the v1 plugin, set the DATAHUB_AIRFLOW_PLUGIN_USE_V1_PLUGIN environment variable to true.
• #8943 - The Unity Catalog ingestion source has a new option include_metastore, which will cause all urns to be changed when disabled.
  This is currently enabled by default to preserve compatibility, but will be disabled by default and then removed in the future.
  If stateful ingestion is enabled, simply setting include_metastore: false will perform all required cleanup.
  Otherwise, we recommend soft deleting all databricks data via the DataHub CLI:
  datahub delete --platform databricks --soft and then reingesting with include_metastore: false.
• #8846 - Changed enum values in resource filters used by policies. RESOURCE_TYPE became TYPE and RESOURCE_URN became URN.
  Any existing policies using these filters (i.e. defined for particular urns or types such as dataset) need to be upgraded
  manually, for example by retrieving their respective dataHubPolicyInfo aspect and changing part using filter i.e.

   "resources": {
     "filter": {
       "criteria": [
         {
           "field": "RESOURCE_TYPE",
           "condition": "EQUALS",
           "values": [
             "dataset"
           ]
         }
       ]
     }

into

   "resources": {
     "filter": {
       "criteria": [
         {
           "field": "TYPE",
           "condition": "EQUALS",
           "values": [
             "dataset"
           ]
         }
       ]
     }

for example, using datahub put command. Policies can also be removed and re-created via UI.

• #9077 - The BigQuery ingestion source by default sets match_fully_qualified_names: true. This means that any dataset_pattern or schema_pattern specified will be matched on the fully qualified dataset name, i.e. <project_name>.<dataset_name>. We attempt to support the old pattern format by prepending .*\\. to dataset patterns lacking a period, so in most cases this should not cause any issues. However, if you have a complex dataset pattern, we recommend you manually convert it to the fully qualified format to avoid any potential issues.

What's Changed

• feat(UI): AccessManagement UI to access the role metadata for a dataset by @Ramendra761 in #8541
• Glossary Navigation Cypress test by @kkorchak in #8804
• ci: upgrade python to 3.10 for builds by @hsheth2 in #8808
• feat(ingestion/looker): Add view file-path as option in view_naming_pattern config by @siddiquebagwan-gslab in #8713
• feat(upgrade): add ability to provide a startingOffset for RestoreIndices by @ukayani in #8539
• fix(index): Do not override the search analyzer for ngram fields by @iprentic in #8818
• test(managed_ingestion): fix managed ingestion test by fixing actions… by @david-leifker in #8820
• docs: add 0.11 docs to docs site by @hsheth2 in #8813
• docs(release): Update updating-datahub.md for 0.11.0 release by @iprentic in #8821
• fix(ingest/mssql): Add UNIQUEIDENTIFIER data type as String by @cjm98332 in #8642
• build(ingest): upgrade to sqlalchemy 1.4, drop 1.3 support by @mayurinehate in #8810
• fix(ingest): use epoch 1 for dev build versions by @hsheth2 in #8824
• ci: make wheel builds more robust by @hsheth2 in #8815
• feat(cli): fix upload ingest cli endpoint by @pedro93 in #8826
• docs(transformer): fix names in sample code of 'pattern_add_dataset_domain' by @Starkie in #8755
• fix(siblingsHook): check number of dbtUpstreams instead of all upStreams by @ethan-cartwright in #8817
• fix(java) Update DataProductMapper to always return a name by @chriscollins3456 in #8832
• build(ingest): Bump jsonschema for Python >= 3.8 by @asikowitz in #8836
• feat(ingest/rest-emitter): Do not raise error on retry failure to get better error messages by @asikowitz in #8837
• ci: add markdown-link-check by @yoonhyejin in #8771
• docs(managed datahub): release notes 0.2.11 by @anshbansal in #8830
• build(ingest): Remove constraint on jsonschema for Python >= 3.8 by @asikowitz in #8842
• fix(build): clean task cleanup generated src by @anshbansal in #8844
• feat(ci): disable ingestion smoke build by @anshbansal in #8845
• fix: fix quickstart page by @yoonhyejin in #8784
• feat(bigquery): add better timers around every API call by @mayurinehate in #8626
• feat(ingestion/dynamodb): Add DynamoDB as new metadata ingestion source by @TonyOuyangGit in #8768
• feat(ingest/bigquery): support bigquery profiling with sampling by @mayurinehate in #8794
• Fix for edit_documentation and glossary_navigation cypress tests by @kkorchak in #8838
• feat(ui/java) Update domains to be nested by @chriscollins3456 in #8841
• dcs(ml-models): enhancing ml model documentation ...

v0.11.0

Release Highlights

Potential Downtime

This release introduces substantial improvements to search ranking which require reindexing indices.

During the reindexing:

• a system-update job will set indices to read-only and create a backup/clone of each index
• new components will be prevented from start-up until the reindex completes
• Helm deployments will go into read-only mode and new ingestion runs will fail

This process can take anywhere from 5 minutes to multiple hours; as a rough estimate, please expect it to take 1 hour for every 2.3 million entities. After the reindex is complete, please check your ingestion run to re-run any that did not complete.

User Experience

New Search and Browse Experience

We have some really exciting improvements to the DataHub user experience in this release! The new search and browse experience, which was first made available in the previous release behind a feature flag, is now on by default. Check out our release notes for v0.10.5 to get more information and documentation on this new Browse experience.

Learn all about the new Search and Browse experience!

Improvements to Search

In addition to the ranking changes mentioned above, this release includes changes to the highlighting of search entities to understand why they match your query. You can also sort your results alphabetically or by last updated times, in addition to relevance. In this release, we suggest a correction if your query has a typo in it.

See the Search improvements in action!

Manage Home Page Posts

In this release we now enable you to create and delete pinned announcements on your DataHub homepage! If you have the “Manage Home Page Posts” platform privilege you’ll see a new section in settings called “Home Page Posts” where you can create and delete text posts and link posts that your users see on the home page.

OpenAPI Endpoints Expanded

OpenAPI entity and aspect endpoints expanded to improve developer experience when using this API with additional aspects to be added in the near future.

Metadata ingestion

Added support for Confluent S3 Sink Connector, extracting stored procedures and jobs from mssql, and snowflake shares. Additionally, sql parsing source now converts query logs into CLL and usage.

Developer Experience

The CLI now supports recursive deletes.

Versioned documentation

Starting from this release, we support versioned documentation on the datahub docs site! Select the version you’re on and browse docs specifically at that version.

Performance Improvements

• Batching of default aspects on initial ingestion (SQL)
• Improvements to multi-threading. Ingestion recipes, if previously reduced to 1 thread, can be restored to the 15 thread default.
• Gradle 7 upgrade moderately improves build speed
• DataHub Ingestion slim images reduced in size by 2GB+

Important Bug Fixes

• Glue Schema Registry fixed

Deprecation Notice

• MAE Events are no longer produced. MAE events have been deprecated for over a year.

What's Changed

• feat(ingest/presto-on-hive): enable partition key for presto-on-hive by @zheyu001 in #8380
• feat(classification): allow parallelisation to reduce time by @mayurinehate in #8368
• feat(ingest): Add metabase database id to platform instance mapping by @k-popov in #8359
• feat(ingest): add ability to read other method types than GET for OAS ingest recipes by @jsmilkstein in #8303
• fix(ingest): fix data platform urn in dataset_urn_to_key and dataset_key_to_urn by @Masterchen09 in #8209
• fix(ingest/s3): wrong sorting in case of multi-partition key by @anshbansal in #8536
• fix(ingest/presto): fix presto on hive test failures by @hsheth2 in #8548
• Cypress test for managing groups by @kkorchak in #8520
• feat(ingest/kafka-connect): add support for Confluent S3 Sink Connector by @tusharm in #8298
• Variable rename - Allows deselection of members in add members modal for a group by @Sukeerthi31 in #8529
• fix(ingest/s3): catch no such bucket exception instead of failing by @anshbansal in #8549
• fix(ingest): add tableau sqlglot dep by @hsheth2 in #8552
• fix(ingetion/mssql): convert dataset urns to lowercase by @siddiquebagwan in #8551
• Fix flaky add_user smoke test by @kkorchak in #8471
• feat(ci): use docker registry cache by @hsheth2 in #8544
• fix(glue): restore glue configurations by @RyanHolstien in #8533
• build(release): Update files for 0.10.5 release by @iprentic in #8556
• docs(release): Update updating-datahub.md for 0.10.5 release by @iprentic in #8557
• feat(ingestion/snowflake): use user email-id in urn generation for top users stat by @siddiquebagwan in #8513
• docs(development.md): Minor grammatical error by @PauloGoncalvesLima in #8558
• fix(usage): Update index lifecycle policy to not delete old datahub usage events by @iprentic in #8565
• fix(ui): Simplify background color for Entity Health Status popover by @jjoyce0510 in #8559
• fix: add --write args on pre-commit prettier by @yoonhyejin in #8560
• docs(observe): Add feature doc for Freshness Assertions by @jjoyce0510 in #8547
• docs(updating): add details on Unified Search & Browse experience by @maggiehays in #8568
• fix: fix features section by @yoonhyejin in #8571
• feat(ingest): allow lower freq profiling based on date of month/day of week by @anshbansal in #8489
• fix(stats): default to 3 months by @anshbansal in #8566
• fix(aspect): count query only for relevant aspect index by @iprentic in #8569
• feat(quickstart): bump quickstart start periods more by @hsheth2 in #8573
• Origin/cypress test for managing policies by @kkorchak in #8554
• feat(ui) Show source documentation when editing entity documentation by @chriscollins3456 in #8516
• fix(ingest): handle redaction of configs with int keys by @hsheth2 in #8545
• fix(ingest/snowflake): maintain qualified name casing, do not lowercase by @mayurinehate in #8574
• feat(docs): add github repo links to readme and docs by @yoonhyejin in #8422
• feat(ebean): Add metric in ebean aspect DAO for failed tries, as well as failed operation… by @iprentic in #8576
• refactor(search) Use search across multiple-entities API, deprecate Aggregator classes by @iprentic in #8498
• feat(siblings): dont show multiple platform icons if the siblings are ghost nodes by @gabe-lyons in #8543
• docs(lineage): Add description to make_lineage_mce by @eboneil in #8596
• doc(ingest/log): failure log at pipeline level document by @anshbansal in #8591
• Dataset ownership test by @kkorchak in #8583
• doc(release): release notes for 0.2.10 by @anshbansal in #8599
• docs(release): fix typo by @anshbansal in #8600
• feat(ui): apply views to: domains, containers, terms by @eboneil in #8572
• feat(search): embedded view dropdown by @joshuaeilers in #8598
• fix(ingest/file): remove entity_type_counts and aspect_counts by @hsheth2 in #8586
• fix(ingest): use hive pure_sasl variant by @hsheth2 in #8570
• Feat(ingest/ldap)fix list index out of range error by @alplatonov in https://githu...

v0.10.5

Release Highlights

NEW: Unified Search and Browse Experience

It’s here, it’s here! We are incredibly excited to roll out our re-designed, streamlined Search and Browse experience. End-users now have a one-stop-shop to search for specific data entities and browse across systems, making it easier than ever to find the most relevant and meaningful resources within DataHub.

Checkout the screenshot below and get a full walk-through in this video!

User Experience

• Column-Level Lineage (CLL) visualization update: you can now visualize CLL relationships through DataJobs (i.e. Airflow DAGs)
• Unique Glossary Terms: We now prevent creating duplicate Glossary Term names within a Term Group
• Domains: You can now configure the Documentation tab to be the default landing page within a Domain
• Formatting updates to Row Count to make large numbers more human readable (ie. 3283337 > 3.2M)
• Stats Tab: Y-axis scale now dynamically set to reflect the minimum & maximum values, improving readability

Metadata ingestion

Ingestion Enhancements:

• BigQuery: Set platform_instance using project_id
• PowerBI: Ingest datasets not used in visualizations (tiles/pages
• Kafka Connect: Ability to set platform_instance
• Nifi: Support for basic auth
• Presto on Hive: Extract all table properties from Hive Metastore
• Elasticsearch: Support for basic profiling
• Add advanced configuration for LDAP manager ingestion

Lineage Improvements:

• Schema-aware SQL parsing to derive column-level lineage
• Column-level lineage support for BigQuery, Tableau, and Snowflake View definitions
• Snowflake: Extract Snowpipe S3 lineage

Developer Experience

• Fine-grained ownership policies
• PATCH support for DataJob Inputs/Outputs
• New endpoints to extract size of time-series indices and truncate/cleanup time-series indices in Elasticsearch; support for bulk-deletes
• Initial support for exception reporting via Sentry
• New OpenAPI endpoint to get Task Status
• SDK: Easily generate container URNs

Docs

• Improvements to our File-Based Lineage doc, specifically focused on Fine-Grained Lineage config components (link)
• Code examples of how to manage Posts within DataHub (link)
• Guide to generating custom browse paths for the new search experience (link)

What's Changed

• refractor(classification): datahub classifier init by @mayurinehate in #8193
• fix(glue): fix typo in reported warning, report with flow_urn by @mayurinehate in #8138
• fix(ingest/delta-lake): fix CI issues due to delta lake version bump by @mayurinehate in #8215
• Upgrade kafka and its dependencies to 3.4 in docker compose by @jinlintt in #8161
• chore(release): update default cli for managed ingestion by @pedro93 in #8226
• fix(ownership): Corrects graphQL resolver for entity operations by @pedro93 in #8219
• fix(cli/quickstart): handle docker hangs gracefully by @hsheth2 in #8211
• fix(cli): make quickstart robust to docker race conditions by @hsheth2 in #8233
• fix(search): tag/term should filter for both entity and field level by @anshbansal in #7881
• docs(tests): document test eval endpoint by @anshbansal in #8227
• feat(ingest/bigquery_v2): enable platform instance using project id by @asikowitz in #8216
• feat(stats): make rowcount more human readable by @joshuaeilers in #8232
• docs(es): Update aws deploy docs to correct ElasticSearch version by @iprentic in #8240
• feat(sdk): support patches as MCPs in file source by @hsheth2 in #8220
• fix(apiAuth): add resources where applicable and update docs by @RyanHolstien in #8234
• feat(patch): support datajob input output by @RyanHolstien in #8190
• feat(ingest/unity): Set external url for containers and datasets by @asikowitz in #8238
• docs(airflow): add docs on custom operators by @matthew-coudert-cko in #7913
• chore(release): update datahub upgrade docs by @pedro93 in #8228
• fix(ingestion/tableau): Remove unused field documentViewId by @mohdsiddique in #8225
• feat(ui): create fast path for immediate processing of ui sourced changes by @RyanHolstien in #8200
• fix(ingest/druid) Handling gracefully if no table returned in a schema by @treff7es in #8203
• fix(kafka-setup): bump kafka version by @david-leifker in #8245
• feat(ingestion/powerbi): Ingest datasets not used in PowerBI visualization(tiles/pages) by @mohdsiddique in #8212
• fix(sdk/dataflow): deprecate cluster and use env and platform_instance instead by @shubhamjagtap639 in #8201
• fix(ingest): pass platform correctly to browse path v2 helper by @asikowitz in #8244
• feat(search): Supporting Aggregations for hasX fields by @jjoyce0510 in #8241
• fix(ingest): Call validator on the base urn as well as aspect components when ingesting by @iprentic in #8250
• docs(website): adjust markprompt z-index so it's not covered by nav by @jeffmerrick in #8255
• fix(patch): Fix exception when using default patch for patching missing aspects by @jjoyce0510 in #8221
• fix(custom-search): revert underscore as quoted by @david-leifker in #8163
• chore(ci): add back optional static sleep for tests by @anshbansal in #8258
• chore(checkbox): darken all checkboxes by @joshuaeilers in #8248
• chore(assertions): catch any exception on assertion delete by @joshuaeilers in #8247
• feat(opensearch): Rollover usage events at a file size rather than time-based manner by @iprentic in #8182
• fix(ingest/okta): Set default of okta_profile_to_username_attr to email by @asikowitz in #8263
• feat(ui) Update Search & Browse to be a unified experience by @chriscollins3456 in #8235
• fix(ingest/tableau): split table columns query from datasources query by @mayurinehate in #8217
• fix(ingest/okta): Set default of okta connector to match OIDC defaults by @anshbansal in #8272
• feat(elasticsearch): Add endpoint for getting the size of timeseries indices by @iprentic in #8265
• feat(ingest/delete-cli): Add configurable batch size; update docs by @asikowitz in #8274
• fix aggregation sorting in browsev2 sidebar by @joshuaeilers in #8276
• Support de-selecting browse paths by @joshuaeilers in #8242
• feat(cli): Initial support for sending exceptions to Sentry by @treff7es in #7172
• fix(ingestion/powerbi): use admin api resolver to fetch modified workspaces by @mohdsiddique in #8273
• fix: dbt-athena types mapping for complex types by @svdimchenko in #8264
• feat(graphql) Prevent duplicate glossary term names within a group by @chriscollins3456 in #8187
• Add retries to JavaEntityClient:deleteReferencesTo by @joshuaeilers in #8268
• feat(ingest): Create zero usage aspects by @asikowitz in #8205
• fix(docs) Update Chrome extension docs to reflect current reality by @chriscollins3456 in #8284
• refactor(validations): Add URL-based Routing to Dataset Validations Tab by @jjoyce0510 in #8254
• fix(metadata-io): retry transactions on serialization errors when using a PostgreSQL database by @Masterchen09 in #8278
• docs(ingest/lineage): Update fine grained file lineage docs by @eboneil in https://github...

v0.10.4

Release Highlights

User Experience

• You can now create and assign Custom Ownership types within DataHub; plus, we now display the owner type on an Entity Page
  

• Various bug fixes to Column Level Lineage visualization

Metadata ingestion

• You can now define column-level lineage (aka fine-grained lineage) via our file-based lineage source
• Looker: Ingest Looks that are not part of a Dashboard
• Glue: Error reporting now includes lineage failures
• BigQuery: Now support deduplicating LogEntries based on insertId, timestamp, and logName

Docs

• CSV Enricher: improvements to sample CSV and recipe
• Guide for changing default DataHub credentials
• Updated guide to apply time-based filters on Lineage

What's Changed

• ci(ingest/kafka): improve kafka integration test reliability by @hsheth2 in #8085
• fix(ingest/bigquery): Deduplicate LogEntries based on insertId, timestamp, logName by @asikowitz in #8132
• feat(ingest/glue): report glue job lineage failures, update doc by @mayurinehate in #8126
• feat(lineage source): add fine grained lineage support by @anshbansal in #7904
• docs(glue): fix broken link by @mayurinehate in #8135
• feat(custom ownership): Adds Custom ownership types as a top level entity by @pedro93 in #8045
• Update updating-datahub.md for v0.10.3 release by @iprentic in #8139
• feat: add dbt-athena adapter support for column types mapping by @svdimchenko in #8116
• docs(csv-enricher): add example csv file & recipe by @gabe-lyons in #8141
• chore(ci): update base requirements file by @anshbansal in #8144
• fix(ingest/s3): Path spec aware folder traversal by @treff7es in #8095
• fix(ui) Fix selecting columns in Lineage tab for CLL by @chriscollins3456 in #8129
• feat(search): adding support for _entityType filter in the application layer + frontend by @gabe-lyons in #8102
• docs(ingest/nifi): fix broken links by @mayurinehate in #8143
• fix(scroll): fix scroll cache key for hazelcast by @RyanHolstien in #8149
• chore(json): fix json vulnerability by @RyanHolstien in #8150
• fix(ingest/json-schema): handle property inheritance in unions by @hsheth2 in #8121
• chore(log): fix log as error instead of info by @anshbansal in #8146
• fix(lineagecounts) Include entities that are filtered out due to sibling logic in the filtered count of lineage counts by @iprentic in #8152
• fix(stats): display consistent query count on stats tab by @joshuaeilers in #8151
• fix(ingest): remove original_table_name logic in sql source by @hsheth2 in #8130
• feat(ingest): add more fail-safes to stateful ingestion by @hsheth2 in #8111
• feat(ingest/snowflake): support for more operation types by @mayurinehate in #8158
• fix(ui) Show Entities first on Domain pages again by @chriscollins3456 in #8159
• fix(ingest/nifi): allow nifi site url with context path by @mayurinehate in #8156
• feat(ingest): Create Browse Paths V2 under flag by @asikowitz in #8120
• fix(ingestion/looker): set project-name for imported_projects views by @mohdsiddique in #8086
• fix(docs): Fix ownership type typos by @pedro93 in #8155
• docs(townhall) feb and march town hall agenda and recording by @maggiehays in #7676
• feat(ingest/unity): Add qualified name to dataset properties by @asikowitz in #8164
• feat(ingest/bigquery_v2): enable platform instance using project id by @Khurzak in #8142
• feat(ingest/snowflake): Deprecate legacy lineage and optimize query history joins by @asikowitz in #8176
• fix(ingest/kafka): Fixing error printing in Kafka properties get call by @treff7es in #8145
• fix(ingest/snowflake): set use_quoted_name to profile lowercase tables by @mayurinehate in #8168
• feat(classification): support for regex based custom infotypes by @mayurinehate in #8177
• fix(restli): update base client retry logic by @david-leifker in #8172
• fix(ingest): Fix modeldocgen; bump feast to relax pyarrow constraint by @asikowitz in #8178
• refactor(ci): move from sleep to kafka lag based testing by @shirshanka in #8094
• docs(lineage): document timestamp filtering in lineage feature by @iprentic in #8174
• build(ingest/feast): Pin feast to minor version by @asikowitz in #8180
• feat(ingest/snowflake): Okta OAuth support; update docs by @asikowitz in #8157
• feat(ingest/presto-on-hive): add support for extra properties and merge property capabilities by @treff7es in #8147
• docs(managed datahub): release notes for v0.2.8 by @anshbansal in #8185
• fix(nocode): fix DeleteLegacyGraphRelationshipsStep for Elasticsearch by @david-leifker in #8181
• feat(docker):Add the jattach tool to the docker container(#7538) by @yangjiandan in #8040
• refactor: Return original exception as caused by by @Jorricks in #7722
• docs(ingest) Add MetadataChangeProposalWrapper import to example code by @iprentic in #8175
• fix(ingest/kafka): Better error handling around topic and topic description extraction by @asikowitz in #8183
• fix(vulnerabilities)/vulnerabilities_fixes_datahub (#8075) by @david-leifker in #8189
• fix: add dedicated guide on changing default credentials by @yoonhyejin in #8153
• feat(classification): configurable minimum values threshold by @mayurinehate in #8186
• fix(ingestion/looker): ingest looks not part of dashboard by @mohdsiddique in #8140
• fix(ingest/profiling): only apply monkeypatches once when profiling by @hsheth2 in #8160
• docs(tableau): site config is required for tableau cloud / tableau online by @mohdsiddique in #8041
• fix(ingest/bigquery): Swap log order to avoid confusion by @asikowitz in #8197
• fix(ingest/redshift): Adding env parameter where it was missing for urn generation by @treff7es in #8199
• revert(ingest/bigquery): Do not emit DataPlatformInstance; remove references to platform_instance by @asikowitz in #8196
• docs(managed datahub): add docs link to v0.2.8 by @anshbansal in #8202
• Add combined health check endpoint which can check multiple components by @iprentic in #8191
• chore(cp-schema-registry): bump minor version by @david-leifker in #8192
• feat(ingest): Produce browse paths v2 on demand and with platform instance by @asikowitz in #8173

New Contributors

• @svdimchenko made their first contribution in #8116
• @Khurzak made their first contribution in #8142
• @Jorricks made their first contribution in #7722

Full Changelog: v0.10.3...v0.10.4

v0.10.3

Release Highlights

User Experience

• Define Data Products via YAML and manage associated entities within a Domain
• Search experience: quickly apply a filter at time of search
• Form-based PowerBI ingestion

Developer Experience

• Progress toward Removing Confluent Schema Registry requirement -- Helm & Quickstart simplifications to follow
  • NOTE: this will only work for new deployments of DataHub; If you have already deployed DataHub with Confluent Schema Registry, you will not be able to disable it
• Delete CLI - correctly handles deleting timeseries aspects
• Ongoing improvements to Quickstart stability
• Support entity types filter in get_urns_by_filter
• Search customization
  • regex based query matching
  • full control over scoring functions (useable on any document field, i.e. tags, deprecated flags, etc)
  • enable/disable fuzzy, prefix, exact match queries

Ingestion

• BigQuery - Improve ingestion disk usage & speed; extract dataset usage from Views
• Unity Catalog - Capture create/last modified timestamps; extract usage; data profiling support
• PowerBI - Update workspace concept mapping; support modified_since, extract_dataset_schema, and more
• Superset – support stateful ingestion
• Business Glossary – Simplify ingestion source
• Kafka – Add description in dataset properties
• S3 – Support stateful ingestion & last_updated
• CSV Enricher – Support updating more types
• PII Classification - Configurable sample size
• Nifi - Support Kerberos authentication

What's Changed

• fix(ingest/bigquery): Add to lineage, not overwrite, when using sql parser by @asikowitz in #7814
• fix(ingest/bigquery): Enable lineage and usage ingestion without tables by @asikowitz in #7820
• fix(ingest/bigquery): Do not query columns when not ingesting tables or views by @asikowitz in #7823
• fix(ingest/bigquery): update usage query, remove erroneous init by @mayurinehate in #7811
• fix(ingest/bigquery): Handle null values from usage aggregation by @asikowitz in #7827
• perf(ingest/bigquery): Improve bigquery usage disk usage and speed by @asikowitz in #7825
• fix(cli): use correct ingestion image in script by @hsheth2 in #7826
• fix(release): prevent republish of images on release edits by @RyanHolstien in #7828
• feat(): finish populating the entity registry by @hsheth2 in #7818
• fix(ui) Fix 404 page routing bug by @chriscollins3456 in #7824
• feat(ui): Support PowerBI Ingestion via UI form by @jjoyce0510 in #7817
• fix(ingest/snowflake): fix column name in snowflake optimised lineage by @mayurinehate in #7834
• feat(ingest/unity): capture create/lastModified timestamps by @hsheth2 in #7819
• fix(test): fix spark lineage test by @david-leifker in #7829
• docs(): add markprompt help chat by @jeffmerrick in #7837
• Update DataJobInputOutput.pdl to express that CLL fields are not shown in the UI right now by @gabe-lyons in #7830
• feat(cli): improve quickstart stability by @hsheth2 in #7839
• chore(ci): regular upgrade base requirements.txt by @anshbansal in #7821
• feat(timeseries): Support sorting timeseries aspects by non-timestampMillis field + fix operations resolver by @jjoyce0510 in #7840
• doc(ingestion/tableau): Fix rendering ingestion quickstart guide by @mohdsiddique in #7808
• fix(ingest): pin sqlparse version by @hsheth2 in #7847
• feat(urn): Add a validator when creating an URN that it is no longer than the li… by @iprentic in #7836
• chore(ingest): bug fix in sqlparse pin by @hsheth2 in #7848
• feat: enriching guide on creating dataset by @yoonhyejin in #7777
• feat(docs): consolidate api guides by @yoonhyejin in #7857
• fix(ingest/salesforce): use report timestamp for operations by @hsheth2 in #7838
• chore(ci): fix CI failing due to lint by @anshbansal in #7863
• fix(mcl): fix improper pass by reference by @RyanHolstien in #7860
• feat(urn) Add validator to reject URNs which contain the character we plan to u… by @iprentic in #7859
• feat(elasticsearch): Add servlet which provides an endpoint for a healthcheck on the ES cl… by @iprentic in #7799
• fix(ui) Add UI fixes and design tweaks to AutoComplete by @chriscollins3456 in #7845
• fix(ui) Get all entity assertions in chrome extension by @chriscollins3456 in #7849
• refactor(platform): Refactoring ES Utils, adding EXISTS condition support to Filter Criterion by @jjoyce0510 in #7832
• chore(ui): change background color to transparent for avatar with photoUrl by @hieunt-itfoss in #7527
• refactor(ingest): Add helper DataHubGraph methods by @asikowitz in #7851
• fix(ui) Disable cache on Domain and Glossary Related Entities pages by @chriscollins3456 in #7867
• fix(cache): Fix cache key serialization in search service by @pedro93 in #7858
• docs(ingest): update dbt and aws docs by @hsheth2 in #7870
• docs(ingest): fix CorpGroup example by @hsheth2 in #7816
• docs(ingest/powerbi): update workspace concept mapping by @eeepmb in #7835
• feat(ingest/powerbi): support modified_since, extract_dataset_schema and many more by @aezomz in #7519
• Remove usages of commons-text library lower than 1.10.0 by @iprentic in #7850
• feat(glue): allow resource links to be ignored by @YusufMahtab in #7639
• feat(ingestion): lookml refinement support by @mohdsiddique in #7781
• feat(ingest/unity): Ingest ownership for containers; lookup service principal display names by @asikowitz in #7869
• Logging and test models fixes by @david-leifker in #7884
• feat(model) Add ContainerPath aspect model by @chriscollins3456 in #7774
• bug(7882): run kafka-configs.sh on DataHubUpgradeHistory_v1 to make sure the retention.ms is set to infinite by @jinlintt in #7883
• fix: refactor toc by @yoonhyejin in #7862
• feat(cli): Modifies ingest-sample-data command to use DataHub url & token based on config by @pedro93 in #7896
• feat(ingest/snowflake): optionally emit all upstreams irrespective of recipe pattern by @mayurinehate in #7842
• fix(ingestion/tableau): backward compatibility with version 2021.1 an… by @mayurinehate in #7864
• fix(ingest/dbt): ensure dbt shows view properties by @hsheth2 in #7872
• docs(airflow): add debug guide on url generation by @hsheth2 in #7885
• feat(sdk): support entity types filter in get_urns_by_filter by @hsheth2 in #7902
• fix(ingest/snowflake): fix optimised lineage query, filter temporary … by @mayurinehate in #7894
• fix(ingest/bigquery): fix handling of time decorator offset queries by @mayurinehate in #7843
• fix(ingest): fix minor bug + protective dep requirements by @hsheth2 in #7861
• fix(cli): remove duplicate labels from quickstart files by @hsheth2 in #7886
• Revert "feat(cli): Modifies ingest-sample-data command to use DataHub… by @pedro93 in #7899
• feat(sdk): add DataHubGraph.get_entity_semityped method by @hsheth2 in #7905
• test(ingest/biz-glossary): add test for enable_auto_id by @hsheth2 in #7911
• feat(ingest): add GCS ingestion source by @mayurinehate in #7903
• [bugfix] Fix remote file ingestion...