Skip to content

v1.8.0

Latest
Latest

Choose a tag to compare

View all tags
@datahub-release-bot datahub-release-bot released this 22 Oct 08:17
v1.8.0
faab576
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v1.8.0

Release Availability Date

22 Oct 2026

Requirements

  • CLI / Python SDK: 1.7.0.14
  • Helm chart: 1.1.8

Upgrade actions are in Updating DataHub.

Highlights

  • OpenSearch 3.x. Supported next to OpenSearch 2.x and Elasticsearch 8.x. The engine version is detected automatically. OpenSearch 3.x will not create new indexes with the nmslib kNN engine, and it enforces the 512-byte document-id limit on bulk writes.
  • Metrics and Semantic Models. Charts, dashboards, and datasets can take a Metric as an upstream, with column-level lineage through metric upstreams. Metrics and Semantic Models are included in global search. The feature ships off. Set METRICS_ENABLED=true on GMS to show it.
  • Dark mode. A dark theme is in this release and stays off until THEME_DARK_MODE_ENABLED=true. With that set, the toggle is under Settings → Appearance.
  • Structured Properties. Creating, editing, and viewing a structured property is a page rather than a side drawer. Allowed values can be reordered, each with an optional description. Leaving the page with unsaved edits prompts before discarding them. The properties table adds Type and Platforms columns.
  • Data Products. Data Products can nest under a parent Data Product. Data Products are filterable in global search. On upgrade, a one-time system update fills the asset-side dataProducts search field from stored membership so that filter matches existing members. It is on by default (BOOTSTRAP_SYSTEM_UPDATE_DATA_PRODUCT_ASSETS_ENABLED=true).
  • New ingestion sources. IBM/HCL Informix, Monte Carlo, and SQLMesh.

Product

  • Compliance forms can be assigned to owners of a specific ownership type.
  • Deprecation date input is editable.
  • Languages in public beta: Japanese, Simplified Chinese, Traditional Chinese, and Russian.
  • Sidebars. Domains, glossary, metrics, and documents sidebars can sort and filter. The home page Domains module lists up to 25 domains.
  • Views can filter on structured properties.
  • Incidents can be raised on ML models, ML features, ML feature tables, and schema fields.

Platform

  • Search document ids (Search V3). Search V3 documents use a SHA-256 of the entity URN as _id. V2 indexes are unchanged. If an environment already wrote V3 documents under URL-encoded ids, hashed ids are not an in-place rewrite. Run RestoreIndices with -a clean, or reindex into a fresh index. A fill without clean leaves the old _id in place and the next update creates a duplicate.
  • GraphQL worker pool. When GRAPHQL_CONCURRENCY_SEPARATE_THREAD_POOL=true, pool sizes no longer follow the node’s CPU count. Defaults are the 8-core sizes: core 40, max 800. The separate pool stays off by default.
  • AWS IRSA. GMS keeps one process-wide default credentials provider, built once and closed on shutdown.
  • Timeseries reads. Dataset profile, usage, and operations, and dashboard usage statistics, require the matching View privileges on Rest.li and OpenAPI as well as GraphQL.
  • Access tokens. Creating a token with no expiry is disabled. Allowed durations default to 1 hour, 1 day, 7 days, 30 days, 90 days, 180 days, and 365 days. Tokens that already exist, including never-expiring ones, keep working.
  • Query SQL. Reading a query, a view definition, or topSqlQueries requires View Entity Queries on a subject dataset, or the platform privilege View All Queries. On upgrade, policies that already grant View Entity Page are backfilled with both. A policy created after the upgrade that only grants View Entity Page does not reveal query SQL.

Ingestion

New sources: IBM/HCL Informix, Monte Carlo, SQLMesh.

Connectors:

  • Snowflake. Logical tables in semantic views keep their stored casing. preserve_column_case keeps Snowflake’s column casing in field paths and column-level lineage. It is off by default. Turning it on changes every column’s schemaField URN.
  • BigQuery. Sharing linked datasets, multi-column partition info, and dataset profiles for materialized views. The profiler lives in its own package.
  • Cube. Cube views are ingested as Semantic Models. emit_semantic_model_entities: true emits first-class semantic model, metric, and semantic-model dataset entities. That path needs METRICS_ENABLED=true.
  • Dataplex. Metadata Export API extraction.
  • Metabase. Column-level lineage, model ingestion, and collection tags and containers.
  • MicroStrategy. Project schema can be ingested as a Semantic Model.
  • Power BI. Lineage for BigQuery EXTERNAL_QUERY.
  • SAP Analytics Cloud. Column-level lineage to SAP Datasphere for DWC live models, and SAC to SAP Datasphere live-model lineage.
  • SAP Datasphere. Calculated column formulas are written into the column description.
  • Sigma. Dataset lineage is recovered from Sigma’s connection metadata. See Breaking Changes.
  • Kafka Connect. SQL Server is a three-level platform. See Breaking Changes.
  • Looker. Per-field explore usage from System Activity.
  • Airbyte. Inactive connections can be ingested by opting in. Doris can ingest Iceberg tables in non-default catalogs.

Ingestion infrastructure:

  • Connector support tiers are Alpha, Beta, and GA. The old names Certified, Incubating, and Testing are removed.
  • SQL lineage timeout is configurable.
  • Structured property settings can be ingested from the Python SDK.
  • Stdin envelope secrets are registered for masking when they arrive.
  • confluent-kafka is 2.15.1. A custom oauth_cb must return an absolute expiry time, not a lifetime in seconds.

Breaking Changes

Full actions are in Updating DataHub.

  • Sigma ingestion. Sigma Dataset lineage URNs are built from connection_to_platform_map rather than chart_sources_platform_mapping. Add the Sigma connectionId to that map with the env and platform_instance your warehouse connector used. Set convert_urns_to_lowercase: true if you want the previous spelling for those table names.
  • Structured property hard-delete. An active structured property cannot be hard-deleted, and its propertyDefinition aspect cannot be deleted directly, until the property is soft-deleted. The UI does both steps. Automation that issued one hard delete must soft-delete, then hard-delete.
  • Timeseries read authorization. API clients that read dataset profile, usage, or operations with only entity GET need View Dataset Profile, View Dataset Usage, or View Dataset Operations. Dedicated timeseries APIs also require Get Timeseries Aspect API when REST API authorization is enabled.
  • Role and group membership. Adding a role (roleMembership) requires Manage Policies. Adding a group member requires Edit Group Members. Adding a group owner requires Edit Owners. When the acting user is the one being added, both require Manage Users & Groups. Edit Entity on a user is no longer enough to grant that user Admin.
  • Analytics API. POST /openapi/v2/analytics/datahub_usage_events/_search requires Analytics API access or Manage System Operations. The in-app analytics dashboard still uses View Analytics.
  • Compliance form assignment. batchAssignForm, batchRemoveForm, createDynamicFormAssignment, and writes that change which forms are on an asset, require Manage Compliance Forms. Completing an assigned form is unchanged.
  • Asset summary settings. updateAssetSettings and writes to assetSettings require Edit Entity or Manage Asset Summary on the asset. OpenAPI and Rest.li still treat the write as a generic entity update, so Manage Asset Summary alone is not enough there.
  • Never-expiring access tokens. Disabled unless ACCESS_TOKEN_ALLOW_NO_EXPIRY=true. Set ACCESS_TOKEN_ALLOWED_DURATIONS to change the allowed TTLs.
  • OpenAPI create is per URN. A principal with only Create Entity cannot overwrite an existing entity through create or UPSERT. Grant Edit Entity to callers that update existing entities.
  • Domain-scoped create. Domain-scoped Create and Edit policies are evaluated against the domains proposed on a new entity. A domains PATCH requires Edit Entity, and domain-scoped Edit must allow both the previous and the new domain membership.
  • Group member removal. removeGroupMembers removes members who only have the legacy groupMembership aspect.
  • Documents. The default Editor role no longer has Manage Documents. Editors can create, edit, and move documents, and can delete documents they own.
  • Password hashes. Reading corpUserCredentials requires Manage User Credentials.
  • Kafka Connect. SQL Server sink URNs gain a dbo schema when schema.name is unset (database.table becomes database.dbo.table). A SQL Server source dataset with no resolvable schema is dropped with a warning. Oracle JDBC sinks emit schema.table.
  • Elasticsearch ingestion. acryl-datahub[elasticsearch] requires opensearch-py 3.x. The 3.x client still talks to OpenSearch 2 clusters.
  • Great Expectations SQL profiler. Removed. The SQLAlchemy profiler is the only SQL profiler. profiling.method and the acryl-datahub[profiling-ge] extra are gone. This does not affect the Great Expectations integration that ingests GX validation results.
  • MySQL, MariaDB, Doris, and TiDB profiling limits. profiling.profile_table_row_limit and profiling.profile_table_size_limit are enforced. They were previously accepted and ignored. Both still default to unset on these sources.
  • sql-queries. A run where every input line is unparseable, or every query fails, exits non-zero. enable_lazy_schema_loading is removed. Report fields num_queries_processed_sequential and num_temp_tables_detected are renamed.
  • Python SDK circuit breakers. Assertion and Operation clients take graph= instead of transport=. GraphQL errors raise GraphError.
  • Connector support status names. SupportStatus.CERTIFIED, INCUBATING, and TESTING are removed. Use GA, BETA, and ALPHA.
  • Python extras. notion, confluence, unstructured, and datahub-documents require Python 3.11+ and are no longer installed by acryl-datahub[all].
  • Kafka OAuth callbacks. Since confluent-kafka 2.13, the second value returned by oauth_cb is an absolute expiry in epoch seconds. A callback that returns a lifetime such as 3600 now looks already expired.
  • Search indexing startup. GMS refuses to start when it consumes MCLs and both UI indexing paths are off (PRE_PROCESS_HOOKS_UI_ENABLED=false and PRE_PROCESS_HOOKS_REPROCESS_ENABLED unset or false).
  • pgQueue idle sleep. DATAHUB_PGQUEUE_EMPTY_POLL_SLEEP_MS and DATAHUB_PGQUEUE_MCL_EMPTY_POLL_SLEEP_MS are removed. Use the new min and per-topic max settings. pgQueue maintenance intervals must divide the clock (for example 15 minutes, not 7).
  • OpenAPI token recipes. get_token is a typed object and cannot be combined with token or bearer_token.
  • Actions Kafka. A pipeline whose Kafka source cannot connect stops, and the actions process exits, instead of staying up while consuming nothing.
  • Log line format. datahub-frontend, datahub-upgrade, and GMS log lines no longer include the source line number.

Deprecations

  • OpenAPI v2 entity and relationship APIs (/openapi/v2/entity, /openapi/v2/relationship) are deprecated in favor of /openapi/v3/entity and /openapi/v3/relationship. Timeseries, platform entity ingest, and timeline under v2 are not deprecated by this.
  • semanticModelInfo.datasets. Membership is stored on the member. Datasets use semanticModelProperties.semanticModel. Metrics use metricInfo.semanticModel. Re-ingest Semantic Models after upgrade so metric-to-dataset lineage is populated. It is missing between the upgrade and that ingest.

Security

Known Issues

None recorded in the v1.8.0 upgrade notes.

Assets 2
Loading