DataHub v0.8.32

Release Highlights

User Experience

We're excited to announce View-based RBAC Policies! You can now create and apply view-only permissions to your DataHub end-users, providing more robust access controls.

We've also included some small (but impactful!) improvements to UX, including:

• Display recent search terms when beginning the search flow
• Consistently displaying entity subtypes for dbt, Looker, Kafka, & more. Think: Kafka entities are displayed as "topics" instead of "datasets"

Ingestion Highlights

• New! Protobuf ingestion (shoutout to @leifker for this Community-led contribution!)
• Initial work to support a "Notebook" entity (shoutout to @tc350981 for spearheading this work!!)
• Stateful ingestion for dbt is now supported
• Ongoing improvements to our Tableau ingestion source from @nandacamargo & @cuong-pham
• Improvements to handling database aliases for Redshift ingestion
• Improvements to S3 source:
  • Add containers for datasets
  • Support platform_instance
  • Support for folder level datasets
  • Increased flexibility to specify dataset paths
• Ingestion Fixes:
  • Snowflake Usage - log warning instead of error out & other error handling
  • Snowflake allow/deny patterns
  • Examples of allow/deny patterns added to docs

Full Commit Log

• #4570 @gabe-lyons fix(search): handle commas in search queries in the UI
• #4557 @daha fix: replace direct and indirect references to linkedin with datahub-project
• #4569 @dexter-mh-lee fix(policy): Add view entity page priv to all entity types
• #4567@anshbansal fix(bigquery): missing dependency
• #4548 @mayurinehate fix(tableau): gracefully stop ingestion if tableau sign in not successful
• #4564 @dexter-mh-lee fix(docs): fix logo links on ingestion docs
• #4396 @Abhiram98 feat(ingestion): schema, table filtering for redshift-usage
• #4554 @darapuk (fix): Update path generated when creating LookML URL
• #4549 @maggiehays docs: add sumup logo
• #4560 @mhw docs: Fix PostgreSQL typo in features.md
• #4562 @gabe-lyons feat(lineage): show fully qualified dataset name on expansion
• #4561 @anshbansal fix: dependencies for usage sources
• #3782 @CorentinDuhamel feat(ingest): indent sql queries for usage sources
• #4551 @pedro93 fix(rollback) Removes status & key aspects from affected aspects count during rollback
• #4538 @dexter-mh-lee fix(policy): Remove all from the resource type choices
• #4544 @anshbansal fix(ingest): snowflake-usage - log warning instead of error out
• #4542 @mattmatravers build(ui): allow custom nodeDistBaseUrl
• #4545 @mayurinehate fix(kafka-connect): add platform for default case in jdbc connector, update tests for platform instance map
• #4547 @anshbansal chore: update pull request template
• #4537 @RyanHolstien fix(dataPlatformInstance): add data platform instance to entity registry
• #4375 @mayurinehate fix(kafka-connect): fix lineage for postgres-like 3-level hierarchy d…
• #4492 @RyanHolstien fix(cli): delete - handle case insensitive entity types
• #4130 @sgomezvillamor feat(ingest): glue - adds platform instance capability
• #4456 @mohdsiddique feat(stateful dbt): add stateful ingestion capability in dbt source
• #4482 @pedro93 feat(platform): adds side-effect report for rollbacks
• #4275 @maggiehays docs: Ingestion Source Docs Template
• #4470 @mayurinehate feat(tableau): emit lineage edge from embedded datasource to upstream…
• #4535 @pppsunil feat(ingestion): Support pluggable Schema Registry for Kafka Source
• #4369 @eburairu feat(ui): Add new loading pattern logo
• #4493 @leifker feat(integration): protobuf - additional annotations and features
• #4435 @zhoxie-cisco perf(docker): datahub-gms - add jetty configuration xml
• #4532 @anshbansal doc: add example of profiling in default example
• #4533 @anshbansal doc: clarify CLI releases
• #4528 @daha fix(doc): Change to forward slash-separated strings, as in the example
• #4315 @daha fix(docs): Minor fixes to pip install commands
• #4521 @anshbansal doc: update docker docs for mentioning Python CLI
• #4523 @kevinhu fix(ingest): mssql - support database_alias
• #4501 @arunvasudevan feat(ingest): kafka-connect - support mapping for multiple DB instances
• #4477 @jjoyce0510 feat(metadata service): Introducing Platform Events
• #4526 @jjoyce0510 Adding has container
• #4494 @cuong-pham fix(ingest): make tableau ingestion more resilient to error
• #4519 @anshbansal doc(ingestion): add examples of running in docker and Kubernetes
• #4485 @andres-lowrie docs(metadata-ingestion): callout props in para
• #4511 @RyanHolstien Oss/urn validation
• #4525 @dexter-mh-lee feat(policy): Add tooltip and view button
• #4507 @mayurinehate feat(assertion): update python example, assertion entity doc
• #4513 @kevinhu feat(ingestion): detect and disable telemetry in CI
• #4516 @dexter-mh-lee feat(policy): Add domain based and view based policies
• #4520 @rslanka Fix: Snowflake Table to View lineage
• #4510 @anshbansal feat(ingest): Add config to improve user exp for initial ingestion and fix docs
• #4517 @anshbansal feat(ingest): option for number of workunits in preview
• #4503 @treff7es feat(ingest): athena - set Athena location as upstream
• #4490 @MugdhaHardikar-GSLab feat(s3): add s3 source
• #4468 @tc350981 feat(notebook): graphqul related logic change for notebook
• #4401 @anshbansal doc: update instructions for updating DataHub on quickstart
• #4500 @ShubhamThakre SecretBuilderModal -> name field validation updated
• #4505 @anshbansal docs: add example of database and schema allow/deny patterns
• #4504 @anshbansal fix(snowflake): allow/deny patterns
• #4496 @shirshanka feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently
• #4480 @anshbansal feat(snowflake): stop querying for usage data when no mix/max dates
• #4483 @anshbansal fix(snowflake-usage): do not ingest for stage as a dataaset
• #4497 @shirshanka moving to dockerhub for actions container
• #4475 @darapuk fix: Update GroupProfile to read from properties over deprecated info aspect
• #4489 @anshbansal fix(ingestion): pin Jinja2 to version < 3.1.0
• #4484 @anshbansal fix(ingestion): stop CLI build failures
• #4467 @anshbansal doc: add caveats to snowflake doc
• #4481 @anshbansal fix(snowflake): don't recommend accountadmin role for snowflake
• #4479 @anshbansal fix: change log level to debug
• #4476 @dexter-mh-lee Add recent searches filtering
• #4469 @tc350981 feat(ingest): add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn
• #4439 @ShubhamThakre Feature/modal-validation-and-UI-fixes-updates
• #4398 @kevinhu feat(ingest): simplify event IDs for function invocations
• #4474 @sgomezvillamor chore: acryl-data 0.6.12
• #4473 @treff7es fix(redshift) Properly handling database alias in redshift usage and redshift lineage generation
• #4471 @gabe-lyons enabling ml tabs
• #4460 @eclaassen-pb fix: java dependency vulnerabilities
• #4453 @treff7es feat(ingest) data-lake: Add s3 properties metadata when ingesting s3 files
• #4464 @anshbansal fix: change for repository change
• #4466 @anshbansal fix(snowflake-usage): add more error handling
• #4445 @nandacamargo fix(ingest): add fix to tableau connector when table has None fields
• #4457 @mayurinehate docs(hive): update recipe with example to specify kerberos auth
• #4313 @pedro-iatzky fix(ingest): bigquery - fix ingestion of external tables
• #4462 @gabe-lyons adding final transport options
• #4223 @tc350981 feat(notebook): add data models for Notebook entity
• #4450 @pedro93 feat(frontend) Adds multiple group claim support
• #4442 @anshbansal feat(ingestion): snowflake, bigquery - enhancements to log and bugfix
• #4333 @anshbansal doc: add guide for ui tabs
• #4443 @jjoyce0510 Fixing privilege option display bug
• #4451 @gabe-lyons fix tableau connector when it cannot connect to URI
• #4237 @tc350981 (docs) add RFC file to introduce Notebook entity data model
• #4446 @kevinneville fix: Replace old repository link with new link
• #4447 @cuong-pham getting database directly from upstream tables incase there are multiple databases in upstreamDatabases
• #4436 @treff7es Passing entity properly on deletion
• #4433 @rslanka Fix bug in the SchemaField type computation for AVRO logical types.