Skip to content

v0.13.0

Choose a tag to compare

View all tags
@ravisuhag ravisuhag released this 19 Apr 04:37
· 16 commits to main since this release
v0.13.0
f709abf

Highlights

This release introduces structured entity/edge metadata, a restructured CLI, and significant improvements to config validation and data consistency.

New Features

  • Structured entity/edge metadata — All 31 extractors now declare their entity types, edge types, and URN patterns via plugins.Info.
  • Restructured CLI — New command structure: plugins list/info, recipe init/gen, top-level entities and edges commands with filtering support (--entity-type, --edge-type, --tag).
  • Notion extractor — Extract page and database metadata from Notion.
  • Confluence extractor — Extract page metadata and relationships from Confluence spaces.
  • dbt extractor — Extract model and source metadata from dbt manifests.
  • GitHub collaborator permissions — Extract collaborator access levels.
  • BigQuery label-based exclusion — Exclude tables by label key-value pairs.

Improvements

  • Config validation tightened — Extract fields validated with oneof, URL format validation on host fields, file existence checks, port ranges, conditional auth field requirements (Kafka TLS/SASL).
  • Edge serialization fixed — Kafka and HTTP sinks now include edges in output (previously silently dropped).
  • Tags standardized — All plugins use consistent platform/domain taxonomy (e.g. gcp/database, oss/bi, saas/collaboration).
  • Descriptions standardized — Consistent format across all plugins.
  • Exclude patterns standardized — Postgres, CouchDB, Redshift, and Presto migrated from comma-separated strings to structured exclude configs.
  • pkg/errors migrated to stdlib — All direct usage replaced with fmt.Errorf and %w wrapping.

Bug Fixes

  • Redshift exclude — The exclude config field was defined but never actually used to filter databases. Now wired up correctly.

Breaking Changes

  • CLI commands renamed — list extractors → plugins list --type extractor, info extractor <name> → plugins info <name>, new recipe → recipe init, gen → recipe gen.
  • Exclude config format — Postgres, CouchDB, Redshift, and Presto now use structured exclude (e.g. exclude: { databases: [db1] }) instead of comma-separated strings.
  • Kafka sink output format — Messages are now JSON (entity + edges) instead of protobuf entity-only.

Full Changelog: v0.12.0...v0.13.0

Assets 15
Loading