Skip to content

Latest commit

 

History

History
48 lines (32 loc) · 1.18 KB

README.md

File metadata and controls

48 lines (32 loc) · 1.18 KB
Raw

datahub Ingestion Tool

Introduction

some tool to ingestion [jdbc-database-schema] and [etl-lineage] metadata.

i split the ingestion procedure to two part: [datahub-producer] and different [metadata-generator]

Roadmap

  • datahub-producer load json avro data.
  • add lineage-hive generator
  • add dataset-jdbc generator[include [mysql, mssql, postgresql, oracle] driver]
  • enhance dataset-jdbc generator [hive-driver]
  • enhance lineage-jdbc generator to lazy iterator mode.
  • add lineage-oracle generator
  • enchance avro parser to show error information

Quickstart

  1. install nix and channel
  sudo install -d -m755 -o $(id -u) -g $(id -g) /nix
  curl https://nixos.org/nix/install | sh
  
  nix-channel --add https://nixos.org/channels/nixos-20.03 nixpkgs
  nix-channel --update nixpkgs
  1. load json data to datahub
    cat sample/mce.json.dat | bin/datahub-producer.hs config
  1. parse hive sql to datahub
    ls sample/hive_*.sql | bin/lineage_hive_generator.hs | bin/datahub-producer.hs config
  1. load jdbc schema(mysql, mssql, postgresql, oracle) to datahub
    bin/dataset-jdbc-generator.hs | bin/datahub-producer.hs config