A Single place to Discover, Collaborate, and Get your data right

Darkseal

Darkseal includes the following:

• Metadata schemas - Defines core abstractions and vocabulary for metadata with schemas for Types, Entities, Relationships between entities. This is the foundation of the Open Metadata Standard.
• Metadata store - Stores metadata graph that connects data assets, user, and tool generated metadata.
• Metadata APIs - For producing and consuming metadata built on schemas for User Interfaces and Integration of tools, systems, and services.
• Ingestion framework - A pluggable framework for integrating tools and ingesting metadata to the metadata store. Ingestion framework already supports well know data warehouses - Google BigQuery, Snowflake, Amazon Redshift, and Apache Hive, and databases - MySQL, Postgres, Oracle, MSSQL, and Guinsoo.
• Metadata User Interface - One single place for users to discover, and collaborate on all data.

Snapshots & Gifs

Welcome

New Service

Explore Tables

Table Lineage

Table Schema

Architecture

Darkseal depends on following components to build a metadata platform:

• JsonSchemas for defining Metadata Schemas
• Dropwizard/Jetty for REST APIs
• MySQL 8.x to store Metadata (Guinsoo is coming)
• ElasticSearch/OpenElasticsearch 7.x to index Metadata and power

Start Darkseal Server

Start From Source

# Clone source
git clone git@github.com:GuinsooLab/darkseal.git

# Package 
mvn clean package -DskipTests

# Prepare database and indexes
./bootstrap/bootstrap_storage.sh create

# Start Server
java -cp catalog-rest-service org.openmetadata.catalog.CatalogApplication server ./conf/openmetadata.yaml

# Start UI
cd openmetadata-ui/src/ui
yarn run start

Start From Built Package

# Clone source
git clone git@github.com:GuinsooLab/darkseal.git

# Package 
mvn clean package -DskipTests

# Change dist dir & un-tar files
cd openmetadata-dist/target
tar zxvf openmetadata-xxx.tar.gz
cd openmetadata-xxx

# Prepare database and indexes
./bootstrap/bootstrap_storage.sh create

# Start Server
./bin/openmetadata.sh start

For more information, please referer to here.

Service Integration

Database Service Support

• Athena
• AzureSQL
• BigQuery
• Clickhouse
• Databricks
• Datalake
• Db2
• DeltaLake
• Druid
• DynamoDB
• Glue
• Hive
• MariaDB
• Mssql
• MySQL
• Oracle
• Pinot
• Postgres
• Presto
• Redshift
• SQLite
• Salesforce
• SingleStore
• Snowflake
• Trino
• Vertica
• GuinsooLab Stack

Message Service Support

• Kafka
• Pulsar

Dashboard Service Support

• Looker
• Metabase
• Mode
• PowerBI
• Redash
• Superset
• Tableau
• Spotrix 🚀

Pipeline Service Support

• Airbyte
• Airflow
• Glue
• Sheenflow

ML Model Service Support

• Mlflow

Documentation & Supports

• Overview
• Install & Deploy
• Quickstart With Darkseal
• Darkseal Main Concepts
• Dearkseal Service Integrates
• Roadmap

Contributors

We ❤️ all contributions, big and small! Check out our CONTRIBUTING guide to get started and let us know how we can help.

Don't want to miss anything? Give the project a ⭐ 🚀

License

Darkseal is released under Apache License, Version 2.0