Skip to content
A Generalized Metadata Search & Discovery Tool
TypeScript Java HTML JavaScript CSS Python Other
Branch: master
Clone or download

Latest commit

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github doc: update PR template Mar 23, 2020
contrib add oracle driver (#1613) Mar 28, 2020
datahub-dao Handle optional aspects/fields for CorpUser gracefully Mar 11, 2020
datahub-frontend Add forward slash escape for Elasticsearch queries Feb 6, 2020
datahub-web Bump acorn from 5.7.3 to 5.7.4 in /datahub-web (#1607) Mar 23, 2020
docker build(quickstart): Small fix in quickstart.sh & doc update Mar 23, 2020
docs Update the agenda for April 3rd Mar 31, 2020
gms refactor: remove unused legacy models (#1614) Mar 27, 2020
gradle/wrapper Bump gradle to 5.6.4 and pegasus to 27.7.18 Feb 13, 2020
li-utils Drop LI internal fabrics (#1581) Mar 2, 2020
metadata-builders Revert "metadata-models 54.0.1 -> 58.0.1:" Mar 26, 2020
metadata-dao-impl metadata-models 54.0.1 -> 58.0.1 (#1610) Mar 27, 2020
metadata-dao metadata-models 54.0.1 -> 58.0.1 (#1610) Mar 27, 2020
metadata-events metadata-models 54.0.1 -> 58.0.1 (#1610) Mar 27, 2020
metadata-ingestion Specify python version in mce-cli requirement.txt Mar 27, 2020
metadata-jobs Update README.md Feb 29, 2020
metadata-models Revert "metadata-models 54.0.1 -> 58.0.1:" Mar 26, 2020
metadata-restli-resource metadata-models 54.0.1 -> 58.0.1 (#1610) Mar 27, 2020
metadata-testing metadata-models 54.0.1 -> 58.0.1 (#1610) Mar 27, 2020
metadata-utils Enable tests for all modules by using global gradle config Feb 21, 2020
metadata-validators Bump gradle to 5.6.4 and pegasus to 27.7.18 Feb 13, 2020
.dockerignore Add docker ignore file Sep 3, 2019
.gitignore Add missing MXE models and fix .gitignore Sep 1, 2019
.travis.yml Update travis configuration to optimize build time Oct 7, 2019
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md Mar 23, 2020
CONTRIBUTING.md Update CONTRIBUTING.md Mar 21, 2020
LICENSE Initial commit Nov 19, 2015
README.md doc: styling tweak Mar 29, 2020
build.gradle Enable tests for all modules by using global gradle config Feb 21, 2020
gradle.properties Add gradle.properties & enable parallel Gradle build Mar 12, 2020
gradlew Bump gradle to 5.6.4 and pegasus to 27.7.18 Feb 13, 2020
gradlew.bat Bump gradle to 5.6.4 and pegasus to 27.7.18 Feb 13, 2020
repositories.gradle Bump gradle to 5.6.4 and pegasus to 27.7.18 Feb 13, 2020
settings.gradle Rename elasticsearch-index-job to mae-consumer-job Nov 21, 2019

README.md

DataHub: A Generalized Metadata Search & Discovery Tool

Version Build Status Get on Slack PRs Welcome License


Quickstart | Documentation | Features | Roadmap | FAQ | Town Hall


DataHub

📣 Next DataHub town hall meeting on April 3rd, 9am-10am PDT:

Mar 2020 Update:

  • DataHub v0.3.1 has just been released. See relase notes for more details.
  • We're on Slack now! Ask questions and keep up with the latest announcement.

Introduction

DataHub is LinkedIn's generalized metadata search & discovery tool. To learn more about DataHub, check out our LinkedIn blog post and Strata presentation. You should also visit DataHub Architecture to get a better understanding of how DataHub is implemented and DataHub Onboarding Guide to understand how to extend DataHub for your own use case.

This repository contains the complete source code for both DataHub's frontend & backend. You can also read about how we sync the changes between our the internal fork and GitHub.

Quickstart

  1. Install docker and docker-compose (if using Linux). Make sure to allocate enough hardware resources for Docker engine. Tested & confirmed config: 2 CPUs, 8GB RAM, 2GB Swap area.
  2. Open Docker either from the command line or the desktop app and ensure it is up and running.
  3. Clone this repo and cd into the root directory of the cloned repository.
  4. Run the following command to download and run all Docker containers locally:
    cd docker/quickstart && source ./quickstart.sh
    
    This step takes a while to run the first time, and it may be difficult to tell if DataHub is fully up and running from the combined log. Please use this guide to verify that each container is running correctly.
  5. At this point, you should be able to start DataHub by opening http://localhost:9001 in your browser. You can sign in using datahub as both username and password. However, you'll notice that no data has been ingested yet.
  6. To ingest provided sample data to DataHub, switch to a new terminal window, cd into the cloned datahub repo, and run the following command:
    docker build -t ingestion -f docker/ingestion/Dockerfile . && cd docker/ingestion && docker-compose up
    
    After running this, you should be able to see and search sample datasets in DataHub.

Please refer to the debugging guide if you encounter any issues during the quickstart.

Documentation

Releases

See Releases page for more details. We follow the SemVer Specification when versioning the releases and adopt the Keep a Changelog convention for the changelog format.

FAQs

Frequently Asked Questions about DataHub can be found here.

Features & Roadmap

Check out DataHub's Features & Roadmap.

Contributing

We welcome contributions from the community. Please refer to our Contributing Guidelines for more details. We also have a contrib directory for incubating experimental features.

Community

Join our slack workspace for important discussions and announcements. You can also find out more about our past and upcoming town hall meetings.

Related Articles & Presentations

You can’t perform that action at this time.