Skip to content
No description, website, or topics provided.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
codequality Getting the docker-compose example working on Windows MinGW / MSys (#234 Feb 20, 2018
gradle/wrapper Upgrade gradle and apply change 83316c (#216) Jan 12, 2018
metacat-app Upgrade to Spring Boot 2.0.8 (#325) Feb 27, 2019
metacat-client Upgrade to Spring Boot 2.0.8 (#325) Feb 27, 2019
metacat-common-server Modified to ignore exception that happens after a successful iceberg … Apr 2, 2019
metacat-common #1 Refactored code in CatalogTraversal. #2. Introduced a boolean prop… Mar 11, 2019
metacat-connector-cassandra Supporting table iceberg get partitions Oct 12, 2018
metacat-connector-druid Refactored the concept of ElasticSearchRefresh code to class CatalogT… Mar 7, 2019
metacat-connector-hive Modified to ignore exception that happens after a successful iceberg … Apr 2, 2019
metacat-connector-jdbc Supporting table iceberg get partitions Oct 12, 2018
metacat-connector-mysql Fixed issue where the getTable API was incorrectly returning TableNot… Mar 16, 2018
metacat-connector-pig Upgrade to Spring Boot 2.0.8 (#325) Feb 27, 2019
metacat-connector-postgresql Added support to have one catalog name pointing to multiple data stor… Mar 7, 2018
metacat-connector-redshift Added support to have one catalog name pointing to multiple data stor… Mar 7, 2018
metacat-connector-s3 Refactored the concept of ElasticSearchRefresh code to class CatalogT… Mar 7, 2019
metacat-connector-snowflake Reverting back the table exists change for snowflake (#257) May 17, 2018
metacat-functional-tests Modified to ignore exception that happens after a successful iceberg … Apr 2, 2019
metacat-main Modified to ignore exception that happens after a successful iceberg … Apr 2, 2019
metacat-metadata-mysql Adding type parameter to tag list. (#310) Nov 20, 2018
metacat-testdata-provider Adding iceberg partition dateCreated override (#309) Nov 16, 2018
metacat-thrift Supporting table iceberg get partitions Oct 12, 2018
metacat-war Upgrade to Spring Boot 2.0.8 (#325) Feb 27, 2019
scripts Add setTags() api in tagController (#264) Jun 6, 2018
.editorconfig Adding a notification service with a SNS implementation (#17) Dec 5, 2016
.gitignore Cassandra Type Converter (#66) May 11, 2017
.travis.yml Update Travis to Collect Spring Logs (#327) Jan 29, 2019
LICENSE Initial import. Mar 19, 2016
OSSMETADATA
README.md Update README to more accurately reflect 1.1.0 state (#178) Aug 3, 2017
build.gradle Upgrade to Spring Boot 2.0.8 (#325) Feb 27, 2019
buildViaTravis.sh Fix intermittent build errors because of docker processes exceeding i… May 21, 2018
gradle.properties Upgrade to Spring Boot 2.0.8 (#325) Feb 27, 2019
gradlew Upgrade to Gradle 4. Enable parallel build (#153) Jun 19, 2017
gradlew.bat Dependency management (#78) May 11, 2017
lombok.config Dependency management (#78) May 11, 2017
sample_config.properties Added copyright header. Mar 23, 2016
settings.gradle Added support for snowflake datastore. Jan 30, 2018

README.md

Metacat

Download License Issues NetflixOSS Lifecycle

Introduction

Metacat is a unified metadata exploration API service. You can explore Hive, RDS, Teradata, Redshift, S3 and Cassandra. Metacat provides you information about what data you have, where it resides and how to process it. Metadata in the end is really data about the data. So the primary purpose of Metacat is to give a place to describe the data so that we could do more useful things with it.

Metacat focusses on solving these three problems:

  • Federate views of metadata systems.
  • Allow arbitrary metadata storage about data sets.
  • Metadata discovery

Documentation

TODO

Releases

Releases

Builds

Metacat builds are run on Travis CI here. Build Status

Getting Started

git clone git@github.com:Netflix/metacat.git
cd metacat
./gradlew clean build

Once the build is completed, the metacat WAR file is generated under metacat-war/build/libs directory. Metacat needs two basic configurations:

  • metacat.plugin.config.location: Path to the directory containing the catalog configuration. Please look at catalog samples used for functional testing.
  • metacat.usermetadata.config.location: Path to the configuration file containing the connection properties to store user metadata. Please look at this sample.

Running Locally

Take the build WAR in metacat-war/build/libs and deploy it to an existing Tomcat as ROOT.war.

The REST API can be accessed @ http://localhost:8080/mds/v1/catalog

Swagger API documentation can be accessed @ http://localhost:8080/swagger-ui.html

Docker Compose Example

Pre-requisite: Docker compose is installed

To start a self contained Metacat environment with some sample catalogs run the command below. This will start a docker-compose cluster containing a Metacat container, a Hive Metastore Container, a Cassandra container and a PostgreSQL container.

./gradlew metacatPorts
  • metacatPorts - Prints out what exposed ports are mapped to the internal container ports. Look for the mapped port (MAPPED_PORT) to port 8080.

REST API can be accessed @ http://localhost:<MAPPED_PORT>/mds/v1/catalog

Swagger API documentation can be accessed @ http://localhost:<MAPPED_PORT>/swagger-ui.html

To stop the docker compose cluster:

./gradlew stopMetacatCluster
You can’t perform that action at this time.