Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of dataset lifecycle management, and much more. Marquez was released and open sourced by WeWork.

Badges

Status

Marquez is an LF AI Foundation incubation project and under active development at Datakin (in collaboration with many other organizations).

Quickstart

The Marquez API provides a simple way to collect metadata. The easiest way to get up and running is with Docker. From the base of the Marquez repository run:

$ docker-compose up

Marquez listens on port 5000 for all API calls and port 5001 for the admin interface. To verify the HTTP API server is running and listening on localhost browse to http://localhost:5001.

Note: By default, the HTTP API does not require any form of authentication or authorization.

Documentation

We invite everyone to help us improve and keep documentation up to date. Documentation is maintained in this repository and can be found under docs/.

Note: To begin collecting metadata with Marquez, follow our quickstart guide. Below you will find the steps to get up and running from source.

Requirements

• Java 11
• PostgreSQL 12.1

Note: To connect to your running PostgreSQL instance, you will need the standard psql tool.

Building

To build the entire project run:

$ ./gradlew shadowJar

The executable can be found under build/libs/

Configuration

To run Marquez, you will have to define marquez.yml. The configuration file is passed to the application and used to specify your database connection. The configuration file creation steps are outlined below.

Step 1: Create Database

When creating your database using createdb, we recommend calling it marquez:

$ createdb marquez

Step 2: Create marquez.yml

With your database created, you can now copy marquez.example.yml:

$ cp marquez.example.yml marquez.yml

You will then need to set the following environment variables (we recommend adding them to your .bashrc): POSTGRES_DB, POSTGRES_USER, and POSTGRES_PASSWORD. The environment variables override the equivalent option in the configuration file.

By default, Marquez uses the following ports:

• TCP port 8080 is available for the HTTP API server.
• TCP port 8081 is available for the admin interface.

Note: All of the configuration settings in marquez.yml can be specified either in the configuration file or in an environment variable.

Running the Application

$ ./gradlew runShadow

Then browse to the admin interface: http://localhost:8081

Related Projects

• marquez-web: Web UI used to view metadata collected and cataloged by Marquez.
• marquez-airflow: Airflow support for Marquez.
• marquez-java: Java client for Marquez.
• marquez-python: Python client for Marquez.

Getting Involved

• Website: https://marquezproject.ai
• Source: https://github.com/MarquezProject/marquez
• Chat: https://gitter.im/marquez-project/community
• Twitter: @MarquezProject

Contributing

See CONTRIBUTING.md for more details about how to contribute.