Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of dataset lifecycle management, and much more. Marquez was released and open sourced by WeWork.
Marquez is an LF AI & Data Foundation incubation project under active development and we'd love your help!
Marquez provides a simple way to collect and view dataset and job metadata. The easiest way to get up and running is with Docker. From the base of the Marquez repository run:
Tip: Use the
--buildflag to build images from source, and/or
--seedto load with seed data.
You can open http://localhost:3000 to begin exploring the web UI. The UI enables you to discover dependencies between jobs and the datasets they produce and consume via the lineage graph, view run metadata of current and previous job runs, and much more!
The HTTP API listens on port
5000 for all calls and port
5001 for the admin interface. The admin interface exposes helpful endpoints like
/metrics. To verify the HTTP API server is running and listening on localhost browse to http://localhost:5001.
Note: By default, the HTTP API does not require any form of authentication or authorization.
We invite everyone to help us improve and keep documentation up to date. Documentation is maintained in this repository and can be found under
Note: To begin collecting metadata with Marquez, follow our quickstart guide. Below you will find the steps to get up and running from source.
Marquez uses a multi-project structure and contains the following modules:
api: core API used to collect metadata
web: web UI used to view metadata
clients: clients that implement the HTTP API
integrations: integrations with other systems (ex:
chart: helm chart
Note: To connect to your running PostgreSQL instance, you will need the standard
To build the
api module run:
$ ./gradlew :api:shadowJar
The executable can be found under
To run Marquez, you will have to define
marquez.yml. The configuration file is passed to the application and used to specify your database connection. The configuration file creation steps are outlined below.
Step 1: Create Database
When creating your database using
createdb, we recommend calling it
$ createdb marquez
Step 2: Create
With your database created, you can now copy
$ cp marquez.example.yml marquez.yml
You will then need to set the following environment variables (we recommend adding them to your
POSTGRES_PASSWORD. The environment variables override the equivalent option in the configuration file.
By default, Marquez uses the following ports:
- TCP port
8080is available for the HTTP API server.
- TCP port
8081is available for the admin interface.
Note: All of the configuration settings in
marquez.ymlcan be specified either in the configuration file or in an environment variable.
$ ./gradlew :api:runShadow
Then browse to the admin interface: http://localhost:8081
OpenLineage: open standard for metadata and lineage collection
- Website: https://marquezproject.ai
- Source: https://github.com/MarquezProject/marquez
- Chat: https://marquezproject.slack.com
- Twitter: @MarquezProject
See CONTRIBUTING.md for more details about how to contribute.