Amundsen Metadata service

Amundsen Metadata service serves Restful API and responsible for providing and also updating metadata, such as table & column description, and tags. Metadata service is using Neo4j as a persistent layer.

Requirements

• Python >= 3.7

Instructions to start the Metadata service from distribution

$ venv_path=[path_for_virtual_environment]
$ python3 -m venv $venv_path
$ source $venv_path/bin/activate
$ pip3 install amundsenmetadata
$ python3 metadata_service/metadata_wsgi.py

-- In different terminal, verify getting HTTP/1.0 200 OK
$ curl -v http://localhost:5000/healthcheck

Instructions to start the Metadata service from the source

$ git clone https://github.com/lyft/amundsenmetadatalibrary.git
$ cd amundsenmetadatalibrary
$ python3 -m venv venv
$ source venv/bin/activate
$ pip3 install -r requirements.txt
$ python3 setup.py install
$ python3 metadata_service/metadata_wsgi.py

-- In different terminal, verify getting HTTP/1.0 200 OK
$ curl -v http://localhost:5000/healthcheck

Instructions to start the service from the Docker

$ docker pull amundsen-metadata
$ docker run -p 5000:5000 amundsen-metadata

-- In different terminal, verify getting HTTP/1.0 200 OK
$ curl -v http://localhost:5000/healthcheck

Production environment

By default, Flask comes with Werkzeug webserver, which is for development. For production environment use production grade web server such as Gunicorn.

$ pip install gunicorn
$ gunicorn metadata_service.metadata_wsgi

Here is documentation of gunicorn configuration.

Configuration outside local environment

By default, Metadata service uses LocalConfig that looks for Neo4j running in localhost. In order to use different end point, you need to create Config suitable for your use case. Once config class has been created, it can be referenced by environment variable: METADATA_SVC_CONFIG_MODULE_CLASS

For example, in order to have different config for production, you can inherit Config class, create Production config and passing production config class into environment variable. Let's say class name is ProdConfig and it's in metadata_service.config module. then you can set as below:

METADATA_SVC_CONFIG_MODULE_CLASS=metadata_service.config.ProdConfig

This way Metadata service will use production config in production environment. For more information on how the configuration is being loaded and used, here's reference from Flask doc.

Developer guide

Code style

• PEP 8: Amundsen Metadata service follows PEP8 - Style Guide for Python Code.
• Typing hints: Amundsen Metadata service also utilizes Typing hint for better readability.

Code structure

Amundsen metadata service consists of three packages, API, Entity, and Proxy.

API package

A package that contains Flask Restful resources that serves Restful API request. The routing of API is being registered here.

Proxy package

Proxy package contains proxy modules that talks dependencies of Metadata service. There are currently two modules in Proxy package, Neo4j and Statsd.

Neo4j proxy module

Neo4j proxy module serves various use case of getting metadata or updating metadata from or into Neo4j. Most of the methods have Cypher query for the use case, execute the query and transform into entity.

Statsd utilities module

Statsd utilities module has methods / functions to support statsd to publish metrics. By default, statsd integration is disabled and you can turn in on from Metadata service configuration. For specific configuration related to statsd, you can configure it through environment variable.

Entity package

Entity package contains many modules where each module has many Python classes in it. These Python classes are being used as a schema and a data holder. All data exchange within Amundsen Metadata service use classes in Entity to ensure validity of itself and improve readability and mainatability.