Skip to content

mitdbg/aurum-datadiscovery

Folders and files

NameName
Last commit message
Last commit date
Sep 20, 2020
Nov 30, 2018
Nov 14, 2016
Feb 14, 2017
Oct 3, 2018
Nov 11, 2020
Dec 14, 2018
Feb 3, 2017
May 20, 2019
Jan 12, 2018
Oct 3, 2018
Aug 25, 2016
Jan 12, 2018
Oct 7, 2018
Aug 9, 2017
Dec 14, 2018
Jun 29, 2018
Jun 29, 2018
Sep 23, 2016
Jul 2, 2018
Nov 30, 2018
Dec 14, 2018
Dec 14, 2018
Jul 17, 2019
Dec 5, 2017
Jun 29, 2018
Aug 17, 2018
Jun 29, 2018
Dec 14, 2018
Jun 29, 2018
Nov 11, 2020
Sep 18, 2018
Nov 30, 2018
Nov 20, 2018
Dec 5, 2017
May 20, 2017
Sep 30, 2016
Oct 8, 2016

Repository files navigation

Aurum: Discovering Data in Lakes, Clouds and Databases

Webpage version of this documentation: http://mitdbg.github.io/aurum-datadiscovery/

Aurum helps users identify relevant content among multiple data sources that may consist of tabular files, such as CSV, and relational tables. These may be stored in relational database management systems (RDBMS), file systems, and they may live in cloud services, data lakes or other on-premise repositories.

Aurum helps you find data through different interfaces. The most flexible one is an API of primitives that can be composed to build queries that describe the data of interest. For example, you can write a query that says "find tables that contain a column with name 'ID' and have at least one column that looks like an input column". You can also query with very simple primitives, such as "find columns that contain the keyword 'caffeine'". You can also do more complex queries, such as figuring out what tables join with a table of interest. The idea is that the API is flexible enough to allow a wide range of use cases, and that it works over all data you feed to the system, regardless where these live.

  • Why do I need Aurum? We show you various scenarios in which Aurum has proven useful.

  • Design Rationale A brief explanation of the system architecture and design rationale.

  • Quick Start A guide to setup Aurum and start running some discovery queries.

  • Tutorial A tutorial that walks you through the different aspects of Aurum, from how to write queries using the discovery API, to how to create new connectors to read data from different data sources to how to store data in different stores.

  • FAQ Collection of frequent questions

Aurum is a work in progress, we expect to release its first open-source version in the 4th quarter of 2018. We are happy to accept contributions of the community. If you are interested in contributing take a look at the CONTRIBUTING and feel free to email raulcf@csail.mit.edu We also have a code of conduct:

Code of Conduct

Check the code of conduct for Aurum here:

https://github.com/mitdbg/aurum-datadiscovery/blob/master/CODE_OF_CONDUCT.md

Please, report violations of the code of conduct by sending an email to raulcf@csail.mit.edu