Add SQL database to graph conversion tool (Db2Graph) #99
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ryansun117
changed the title
- delta change to avoid div by zero error - limit size max 1 billion as too laarge values lead to postgres error
JasonMoho
requested changes
May 18, 2022
Renamed the file to marius_db2graph to align with commands like marius_preprocess; Created two new functions for get_fetch_size to avoid duplicate code; Added the marius_db2graph command to setup.cfg (but haven't tested it because I'm not sure if pip installing it right now would work); Added 'my-sql' as an option to use mysql-connector because this wasn't added previously;
JasonMoho
requested changes
May 21, 2022
JasonMoho
reviewed
May 21, 2022
JasonMoho
requested changes
May 22, 2022
Renamed edge_entity_entity_queries to edge_queries, edge_entity_entity_queries_list to edge_queries_list, and edge_entity_entity_rel_list to edge_rel_list
JasonMoho
requested changes
May 25, 2022
There was a problem hiding this comment.
The example needs some work.
There is a lot of setup that is not relevant to the tool and my attempt to run the example as written failed. Also, the fact that the dataset requires creating a Kaggle account and does not have a simple download link is a problem.
Main todos:
- Create a dockerfile which performs all setup (Postgres setup, database download and setup, marius install with pip) up to the creation of the conf/config.yaml and conf/edges_queries.txt files
- Find a new dataset (or a version of this dataset) which can be downloaded easily without any account requirements.
Once this is fixed I will do another attempt
Added dockerfile, run.sh, sakila.yaml Modified basicConfig in marius_db2graph.py to avoid Python version issue
…e to install from marius main
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introducing a new feature to Marius: Db2Graph, a SQL database to graph conversion tool. Db2Graph converts relational databases into graphs as sets of triples which can be used as input datasets for Marius, allowing streamlined preprocessing from database to Marius.
Db2Graph is contained in Marius but can be used as a standalone tool. Db2Graph currently supports graph conversion from three relational database management systems: MySQL, MariaDB, and PostgreSQL. Conversion with Db2Graph is achieved in the following steps:
This pull request adds the source file
src/python/tools/db2graph/db2graph.pyand a documentation pagedocs/db2graph/db2graph.rstwhich describes the requirements, definitions, and steps for using Db2Graph, and a real example use case.Testing is provided using pytest and GitHub actions to validate the correctness of the db2graph functions.