Fix #2946: Presto Ingestion issues fix #2986

ulixius9 · 2022-02-25T07:41:48Z

Describe your changes :

Fix #2946: Presto Ingestion issues fix

The error [2022-02-25 12:43:58,436] ERROR {metadata.ingestion.source.sql_source:170} - Failed to generate sample data for dbgen_version - (pyhive.exc.DatabaseError) {'message': "Could not serialize column 'dv_create_time' of type 'time(3)' at position 1:3" is a bug with presto and has been fixed in later version of trino.

Although while ingesting presto I noticed datatype related errors

/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'char(16)' of column 'w_warehouse_id'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))
/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'varchar(20)' of column 'w_warehouse_name'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))
/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'char(10)' of column 'w_street_number'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))
/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'varchar(60)' of column 'w_street_name'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))
/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'char(15)' of column 'w_street_type'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))
/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'char(10)' of column 'w_suite_number'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))
/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'varchar(60)' of column 'w_city'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))
/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'varchar(30)' of column 'w_county'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))
/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'char(2)' of column 'w_state'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))
/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'char(10)' of column 'w_zip'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))
/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'varchar(20)' of column 'w_country'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))
/Users/mayursingal/Desktop/ulixius1/OpenMetadata/venv/lib/python3.8/site-packages/pyhive/sqlalchemy_presto.py:151: SAWarning: Did not recognize type 'decimal(5,2)' of column 'w_gmt_offset'
  util.warn("Did not recognize type '%s' of column '%s'" % (row.Type, row.Column))

which I've fixed in this PR

Type of change :

Bug fix

Checklist:

I have read the CONTRIBUTING document.
I have performed a self-review of my own.
I have tagged my reviewers below.
I have commented on my code, particularly in hard-to-understand areas.
My changes generate no new warnings.
I have added tests that prove my fix is effective or that my feature works.
All new and existing tests passed.

Reviewers

@harshach @ayush-shah @pmbrull

harshach · 2022-02-25T17:27:30Z

ingestion/src/metadata/ingestion/source/presto.py

 from metadata.generated.schema.entity.services.databaseService import (
    DatabaseServiceType,
 )
 from metadata.ingestion.ometa.openmetadata_rest import MetadataServerConfig
 from metadata.ingestion.source.sql_source import SQLSource
 from metadata.ingestion.source.sql_source_common import SQLConnectionConfig

+_type_map = {
+    "boolean": types.Boolean,
+    "tinyint": mysql.MSTinyInteger,
