PostgresqlFirehoseDatabaseConnector not bundled

[nl-brett-stime]

Affected Version

0.14.2-incubating

Description

In spite of including postgresql-metadata-storage in druid.extensions.loadList, when I try to run a SQL index task with a PostgreSQL source, I get the following:

{"error":"Could not resolve type id 'postgresql' into a subtype of [simple type, class org.apache.druid.metadata.SQLFirehoseDatabaseConnector]: known type ids = [SQLFirehoseDatabaseConnector]\n at [Source: HttpInputOverHTTP@436b4796[c=1881,q=0,[0]=null,s=STREAM]; line: 48, column: 11] (through reference chain: org.apache.druid.indexing.common.task.IndexTask[\"spec\"]->org.apache.druid.indexing.common.task.IndexIngestionSpec[\"ioConfig\"]->org.apache.druid.indexing.common.task.IndexIOConfig[\"firehose\"]->org.apache.druid.segment.realtime.firehose.SqlFirehoseFactory[\"database\"])"}

After a bit of digging it looks like the PostgresqlFirehoseDatabaseConnector isn't included in the main download:

apache-druid-0.14.2-incubating brett.stime$ find . -type f -name *.jar -exec unzip -l {} \; | grep FirehoseDatabaseConnector
     1764  05-10-2019 13:16   org/apache/druid/firehose/sql/MySQLFirehoseDatabaseConnector.class
     4301  05-10-2019 13:14   org/apache/druid/metadata/SQLFirehoseDatabaseConnector.class

[nl-brett-stime]

Nevermind. I think my github-fu failed. I'd selected the tag and searched for PostgresqlFirehoseDatabaseConnector and found results but it maybe the search results were actually from downstream of druid-0.14.2-incubating. Anyway, after cloning and pulling what must've been half of the public Maven repo I'm now seeing that the class isn't present at that tag.

[Suroor-Ahmmad]

@nl-brett-stime I'm facing same issue, can you please explain how you fixed this?

[nl-brett-stime]

@Suroor-Ahmmad This might already be included in 0.15.0-incubating but for 0.14.2-incubating I had to cherry-pick the following commit and build from source: https://github.com/apache/incubator-druid/commits/0fa90008496926c15426710b0dd4698bdc224bac/extensions-core/postgresql-metadata-storage/src/main/java/org/apache/druid/firehose/PostgresqlFirehoseDatabaseConnector.java