Discover Schema sets Namespace field. #2767
Conversation
…le depending on where is class is located. E.g. this does not work when running tests using the Intellij runner. Write to a tmp file instead.
|
/test connector=connectors/source-postgres
|
|
/test connector=connectors/source-mysql
|
|
/test connector=connectors/source-mssql
|
|
/test connector=connectors/source-jdbc
|
|
/test connector=connectors/source-oracle
|
airbyte-commons/src/main/java/io/airbyte/commons/resources/MoreResources.java
Outdated
Show resolved
Hide resolved
davinchia
requested review from
cgardens and
jrhizor
and removed request for
michel-tricot and
ChristopheDuong
|
Aside form the e2e_docker test and the oracle test (both of which are passing on my local system but failing on here, not sure why, will look tmrw), this is done. I am going to hold off on any connector version release until my next PR to remove the schema name from the stream name. |
| @@ -49,10 +49,18 @@ public static AirbyteStream createAirbyteStream(String streamName, Field... fiel | |||
| return createAirbyteStream(streamName, Arrays.asList(fields)); | |||
| } | |||
|
|
|||
| public static AirbyteStream createAirbyteStream(String streamName, String schemaName, Field... fields) { | |||
There was a problem hiding this comment.
Some method duplication here now; I will remove them in my next PR as part of removing schema names from stream names.
airbyte-commons/src/main/java/io/airbyte/commons/resources/MoreResources.java
Outdated
Show resolved
Hide resolved
...ectors/source-jdbc/src/main/java/io/airbyte/integrations/source/jdbc/AbstractJdbcSource.java
Show resolved
Hide resolved
Last step (besides documentation) of namespace changes. This is a follow up to #2767 . After this change, the following JDBC sources will change their behaviour to the behaviour described in the above document. Namely, instead of streamName = schema.tableName, this will become streamName = tableName and namespace = schema. This means that, when replicating from these sources, data will be replicated into a form matching the source. e.g. public.users (postgres source) -> public.users (postgres destination) instead of current behaviour of public.public_users. Since MySQL does not have schemas, the MySQL source uses the database as it's namespace. I cleaned up some bits of the CatalogHelpers. This affected the destinations, so I'm also running the destination tests.
What
This PR is step 5 of this tech spec.
Follow up PR to #2704.
The first of (at least) 2 PRs to implement this on the source side. I made some headway before deciding to break the changes into one PR implementing this for discover schema job, and another PR implementing this for read. The combined PR would have been too big otherwise.
I had to refactor the
MoreResourcesclass along the way. Left inline comments explaining why.I've scoped the changes pretty tightly by commit, so feel free to review by commit (especially for the
MoreResourceschange).How
TableInfoclass inAbstractJdbcSource. Use this to set an Airbyte Stream's namespace field in the Catalog. Update tests.Pre-merge Checklist
Recommended reading order
AbstractJdbcSourcefor main change andAbstractJdbcSourceStandardTestfor matching testing changes (Discover changes).config.yaml,CatalogConverterandAcceptanceTest(understand API changes).ManyResources(to understand the method that was failing for me)The various test classes propagate patterns in the above classes, so not terribly interesting.