🎉 Source Postgres: support all Postgres 14 types

[Liren Tu (tuliren)]

What

• Support all Postgres types.
• Except two remaining issues that we need to confirm on the destination side if the solution works:
  • Postgres source date includes time, and cannot handle BC dates #8903
  • https://github.com/airbytehq/airbyte-internal-issues/issues/358

How

• Postgres does not have a public type enum. So the MySQL approach won't work for Postgres.
• We use JDBCType for most of the types. For corner cases, we check the column type name returned from JDBC, and treat them differently.

Recommended reading order

1. PostgresSourceDatatypeTest.java
2. PostgresSourceOperations.java

🚨 User Impact 🚨

• This PR changes some of the Postgres data types. It is published with a minor version bump.

Pre-merge Checklist

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the new connector version is published, connector version bumped in the seed directory as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

[Liren Tu (tuliren)]

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1571824409
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1571824409
🐛 https://gradle.com/s/ikzuaoro3mups

[Sherif A. Nada (sherifnada)]

does json have a concept of infinity and Nan? how are these represented when output from the connector itself?

[Liren Tu (tuliren)]

Good catch. Actually Json numbers do not support NaN or infinity. To support these three special values, we need to make sure that the destination can handle them. Otherwise, the destination will fail. Too bad, I need to revert this.

[Christophe Duong (ChristopheDuong)]

like how we currently handle dates, you can output them in a string type with a format hint.

Then have normalization handle the special format hint for these special numbers accordingly for the destination

[Liren Tu (tuliren)]

Christophe Duong (@ChristopheDuong), nice.

Where the code that handles this? I'd like to see what the format hint looks like.

[Christophe Duong (ChristopheDuong)]

In the json, it's "string" type.

Then, on normalization, if we find a format hint, we know it's actually a date not a string:

airbyte/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/utils.py

Line 33 in c5fc568

and (definition["format"] == "date" or "date" in definition["format"])

[Christophe Duong (ChristopheDuong)]

Example of catalog.json with date:

airbyte/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/catalog.json

Line 17 in c5fc568

"format": "date"

so you can have a special string-float type instead of using json's number in order to encode special float values:

"db_float_column": {
  "type": "string",
  "format": "float"
}

and normalization can handle it in SQL: For example in bigquery, based on https://stackoverflow.com/a/53692265:

there is no literal representation of NaN or infinity, but the following case-insensitive strings can be explicitly cast to float:

"NaN"
"inf" or "+inf"
"-inf"

case 
  when db_float_column = '-infinity' then cast("-inf" as float) 
  when db_float_column = '+infinity' then cast("+inf" as float)
  else cast(db_float_coumn as float) end

or differently for another destination

[Liren Tu (tuliren)]

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1599988085
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1599988085
🐛 https://gradle.com/s/qcegtyclrphw2

[Liren Tu (tuliren)]

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1600389644
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1600389644
No Python unittests run

[Liren Tu (tuliren)]

/publish connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1600584483
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1600584483