Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Source DB2: Schemas and tables where the connected user doesn't have privileges appear on Airbyte UI #8171

Closed
alexandertsukanov opened this issue Nov 22, 2021 · 0 comments · Fixed by #9875
Closed

Source DB2: Schemas and tables where the connected user doesn't have privileges appear on Airbyte UI #8171

alexandertsukanov opened this issue Nov 22, 2021 · 0 comments · Fixed by #9875
Assignees
@sashaNeshcheret
Labels
area/connectors Connector related issues area/databases connectors/source/db2 connectors/sources-database type/bug Something isn't working

Comments

@alexandertsukanov
Copy link
Contributor

Enviroment

  • Airbyte version: v0.32.25-alpha
  • OS Version / Instance: macOS
  • Deployment: Docker
  • Source Connector and version: Source DB2 0.1.3
  • Destination Connector and version: Destination PostgreSQL 0.3.12
  • Severity: High
  • Step where error happened: Sync job / Setup new connection

Current Behavior

Schemas and tables where the connected user doesn't have privileges appear on Airbyte UI.

Expected Behavior

Schemas and tables where the connected user doesn't have privileges don't appear on Airbyte UI

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG 

2021-11-22 09:35:39 INFO () WorkerRun(call):49 - Executing worker wrapper. Airbyte version: dev
2021-11-22 09:35:39 INFO () TemporalAttemptExecution(get):116 - Executing worker wrapper. Airbyte version: dev
2021-11-22 09:35:39 WARN () Databases(createPostgresDatabaseWithRetry):41 - Waiting for database to become available...
2021-11-22 09:35:39 INFO () JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready...
2021-11-22 09:35:39 INFO () Databases(createPostgresDatabaseWithRetry):58 - Database available!
2021-11-22 09:35:39 INFO () DefaultReplicationWorker(run):82 - start sync worker. job id: 44 attempt id: 0
2021-11-22 09:35:39 INFO () DefaultReplicationWorker(run):91 - configured sync modes: {ANOTHER_SOURCE_SCHEMA.ANOTHER_SOURCE_TABLE=full_refresh - append}
2021-11-22 09:35:39 INFO () DefaultAirbyteDestination(start):64 - Running destination...
2021-11-22 09:35:39 INFO () LineGobbler(voidCall):82 - Checking if airbyte/destination-postgres:0.3.12 exists...
2021-11-22 09:35:40 INFO () LineGobbler(voidCall):82 - airbyte/destination-postgres:0.3.12 was found locally.
2021-11-22 09:35:40 INFO () DockerProcessFactory(create):125 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/44/0 --log-driver none airbyte/destination-postgres:0.3.12 write --config destination_config.json --catalog destination_catalog.json
2021-11-22 09:35:40 INFO () LineGobbler(voidCall):82 - Checking if airbyte/source-db2:0.1.3 exists...
2021-11-22 09:35:40 INFO () LineGobbler(voidCall):82 - airbyte/source-db2:0.1.3 was found locally.
2021-11-22 09:35:40 INFO () DockerProcessFactory(create):125 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/44/0 --log-driver none airbyte/source-db2:0.1.3 read --config source_config.json --catalog source_catalog.json --state input_state.json
2021-11-22 09:35:40 INFO () DefaultReplicationWorker(lambda$getDestinationOutputRunnable$3):226 - Destination output thread started.
2021-11-22 09:35:40 INFO () DefaultReplicationWorker(run):119 - Waiting for source thread to join.
2021-11-22 09:35:40 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):190 - Replication thread started.
�[34msource�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[32mINFO�[m i.a.i.s.d.Db2Source(main):40 - {} - starting source: class io.airbyte.integrations.source.db2.Db2Source
�[34msource�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):76 - {} - Running integration: io.airbyte.integrations.source.db2.Db2Source
�[34msource�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[32mINFO�[m i.a.i.b.IntegrationCliParser(parseOptions):118 - {} - integration args: {read=null, catalog=source_catalog.json, state=input_state.json, config=source_config.json}
�[34msource�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):80 - {} - Command: READ
�[34msource�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):81 - {} - Integration config: IntegrationConfig{command=READ, configPath='source_config.json', catalogPath='source_catalog.json', statePath='input_state.json'}
�[35mdestination�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[32mINFO�[m i.a.i.d.p.PostgresDestination(main):69 - {} - starting destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
�[34msource�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[34msource�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[34msource�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[34msource�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[34msource�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[32mINFO�[m i.a.i.s.r.CdcStateManager(<init>):26 - {} - Initialized CDC state with: null
�[34msource�[0m - 2021-11-22 09:35:42 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:42 �[32mINFO�[m i.a.i.s.r.StateManager(createCursorInfoForStream):118 - {} - No cursor field set in catalog but not present in state. Stream: AirbyteStreamNameNamespacePair{name='ANOTHER_SOURCE_TABLE', namespace='ANOTHER_SOURCE_SCHEMA'}, New Cursor Field: null. Resetting cursor value
�[35mdestination�[0m - 2021-11-22 09:35:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:43 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):76 - {} - Running integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
�[35mdestination�[0m - 2021-11-22 09:35:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:43 �[32mINFO�[m i.a.i.b.IntegrationCliParser(parseOptions):118 - {} - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
�[35mdestination�[0m - 2021-11-22 09:35:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:43 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):80 - {} - Command: WRITE
�[35mdestination�[0m - 2021-11-22 09:35:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:43 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):81 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
�[35mdestination�[0m - 2021-11-22 09:35:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:43 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[35mdestination�[0m - 2021-11-22 09:35:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:43 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[35mdestination�[0m - 2021-11-22 09:35:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:43 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[35mdestination�[0m - 2021-11-22 09:35:43 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:43 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
�[35mdestination�[0m - 2021-11-22 09:35:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:44 �[32mINFO�[m i.a.i.b.s.SshTunnel(getInstance):170 - {} - Starting connection with method: NO_TUNNEL
�[35mdestination�[0m - 2021-11-22 09:35:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:44 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - {} - Write config: WriteConfig{streamName=ANOTHER_SOURCE_TABLE, namespace=ANOTHER_SOURCE_SCHEMA, outputSchemaName=ANOTHER_SOURCE_SCHEMA, tmpTableName=_airbyte_tmp_crk_ANOTHER_SOURCE_TABLE, outputTableName=_airbyte_raw_ANOTHER_SOURCE_TABLE, syncMode=append}
�[35mdestination�[0m - 2021-11-22 09:35:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:44 �[32mINFO�[m i.a.i.d.b.BufferedStreamConsumer(startTracked):124 - {} - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started.
�[35mdestination�[0m - 2021-11-22 09:35:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:44 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):121 - {} - Preparing tmp tables in destination started for 1 streams
�[35mdestination�[0m - 2021-11-22 09:35:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:44 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - {} - Preparing tmp table in destination started for stream ANOTHER_SOURCE_TABLE. schema: ANOTHER_SOURCE_SCHEMA, tmp table name: _airbyte_tmp_crk_ANOTHER_SOURCE_TABLE
�[34msource�[0m - 2021-11-22 09:35:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:44 �[32mINFO�[m i.a.i.s.r.AbstractRelationalDbSource(queryTableFullRefresh):35 - {} - Queueing query for table: ANOTHER_SOURCE_TABLE
�[34msource�[0m - 2021-11-22 09:35:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:44 �[32mINFO�[m i.a.i.s.r.AbstractDbSource(lambda$read$2):121 - {} - Closing database connection pool.
�[34msource�[0m - 2021-11-22 09:35:44 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:44 �[32mINFO�[m i.a.i.s.r.AbstractDbSource(lambda$read$2):123 - {} - Closed database connection pool.
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-551, SQLSTATE=42501, SQLERRMC=TEST;SELECT;ANOTHER_SOURCE_SCHEMA.ANOTHER_SOURCE_TABLE, DRIVER=4.28.11
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.source.relationaldb.AbstractRelationalDbSource.lambda$queryTable$0(AbstractRelationalDbSource.java:64)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:37)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:63)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:119)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.source.db2.Db2Source.main(Db2Source.java:41)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - Caused by: java.lang.RuntimeException: com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-551, SQLSTATE=42501, SQLERRMC=TEST;SELECT;ANOTHER_SOURCE_SCHEMA.ANOTHER_SOURCE_TABLE, DRIVER=4.28.11
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.db.jdbc.StreamingJdbcDatabase.query(StreamingJdbcDatabase.java:104)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.db.jdbc.JdbcDatabase.query(JdbcDatabase.java:126)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.source.relationaldb.AbstractRelationalDbSource.lambda$queryTable$0(AbstractRelationalDbSource.java:61)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	... 20 more
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - Caused by: com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-551, SQLSTATE=42501, SQLERRMC=TEST;SELECT;ANOTHER_SOURCE_SCHEMA.ANOTHER_SOURCE_TABLE, DRIVER=4.28.11
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.b7.a(b7.java:810)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.b7.a(b7.java:66)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.b7.a(b7.java:140)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.k6.c(k6.java:2825)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.k6.d(k6.java:2809)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.k6.a(k6.java:2235)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.k7.a(k7.java:8277)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.t4.ab.i(ab.java:204)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.t4.ab.b(ab.java:94)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.t4.p.a(p.java:32)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.t4.av.i(av.java:150)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.k6.al(k6.java:2204)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.k7.bq(k7.java:3763)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.k7.a(k7.java:4642)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.k7.b(k7.java:4215)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.k7.bd(k7.java:785)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at com.ibm.db2.jcc.am.k7.executeQuery(k7.java:750)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.db.jdbc.StreamingJdbcDatabase.query(StreamingJdbcDatabase.java:94)
�[34msource�[0m - 2021-11-22 09:35:44 ERROR () LineGobbler(voidCall):82 - 	... 22 more
2021-11-22 09:35:44 INFO () DefaultReplicationWorker(run):121 - Source thread complete.
2021-11-22 09:35:44 INFO () DefaultReplicationWorker(run):122 - Waiting for destination thread to join.
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):131 - {} - Preparing tables in destination completed.
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):60 - {} - Airbyte message consumer: succeeded.
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.d.b.BufferedStreamConsumer(close):199 - {} - executing on success close procedure.
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):160 - {} - Finalizing tables in destination started for 1 streams
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - {} - Finalizing stream ANOTHER_SOURCE_TABLE. schema ANOTHER_SOURCE_SCHEMA, tmp table _airbyte_tmp_crk_ANOTHER_SOURCE_TABLE, final table _airbyte_raw_ANOTHER_SOURCE_TABLE
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):178 - {} - Executing finalization of tables.
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):180 - {} - Finalizing tables in destination completed.
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):183 - {} - Cleaning tmp tables in destination started for 1 streams
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - {} - Cleaning tmp table in destination started for stream ANOTHER_SOURCE_TABLE. schema ANOTHER_SOURCE_SCHEMA, tmp table name: _airbyte_tmp_crk_ANOTHER_SOURCE_TABLE
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):192 - {} - Cleaning tmp tables in destination completed.
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):133 - {} - Completed integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
�[35mdestination�[0m - 2021-11-22 09:35:45 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-22 09:35:45 �[32mINFO�[m i.a.i.d.p.PostgresDestination(main):71 - {} - completed destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2021-11-22 09:35:45 INFO () DefaultReplicationWorker(run):124 - Destination thread complete.
2021-11-22 09:35:45 ERROR () DefaultReplicationWorker(run):128 - Sync worker failed.
io.airbyte.workers.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled.
	at io.airbyte.workers.protocols.airbyte.DefaultAirbyteSource.close(DefaultAirbyteSource.java:122) ~[io.airbyte-airbyte-workers-0.32.0-alpha.jar:?]
	at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:126) ~[io.airbyte-airbyte-workers-0.32.0-alpha.jar:?]
	at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:32) ~[io.airbyte-airbyte-workers-0.32.0-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:167) ~[io.airbyte-airbyte-workers-0.32.0-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
2021-11-22 09:35:45 INFO () DefaultReplicationWorker(run):152 - sync summary: io.airbyte.config.ReplicationAttemptSummary@6e23c9e6[status=failed,recordsSynced=0,bytesSynced=0,startTime=1637573739968,endTime=1637573745370]
2021-11-22 09:35:45 INFO () DefaultReplicationWorker(run):161 - Source did not output any state messages
2021-11-22 09:35:45 WARN () DefaultReplicationWorker(run):169 - State capture: No new state, falling back on input state: io.airbyte.config.State@613c7057[state={}]
2021-11-22 09:35:45 INFO () TemporalAttemptExecution(get):137 - Stopping cancellation check scheduling...
2021-11-22 09:35:45 INFO () ReplicationActivityImpl(replicate):121 - sync summary: io.airbyte.config.StandardSyncOutput@70d565a7[standardSyncSummary=io.airbyte.config.StandardSyncSummary@6ae9c1ed[status=failed,recordsSynced=0,bytesSynced=0,startTime=1637573739968,endTime=1637573745370],state=io.airbyte.config.State@613c7057[state={}],outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@74fe090f[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@49f343a9[stream=io.airbyte.protocol.models.AirbyteStream@1250ef6d[name=ANOTHER_SOURCE_TABLE,jsonSchema={"type":"object","properties":{"ID":{"type":"number"},"NAME":{"type":"string"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=ANOTHER_SOURCE_SCHEMA,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=append,primaryKey=[],additionalProperties={}]],additionalProperties={}]]
2021-11-22 09:35:45 INFO () ConfigRepository(updateConnectionState):500 - Updating connection d1f3f609-7c5c-49d0-9c5c-ceaa2484c69c state: io.airbyte.config.State@611d8029[state={}]
2021-11-22 09:35:45 INFO () DatabaseConfigPersistence(updateConfigRecord):258 - Updating STANDARD_SYNC_STATE record d1f3f609-7c5c-49d0-9c5c-ceaa2484c69c
2021-11-22 09:35:45 INFO () TemporalAttemptExecution(get):116 - Executing worker wrapper. Airbyte version: dev
2021-11-22 09:35:45 WARN () Databases(createPostgresDatabaseWithRetry):41 - Waiting for database to become available...
2021-11-22 09:35:45 INFO () JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready...
2021-11-22 09:35:45 INFO () Databases(createPostgresDatabaseWithRetry):58 - Database available!
2021-11-22 09:35:45 INFO () DefaultNormalizationWorker(run):46 - Running normalization.
2021-11-22 09:35:45 INFO () DefaultNormalizationRunner(runProcess):114 - Running with normalization version: airbyte/normalization:0.1.59
2021-11-22 09:35:45 INFO () LineGobbler(voidCall):82 - Checking if airbyte/normalization:0.1.59 exists...
2021-11-22 09:35:45 INFO () LineGobbler(voidCall):82 - airbyte/normalization:0.1.59 was found locally.
2021-11-22 09:35:45 INFO () DockerProcessFactory(create):125 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/44/0/normalize --log-driver none airbyte/normalization:0.1.59 run --integration-type postgres --config destination_config.json --catalog destination_catalog.json
�[32mnormalization�[0m - 2021-11-22 09:35:46 INFO () LineGobbler(voidCall):82 - Running: transform-config --config destination_config.json --integration-type postgres --out /data/44/0/normalize
�[32mnormalization�[0m - 2021-11-22 09:35:46 INFO () LineGobbler(voidCall):82 - Namespace(config='destination_config.json', integration_type=<DestinationType.postgres: 'postgres'>, out='/data/44/0/normalize')
�[32mnormalization�[0m - 2021-11-22 09:35:46 INFO () LineGobbler(voidCall):82 - transform_postgres
�[32mnormalization�[0m - 2021-11-22 09:35:46 INFO () LineGobbler(voidCall):82 - Running: transform-catalog --integration-type postgres --profile-config-dir /data/44/0/normalize --catalog destination_catalog.json --out /data/44/0/normalize/models/generated/ --json-column _airbyte_data
�[32mnormalization�[0m - 2021-11-22 09:35:47 INFO () LineGobbler(voidCall):82 - Processing destination_catalog.json...
�[32mnormalization�[0m - 2021-11-22 09:35:47 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/another_source_schema/another_source_table_ab1.sql from ANOTHER_SOURCE_TABLE
�[32mnormalization�[0m - 2021-11-22 09:35:47 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/another_source_schema/another_source_table_ab2.sql from ANOTHER_SOURCE_TABLE
�[32mnormalization�[0m - 2021-11-22 09:35:47 INFO () LineGobbler(voidCall):82 -   Generating airbyte_ctes/another_source_schema/another_source_table_ab3.sql from ANOTHER_SOURCE_TABLE
�[32mnormalization�[0m - 2021-11-22 09:35:47 INFO () LineGobbler(voidCall):82 -   Generating airbyte_incremental/another_source_schema/another_source_table.sql from ANOTHER_SOURCE_TABLE
�[32mnormalization�[0m - 2021-11-22 09:35:47 INFO () LineGobbler(voidCall):82 - detected no config file for ssh, assuming ssh is off.
�[32mnormalization�[0m - 2021-11-22 09:35:49 INFO () LineGobbler(voidCall):82 - Running with dbt=0.21.0
�[32mnormalization�[0m - 2021-11-22 09:35:49 INFO () LineGobbler(voidCall):82 - Unable to do partial parsing because ../build/partial_parse.msgpack not found
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - [�[33mWARNING�[0m]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - There are 2 unused configuration paths:
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - - models.airbyte_utils.generated.airbyte_tables
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - - models.airbyte_utils.generated.airbyte_views
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - 
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - Found 4 models, 0 tests, 0 snapshots, 0 analyses, 478 macros, 0 operations, 0 seed files, 1 source, 0 exposures
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - 
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - 09:35:51 | Concurrency: 32 threads (target='prod')
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - 09:35:51 | 
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - 09:35:51 | 1 of 1 START incremental model another_source_schema.another_source_table.................................... [RUN]
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - 09:35:51 | 1 of 1 OK created incremental model another_source_schema.another_source_table............................... [�[32mINSERT 0 0�[0m in 0.23s]
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - 09:35:51 | 
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - 09:35:51 | Finished running 1 incremental model in 0.59s.
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - 
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - �[32mCompleted successfully�[0m
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - 
�[32mnormalization�[0m - 2021-11-22 09:35:51 INFO () LineGobbler(voidCall):82 - Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
2021-11-22 09:35:52 INFO () DefaultNormalizationWorker(run):69 - Normalization executed in 6 seconds.
2021-11-22 09:35:52 INFO () TemporalAttemptExecution(get):137 - Stopping cancellation check scheduling...

Steps to Reproduce

  1. Start Airbyte
  2. Set DB2 as the source
  3. Set any destination
  4. Proceed to Setup new connection, verify schemas and tables where the connected user doesn't have privileges appear on Airbyte UI
@alexandertsukanov alexandertsukanov added the type/bug Something isn't working label Nov 22, 2021
@alexandertsukanov alexandertsukanov self-assigned this Nov 22, 2021
@alexandertsukanov alexandertsukanov mentioned this issue Nov 22, 2021
Closed
@sashaNeshcheret sashaNeshcheret self-assigned this Jan 20, 2022
@sashaNeshcheret sashaNeshcheret linked a pull request Jan 31, 2022 that will close this issue
Db2 source: discover only permitted tables #9875
Merged
40 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sashaNeshcheret sashaNeshcheret

Labels
area/connectors Connector related issues area/databases connectors/source/db2 connectors/sources-database type/bug Something isn't working
Projects
GL Roadmap
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Db2 source: discover only permitted tables airbytehq/airbyte
4 participants
@bleonard @karinakuz @alexandertsukanov @sashaNeshcheret