Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Source Smartsheets: resolve #8099 and add support for syncing row metadata #11003

Closed
wants to merge 40 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
f736b6f
filter out _sql files in stream
zzstoatzz Aug 4, 2021
2733185
filter out _sql files in stream
zzstoatzz Aug 4, 2021
b76aa99
filter out empty files
zzstoatzz Aug 4, 2021
4969feb
loosen handling
zzstoatzz Aug 4, 2021
bb1d28f
Merge branch 'airbytehq:master' into master
zzstoatzz Aug 9, 2021
5290757
generate scaffolding, implement spec, start check
zzstoatzz Aug 12, 2021
c8ffbaa
remove soap sub-module
zzstoatzz Aug 12, 2021
487f48a
fix column, value zip sorting
zzstoatzz Aug 26, 2021
b34641f
Merge branch 'airbytehq:master' into master
zzstoatzz Aug 26, 2021
723dd2f
rm assert that depended on specific config json schema
zzstoatzz Aug 26, 2021
df29efd
Merge branch 'master' of https://github.com/zzstoatzz/airbyte
zzstoatzz Aug 26, 2021
2ab3b39
rm old scaffolding
zzstoatzz Aug 26, 2021
f629387
Merge branch 'airbytehq:master' into master
zzstoatzz Sep 6, 2021
7dffd62
Merge branch 'airbytehq:master' into master
zzstoatzz Sep 6, 2021
e499557
resolve merge conflicts
zzstoatzz Mar 9, 2022
f7e961e
Merge pull request #2 from zzstoatzz/airbytehq-master
zzstoatzz Mar 9, 2022
fbf45a3
init include metadata
zzstoatzz Mar 9, 2022
14a6a88
Merge branch 'master' of github.com:zzstoatzz/airbyte
zzstoatzz Mar 9, 2022
c8f128e
default string
zzstoatzz Mar 9, 2022
0046cc3
Merge branch 'airbytehq:master' into master
zzstoatzz Mar 9, 2022
7f21b27
add comment and better title for metadata setup button
zzstoatzz Mar 9, 2022
d15914e
rm unused variable
zzstoatzz Mar 9, 2022
90f35b8
Merge branch 'master' of github.com:zzstoatzz/airbyte
zzstoatzz Mar 9, 2022
8f6121c
rm old fork changes and unused imports
zzstoatzz Mar 9, 2022
c9721d2
rm sample file
zzstoatzz Mar 9, 2022
49c1115
default None for metadata fields if no value for given row
zzstoatzz Mar 10, 2022
ecfd868
add id
zzstoatzz Mar 10, 2022
2c1de53
Merge branch 'airbytehq:master' into master
zzstoatzz Mar 14, 2022
d4bdbdd
satisfy flake
zzstoatzz Mar 14, 2022
2054b4b
documentation and cleaning up source.read()
zzstoatzz Mar 14, 2022
fb81faa
better name for iterator in row dict comprehension
zzstoatzz Mar 14, 2022
fa614cf
try markdown link
zzstoatzz Mar 14, 2022
4dc46ca
add file ext to image in readme
zzstoatzz Mar 14, 2022
133d02d
update user prompt in spec
zzstoatzz Mar 14, 2022
54a9910
Merge branch 'airbytehq:master' into master
zzstoatzz Mar 14, 2022
3a7f1f1
add bootstrap
zzstoatzz Mar 14, 2022
468d027
Merge branch 'airbytehq:master' into master
zzstoatzz Mar 17, 2022
11798fd
Merge branch 'airbytehq:master' into master
zzstoatzz Mar 23, 2022
a65d3ca
Merge branch 'airbytehq:master' into master
zzstoatzz Mar 29, 2022
b38bf67
cloned changes
zzstoatzz Mar 29, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4,018 changes: 4,018 additions & 0 deletions airbyte-api/bin/main/config.yaml

Large diffs are not rendered by default.

Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
206 changes: 206 additions & 0 deletions airbyte-commons/bin/main/log4j2.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Properties>

<!-- Mask the string apikey=<string> to apikey=***** to prevent secrets leaking. -->
<Property name="default-pattern">%d{yyyy-MM-dd HH:mm:ss}{GMT+0} %highlight{%p} %C{1.}(%M):%L - %replace{%m}{apikey=[\w\-]*}{apikey=*****}%n</Property>
<!--Logs the timestamp and log_source/application name in the beginning of the line if it exists with a > separator, and then always the rest of the line.-->
<Property name="simple-pattern">%d{yyyy-MM-dd HH:mm:ss}{GMT+0}%replace{ %X{log_source}}{^ -}{} > %replace{%m}{apikey=[\w\-]*}{apikey=*****}%n</Property>

<!-- Always log INFO by default. -->
<Property name="log-level">${sys:LOG_LEVEL:-${env:LOG_LEVEL:-INFO}}</Property>

<Property name="s3-bucket">${sys:S3_LOG_BUCKET:-${env:S3_LOG_BUCKET}}</Property>
<Property name="s3-region">${sys:S3_LOG_BUCKET_REGION:-${env:S3_LOG_BUCKET_REGION}}</Property>
<Property name="s3-aws-key">${sys:AWS_ACCESS_KEY_ID:-${env:AWS_ACCESS_KEY_ID}}</Property>
<Property name="s3-aws-secret">${sys:AWS_SECRET_ACCESS_KEY:-${env:AWS_SECRET_ACCESS_KEY}}</Property>
<Property name="s3-minio-endpoint">${sys:S3_MINIO_ENDPOINT:-${env:S3_MINIO_ENDPOINT}}</Property>
<Property name="s3-path-style-access">${sys:S3_PATH_STYLE_ACCESS:-${env:S3_PATH_STYLE_ACCESS}}</Property>

<Property name="gcs-log-bucket">${sys:GCS_LOG_BUCKET:-${env:GCS_LOG_BUCKET}}</Property>
</Properties>

<Appenders>
<Console name="Default" target="SYSTEM_OUT">
<ContextMapFilter onMatch="DENY" onMismatch="ACCEPT">
<KeyValuePair key="simple" value="true"/>
</ContextMapFilter>
<PatternLayout pattern="${default-pattern}"/>
</Console>

<Console name="SimpleDefault" target="SYSTEM_OUT">
<ContextMapFilter onMatch="ACCEPT" onMismatch="DENY">
<KeyValuePair key="simple" value="true"/>
</ContextMapFilter>
<PatternLayout pattern="${simple-pattern}"/>
</Console>

<Routing name="LogSplit">
<ContextMapFilter onMatch="DENY" onMismatch="ACCEPT">
<KeyValuePair key="simple" value="true"/>
</ContextMapFilter>
<Routes pattern="$${ctx:job_log_path}">
<!-- Don't split logs if job_root isn't defined -->
<Route key="$${ctx:job_log_path}">
<Null name="/dev/null"/>
</Route>
<Route>
<File name="${ctx:job_log_path}-local" fileName="${ctx:job_log_path}">
<PatternLayout pattern="${default-pattern}"/>
</File>
</Route>
</Routes>
<IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
</Routing>

<Routing name="SimpleLogSplit">
<ContextMapFilter onMatch="ACCEPT" onMismatch="DENY">
<KeyValuePair key="simple" value="true"/>
</ContextMapFilter>
<Routes pattern="$${ctx:job_log_path}">
<!-- Don't split logs if job_root isn't defined -->
<Route key="$${ctx:job_log_path}">
<Null name="/dev/null"/>
</Route>
<Route>
<File name="${ctx:job_log_path}-local" fileName="${ctx:job_log_path}">
<PatternLayout pattern="${simple-pattern}"/>
</File>
</Route>
</Routes>
<IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
</Routing>

<!--
Separate routers are created for each cloud logger as
1) a Route only accepts 1 appender
2) Routes don't support routing log output to more than Route
-->
<Routing name="LogSplitCloud">
<ContextMapFilter onMatch="DENY" onMismatch="ACCEPT">
<KeyValuePair key="simple" value="true"/>
</ContextMapFilter>
<Routes pattern="$${ctx:cloud_job_log_path}">
<!-- Don't split logs if job_root isn't defined -->
<Route key="$${ctx:cloud_job_log_path}">
<Null name="/dev/null"/>
</Route>
<Route>
<Log4j2Appender name="${ctx:cloud_job_log_path}"
verbose="true"
stagingBufferAge="1"
s3Bucket="${s3-bucket}" s3Path="job-logging${ctx:cloud_job_log_path}" s3Region="${s3-region}"
s3AwsKey="${s3-aws-key}" s3AwsSecret="${s3-aws-secret}"
s3ServiceEndpoint="${s3-minio-endpoint}" s3PathStyleAccess="${s3-path-style-access}"
gcpStorageBucket="${gcs-log-bucket}" gcpStorageBlobNamePrefix="job-logging${ctx:cloud_job_log_path}">
<PatternLayout pattern="${default-pattern}"/>
</Log4j2Appender>
</Route>
</Routes>
<IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
</Routing>

<Routing name="SimpleLogSplitCloud">
<ContextMapFilter onMatch="ACCEPT" onMismatch="DENY">
<KeyValuePair key="simple" value="true"/>
</ContextMapFilter>
<Routes pattern="$${ctx:cloud_job_log_path}">
<!-- Don't split logs if job_root isn't defined -->
<Route key="$${ctx:cloud_job_log_path}">
<Null name="/dev/null"/>
</Route>
<Route>
<Log4j2Appender name="${ctx:cloud_job_log_path}"
verbose="true"
stagingBufferAge="1"
s3Bucket="${s3-bucket}" s3Path="job-logging${ctx:cloud_job_log_path}" s3Region="${s3-region}"
s3AwsKey="${s3-aws-key}" s3AwsSecret="${s3-aws-secret}"
s3ServiceEndpoint="${s3-minio-endpoint}" s3PathStyleAccess="${s3-path-style-access}"
gcpStorageBucket="${gcs-log-bucket}" gcpStorageBlobNamePrefix="job-logging${ctx:cloud_job_log_path}">
<PatternLayout pattern="${simple-pattern}"/>
</Log4j2Appender>
</Route>
</Routes>
<IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
</Routing>

<Routing name="AppLogSplit">
<Routes pattern="$${ctx:workspace_app_root}">
<!-- Don't split logs if workspace_app_log_root isn't defined -->
<Route key="$${ctx:workspace_app_root}">
<Null name="/dev/null"/>
</Route>
<Route>
<RollingFile
name="${ctx:workspace_app_root}-local"
fileName="${ctx:workspace_app_root}/logs.log"
filePattern="${ctx:workspace_app_root}/logs.%i.log.gz"
ignoreExceptions="false">
<PatternLayout pattern="${default-pattern}"/>
<Policies>
<SizeBasedTriggeringPolicy size="100MB" />
</Policies>
<DefaultRolloverStrategy max="3" />
</RollingFile>
</Route>
</Routes>
<IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
</Routing>
<Routing name="AppLogSplitCloud">
<Routes pattern="$${ctx:cloud_workspace_app_root}">
<!-- Don't split logs if workspace_app_log_root isn't defined -->
<Route key="$${ctx:cloud_workspace_app_root}">
<Null name="/dev/null"/>
</Route>
<Route>
<Log4j2Appender name="app-logging/${ctx:cloud_workspace_app_root}/"
stagingBufferAge="1"
s3Bucket="${s3-bucket}" s3Path="app-logging${ctx:cloud_workspace_app_root}" s3Region="${s3-region}"
s3AwsKey="${s3-aws-key}" s3AwsSecret="${s3-aws-secret}"
s3ServiceEndpoint="${s3-minio-endpoint}" s3PathStyleAccess="${s3-path-style-access}"
gcpStorageBucket="${gcs-log-bucket}" gcpStorageBlobNamePrefix="app-logging${ctx:cloud_workspace_app_root}">
<PatternLayout pattern="${default-pattern}"/>
</Log4j2Appender>
</Route>
</Routes>
<IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
</Routing>
</Appenders>

<Loggers>
<Root level="${log-level}">
<AppenderRef ref="Default"/>
<AppenderRef ref="SimpleDefault"/>
<AppenderRef ref="LogSplit"/>
<AppenderRef ref="SimpleLogSplit"/>
<AppenderRef ref="LogSplitCloud"/>
<AppenderRef ref="SimpleLogSplitCloud"/>
<AppenderRef ref="AppLogSplit"/>
<AppenderRef ref="AppLogSplitCloud"/>
</Root>

<Logger name="org.eclipse.jetty" level="INFO" />
<Logger name="com.github.dockerjava" level="INFO" />
<Logger name="org.apache.hc" level="INFO" />
<Logger name="org.jooq" level="INFO" />
<logger name="org.jooq.Constants" level="OFF" />
<Logger name="com.networknt.schema" level="INFO" />
<Logger name="me.andrz.jackson" level="INFO" />
<Logger name="com.leansoft.bigqueue" level="INFO" />
<Logger name="io.netty" level="INFO" />
<Logger name="io.grpc" level="INFO" />
<Logger name="io.temporal" level="INFO" />
<Logger name="org.apache" level="WARN" />
<Logger name="httpclient" level="WARN" />
<Logger name="com.amazonaws" level="WARN" />
<!--MySQL Debezium connector generates a log whenever it converts an invalid value to empty value.
Ex: Invalid value '0000-00-00 00:00:00' stored in column 'column_name' of table 'table_name' converted to empty value
If a database has tons of such values, the logs would be filled with such messages-->
<Logger name="io.debezium.connector.mysql.MySqlValueConverters" level="OFF" />
<!--MySQL Debezium connector generates a log whenever it comes across a DDL query to mention that it skipped it.
If a database has tons of DDL queries, the logs would be filled with such messages-->
<Logger name="io.debezium.relational.history" level="OFF" />

</Loggers>

</Configuration>
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
mock-maker-inline
1 change: 1 addition & 0 deletions airbyte-commons/bin/test/resource_test
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
content1
1 change: 1 addition & 0 deletions airbyte-commons/bin/test/resource_test_a
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
content1
1 change: 1 addition & 0 deletions airbyte-commons/bin/test/subdir/resource_test_a
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
content2
1 change: 1 addition & 0 deletions airbyte-commons/bin/test/subdir/resource_test_sub
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
content2
1 change: 1 addition & 0 deletions airbyte-commons/bin/test/subdir/resource_test_sub_2
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
content3
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
20 changes: 20 additions & 0 deletions airbyte-config/models/bin/main/types/ActorCatalog.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/AttemptFailureSummary.yaml
title: ActorCatalog
description: Catalog of an actor.
type: object
additionalProperties: false
required:
- id
- catalog
- catalogHash
properties:
id:
type: string
format: uuid
catalog:
type: object
existingJavaType: com.fasterxml.jackson.databind.JsonNode
catalogHash:
type: string
27 changes: 27 additions & 0 deletions airbyte-config/models/bin/main/types/ActorCatalogFetchEvent.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/AttemptFailureSummary.yaml
title: ActorCatalogFetchEvent
description: Link actor to their actual catalog
type: object
additionalProperties: false
required:
- id
- actorCatalogId
- actorId
- configHash
- connectorVersion
properties:
id:
type: string
format: uuid
actorId:
type: string
format: uuid
actorCatalogId:
type: string
format: uuid
configHash:
type: string
connectorVersion:
type: string
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/ActorDefinitionResourceRequirements.yaml
title: ActorDefinitionResourceRequirements
description: actor definition specific resource requirements
type: object
# set to false because we need the validations on seeds to be strict. otherwise, we will just add whatever is in the seed file into the db.
additionalProperties: false
properties:
default:
description: if set, these are the requirements that should be set for ALL jobs run for this actor definition.
"$ref": ResourceRequirements.yaml
jobSpecific:
type: array
items:
"$ref": "#/definitions/JobTypeResourceLimit"
definitions:
JobTypeResourceLimit:
description: sets resource requirements for a specific job type for an actor definition. these values override the default, if both are set.
type: object
# set to false because we need the validations on seeds to be strict. otherwise, we will just add whatever is in the seed file into the db.
additionalProperties: false
required:
- jobType
- resourceRequirements
properties:
jobType:
"$ref": JobType.yaml
resourceRequirements:
"$ref": ResourceRequirements.yaml
9 changes: 9 additions & 0 deletions airbyte-config/models/bin/main/types/ActorType.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/ActorType.yaml
title: ActorType
description: enum that describes different types of actors
type: string
enum:
- source
- destination
18 changes: 18 additions & 0 deletions airbyte-config/models/bin/main/types/AttemptFailureSummary.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/AttemptFailureSummary.yaml
title: AttemptFailureSummary
description: Attempt-level summarization of failures that occurred during a sync workflow.
type: object
additionalProperties: false
required:
- failures
properties:
failures:
description: Ordered list of failures that occurred during the attempt.
type: array
items:
"$ref": FailureReason.yaml
partialSuccess:
description: True if the number of committed records for this attempt was greater than 0. False if 0 records were committed. Blank if number of committed records is unknown.
type: boolean
12 changes: 12 additions & 0 deletions airbyte-config/models/bin/main/types/DataType.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/DataType.yaml
title: DataType
description: standard data types.
type: string
enum:
- string
- number
- boolean
- object
- array
35 changes: 35 additions & 0 deletions airbyte-config/models/bin/main/types/DestinationConnection.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/DestinationConnection.yaml
title: DestinationConnection
description: information required for connection to a destination.
type: object
required:
- name
- destinationDefinitionId
- workspaceId
- destinationId
- configuration
- tombstone
additionalProperties: false
properties:
name:
type: string
destinationDefinitionId:
type: string
format: uuid
workspaceId:
type: string
format: uuid
destinationId:
type: string
format: uuid
configuration:
description: Integration specific blob. Must be a valid JSON string.
type: object
existingJavaType: com.fasterxml.jackson.databind.JsonNode
tombstone:
description:
if not set or false, the configuration is active. if true, then this
configuration is permanently off.
type: boolean
Loading