Skip to content

Commit

Permalink
Adds the bottleneck datasources and respective catalogs and updated d…
Browse files Browse the repository at this point in the history
…escription for the harness (#26503)

* Adds the bottleneck datasources and respective catalogs and updated description for the harness

* Automated Commit - Formatting Changes

---------

Co-authored-by: ryankfu <ryankfu@users.noreply.github.com>
  • Loading branch information
ryankfu and ryankfu committed May 24, 2023
1 parent 75b051d commit c87ffb3
Show file tree
Hide file tree
Showing 5 changed files with 80 additions and 3 deletions.
7 changes: 4 additions & 3 deletions .github/workflows/connector-performance-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,10 @@ jobs:
with:
comment-id: ${{ github.event.inputs.comment-id }}
body: |
#### Note: The following `dataset=` values are supported: `1m`<sub>(default)</sub>, `10m`, `20m`, `bottleneck_stream1`.
For destinations only: you can also use `stream-numbers=N` to simulate N number of parallel streams. Additionally,
`sync-mode=incremental` is supported for destinations. For example: `dataset=1m stream-numbers=2 sync-mode=incremental`
#### Note: The following `dataset=` values are supported: `1m`<sub>(default)</sub>, `10m`, `20m`,
`bottleneck_stream1`, `bottleneck_stream_randomseed. For destinations only: you can also use `stream-numbers=N`
to simulate N number of parallel streams. Additionally, `sync-mode=incremental` is supported for destinations.
For example: `dataset=1m stream-numbers=2 sync-mode=incremental`
> :runner: ${{github.event.inputs.connector}} https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}.
- name: Search for valid connector name format
id: regex
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
{
"streams": [
{
"stream": {
"name": "bottleneck_stream1",
"namespace": "PERF_TEST_HARNESS",
"json_schema": {
"type": "object",
"properties": {
"field1": {
"type": "string"
},
"field2": {
"type": "string"
},
"field3": {
"type": "string"
},
"field4": {
"type": "string"
},
"field5": {
"type": "string"
}
}
},
"default_cursor_field": [],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_primary_key": [["field1"]]
},
"sync_mode": "full_refresh",
"primary_key": [["field1"]],
"cursor_field": ["field1"],
"destination_sync_mode": "append"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
https://storage.googleapis.com/airbyte-performance-testing-public/sample-data/bottleneck_detector_data_sets/bottleneck_stream1.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
{
"streams": [
{
"stream": {
"name": "bottleneck_stream_randomseed",
"namespace": "PERF_TEST_HARNESS",
"json_schema": {
"type": "object",
"properties": {
"field1": {
"type": "string"
},
"field2": {
"type": "string"
},
"field3": {
"type": "string"
},
"field4": {
"type": "string"
},
"field5": {
"type": "string"
}
}
},
"default_cursor_field": [],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_primary_key": [["field1"]]
},
"sync_mode": "full_refresh",
"primary_key": [["field1"]],
"cursor_field": ["field1"],
"destination_sync_mode": "append"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
https://storage.googleapis.com/airbyte-performance-testing-public/sample-data/bottleneck_detector_data_sets/bottleneck_stream_randomseed.csv

0 comments on commit c87ffb3

Please sign in to comment.