Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Adding CDC sample #2263

Merged
merged 103 commits into from
Nov 28, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
c9db728
feat: Adding CDC Sample
farhan0102 Oct 4, 2023
df3fc13
fix: minor tweak
farhan0102 Nov 1, 2023
22e37ff
fix: minor tweak
farhan0102 Nov 1, 2023
effcd59
fix: minor style fix
farhan0102 Nov 1, 2023
9e5a6aa
fix: testing json type
farhan0102 Nov 1, 2023
7772351
fix: minor tweak
farhan0102 Nov 2, 2023
bb9ad7c
fix: Changing to json format
farhan0102 Nov 2, 2023
5541530
fix: field names
farhan0102 Nov 2, 2023
d8394e2
fix: testing
farhan0102 Nov 3, 2023
2552ff5
fix: added json type in bq to bq schema converter
farhan0102 Nov 3, 2023
5a479d6
fix: filepath
farhan0102 Nov 3, 2023
1abf42c
fix: filepath
farhan0102 Nov 3, 2023
578d3b9
fix: checkstyle
farhan0102 Nov 3, 2023
f2806e6
fix: filepath
farhan0102 Nov 3, 2023
868bb28
fix: filepath
farhan0102 Nov 3, 2023
931f4bd
fix: filepath
farhan0102 Nov 3, 2023
dc0c554
fix: checkstyle
farhan0102 Nov 3, 2023
bf5def7
fix: testing
farhan0102 Nov 3, 2023
0912819
fix: add brace
farhan0102 Nov 3, 2023
96c4064
fix: filepath
farhan0102 Nov 3, 2023
824b276
fix: filepath
farhan0102 Nov 3, 2023
1cd97fc
fix: json format
farhan0102 Nov 3, 2023
17a9c1c
fix: testing commas
farhan0102 Nov 6, 2023
b99a7a1
fix: remove white space
farhan0102 Nov 6, 2023
000716e
fix: trying out escaping in json
farhan0102 Nov 6, 2023
719899b
fix: added quotes around json dictionary
farhan0102 Nov 6, 2023
980fa10
fix: make json array
farhan0102 Nov 6, 2023
efb2a0e
fix: added braces
farhan0102 Nov 6, 2023
407d0d5
fix: removed braces
farhan0102 Nov 6, 2023
fe2ccca
fix: minor tweak
farhan0102 Nov 6, 2023
fa1db3b
fix: tweak to jsonarray
farhan0102 Nov 7, 2023
0eb1d7a
fix: tweak to jsonarray
farhan0102 Nov 7, 2023
a85aff7
fix: testing
farhan0102 Nov 7, 2023
2986c68
fix: testing
farhan0102 Nov 7, 2023
73cd65e
fix: trying exact json
farhan0102 Nov 7, 2023
b8e17dc
fix: testing
farhan0102 Nov 7, 2023
271df01
fix: removing whitespaces
farhan0102 Nov 7, 2023
2d6dec2
fix: testing with no square brackets
farhan0102 Nov 7, 2023
11e7984
fix: tweaking the json format
farhan0102 Nov 7, 2023
7b966cd
fix: trying without json field
farhan0102 Nov 7, 2023
00ae3d9
feat: Add in-stream retry logic for retryable and quota errors (#2243)
egreco12 Oct 6, 2023
2f31416
build: do not declare javadoc plugin version (#2268)
alicejli Oct 10, 2023
7cf0e5e
chore(deps): update dependency com.google.cloud:google-cloud-bigquery…
renovate-bot Oct 10, 2023
d057127
chore(deps): update dependency com.google.cloud:libraries-bom to v26.…
renovate-bot Oct 10, 2023
95b75fe
deps: update dependency com.google.cloud:google-cloud-bigquery to v2.…
renovate-bot Oct 10, 2023
91b0c43
deps: bumping google-cloud-shared-config to v1.5.8 (#2269)
ddixit14 Oct 10, 2023
3c45f48
deps: update dependency com.google.cloud:google-cloud-shared-dependen…
renovate-bot Oct 10, 2023
3150f60
chore(main): release 2.43.1-SNAPSHOT (#2256)
release-please[bot] Oct 10, 2023
66e7b6a
chore(main): release 2.44.0 (#2270)
release-please[bot] Oct 11, 2023
e23e301
build(deps): update dependency com.google.cloud:google-cloud-shared-c…
renovate-bot Oct 17, 2023
ca13e54
deps: update dependency org.json:json to v20231013 [security] (#2278)
renovate-bot Oct 17, 2023
8b1f356
deps: update dependency org.json:json to v20231013 - abandoned (#2276)
renovate-bot Oct 17, 2023
ba075b6
deps: update dependency com.google.cloud:google-cloud-bigquery to v2.…
renovate-bot Oct 17, 2023
10036ad
chore(deps): update dependency com.google.cloud:google-cloud-bigquery…
renovate-bot Oct 17, 2023
24dcf16
chore(deps): update dependency com.google.cloud:libraries-bom to v26.…
renovate-bot Oct 17, 2023
4d0bbe2
deps: update actions/checkout digest to b4ffde6 (#2281)
renovate-bot Oct 19, 2023
e63c4d1
chore: track customization in templates (#1876) (#2279)
gcf-owl-bot[bot] Oct 25, 2023
a273e60
chore(main): release 2.44.1-SNAPSHOT (#2271)
release-please[bot] Oct 25, 2023
70397ec
deps: update dependency com.google.cloud:google-cloud-shared-dependen…
renovate-bot Oct 25, 2023
1def89d
test: Add e2e test for default value use cases for default stream and…
GaoleMeng Oct 25, 2023
246cb29
chore(main): release 2.44.1 (#2286)
release-please[bot] Oct 26, 2023
8fe613f
chore(deps): update dependency com.google.cloud:google-cloud-bigquery…
renovate-bot Oct 26, 2023
6bee258
deps: update dependency com.google.cloud:google-cloud-bigquery to v2.…
renovate-bot Oct 26, 2023
53be322
chore(deps): update dependency com.google.cloud:libraries-bom to v26.…
renovate-bot Oct 27, 2023
c77ef23
feat: add support for proto3 optional tag (#2295)
agrawal-siddharth Oct 31, 2023
b5fb8f0
test: add e2e test for default value use case for stream writer (#2288)
GaoleMeng Oct 31, 2023
7a894cb
test: add e2e test to verify proto3 optional tag works (#2298)
agrawal-siddharth Oct 31, 2023
84dacd7
fix(bigquery): Make exponential backoff retry second based (#2212)
mina-asham Nov 1, 2023
3b8d302
fix: add native image configurations for com.google.rpc classes (#2305)
mpeddada1 Nov 6, 2023
2fa8443
chore: setup 2.38.x lts branch (#2208)
burkedavison Nov 6, 2023
ef933b7
deps: update dependency com.google.cloud:google-cloud-shared-dependen…
renovate-bot Nov 6, 2023
ed1289c
chore: update graalvm version to 22.3.3 in kokoro cfg files. (#1889) …
gcf-owl-bot[bot] Nov 6, 2023
4871fad
build(deps): update dependency com.google.cloud:google-cloud-shared-c…
renovate-bot Nov 6, 2023
4bb1112
chore(main): release 2.44.2-SNAPSHOT (#2289)
release-please[bot] Nov 6, 2023
5aa3cf2
deps: update dependency com.google.cloud:google-cloud-bigquery to v2.…
renovate-bot Nov 7, 2023
a4455e4
chore(main): release 2.45.0 (#2306)
release-please[bot] Nov 7, 2023
1388610
fix: add comment/documentation for setRetrySettings (#2309)
yirutang Nov 7, 2023
a00e8f3
feat: Add integration tests with RetrySettings enabled. (#2275)
egreco12 Nov 13, 2023
4259507
feat: enable 20MB request limit, this feature is allowlist only. (#2311)
yirutang Nov 14, 2023
e86dbbe
fix: Create next attempt after first attempt to initialize exponentia…
egreco12 Nov 14, 2023
1923894
feat: add new configuration for nightly retry-related tests. (#2319)
egreco12 Nov 14, 2023
8a9d39b
deps: update actions/github-script action to v7 (#2317)
renovate-bot Nov 15, 2023
af48969
chore(deps): update dependency com.google.cloud:libraries-bom to v26.…
renovate-bot Nov 15, 2023
afb13f3
fix: enable request limit e2e test. (#2320)
yirutang Nov 15, 2023
47b20c3
chore(main): release 2.45.1-SNAPSHOT (#2310)
release-please[bot] Nov 15, 2023
30e19b4
chore(main): release 2.46.0 (#2322)
release-please[bot] Nov 15, 2023
8b99088
chore: fix retry-quota jobs (#2324)
Neenu1995 Nov 16, 2023
26e6aef
chore: fix retry-quota jobs (#2325)
Neenu1995 Nov 16, 2023
766722d
chore: fix graalvm kokoro builds (#2327)
Neenu1995 Nov 21, 2023
2f1b892
Update CDC example to use work with UPSERT
PhongChuong Nov 23, 2023
71f4118
fix import order + reading json file
PhongChuong Nov 23, 2023
8cbea70
fix lint errors
PhongChuong Nov 23, 2023
86282e5
fix more lint errors due to import order
PhongChuong Nov 23, 2023
6556c7a
fix even more lint errors due to import order. Sad.
PhongChuong Nov 23, 2023
8637985
fix json path issue
PhongChuong Nov 24, 2023
fb23c25
fix format
PhongChuong Nov 24, 2023
18c705a
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Nov 24, 2023
08af044
Add modified customer data to the sample
PhongChuong Nov 24, 2023
a5ab8d9
Fix formating.
PhongChuong Nov 24, 2023
5fb5166
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Nov 24, 2023
bf21b3c
Fix lint errors.
PhongChuong Nov 24, 2023
2415a28
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Nov 24, 2023
31f799b
Fix modifiedCustomersDataFile reading from the wrong input arg.
PhongChuong Nov 28, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/.OwlBot.lock.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@
# limitations under the License.
docker:
image: gcr.io/cloud-devrel-public-resources/owlbot-java:latest
digest: sha256:88ba8dcc5c2c7792e1c3511381f4ab329002a1c42c512f66ca87ced572dfbf9f
# created: 2023-09-05T18:54:42.225408832Z
digest: sha256:bc2bf47c7370f1b1a8a46b0c104ce7e43644ac58902c9de265fe1f253fcc2506
# created: 2023-10-31T13:44:21.909247346Z
4 changes: 4 additions & 0 deletions .github/release-please.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,7 @@ branches:
handleGHRelease: true
releaseType: java-backport
branch: 2.25.x
- bumpMinorPreMajor: true
handleGHRelease: true
releaseType: java-backport
branch: 2.38.x
14 changes: 14 additions & 0 deletions .github/sync-repo-settings.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,20 @@ branchProtectionRules:
- 'Kokoro - Test: Integration'
- cla/google
- OwlBot Post Processor
- pattern: 2.38.x
isAdminEnforced: true
requiredApprovingReviewCount: 1
requiresCodeOwnerReviews: true
requiresStrictStatusChecks: false
requiredStatusCheckContexts:
- dependencies (17)
- lint
- clirr
- units (8)
- units (11)
- 'Kokoro - Test: Integration'
- cla/google
- OwlBot Post Processor
permissionRules:
- team: yoshi-admins
permission: admin
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/approve-readme.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
runs-on: ubuntu-latest
if: github.repository_owner == 'googleapis' && github.head_ref == 'autosynth-readme'
steps:
- uses: actions/github-script@v6
- uses: actions/github-script@v7
with:
github-token: ${{secrets.YOSHI_APPROVER_TOKEN}}
script: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/auto-release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
runs-on: ubuntu-latest
if: contains(github.head_ref, 'release-please')
steps:
- uses: actions/github-script@v6
- uses: actions/github-script@v7
with:
github-token: ${{secrets.YOSHI_APPROVER_TOKEN}}
debug: true
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
matrix:
java: [11, 17]
steps:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
- uses: actions/setup-java@v2
with:
distribution: temurin
Expand All @@ -28,7 +28,7 @@ jobs:
name: "units (8)"
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
- uses: actions/setup-java@v3
with:
java-version: 8
Expand All @@ -48,7 +48,7 @@ jobs:
windows:
runs-on: windows-latest
steps:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
- uses: actions/setup-java@v2
with:
distribution: temurin
Expand All @@ -63,7 +63,7 @@ jobs:
matrix:
java: [17]
steps:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
- uses: actions/setup-java@v2
with:
distribution: temurin
Expand All @@ -73,7 +73,7 @@ jobs:
javadoc:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
- uses: actions/setup-java@v3
with:
distribution: temurin
Expand All @@ -85,7 +85,7 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
- uses: actions/setup-java@v2
with:
distribution: temurin
Expand All @@ -97,7 +97,7 @@ jobs:
clirr:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
- uses: actions/setup-java@v2
with:
distribution: temurin
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/samples.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jobs:
checkstyle:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
- uses: actions/setup-java@v1
with:
java-version: 8
Expand Down
31 changes: 29 additions & 2 deletions .kokoro/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -65,18 +65,29 @@ integration)
-DtrimStackTrace=false \
-Dclirr.skip=true \
-Denforcer.skip=true \
-Dit.test=!ITBigQueryWrite*RetryTest \
-Dsurefire.failIfNoSpecifiedTests=false \
-Dfailsafe.failIfNoSpecifiedTests=false \
-fae \
verify
RETURN_CODE=$?
;;
graalvm)
# Run Unit and Integration Tests with Native Image
mvn -B ${INTEGRATION_TEST_ARGS} -ntp -Pnative -Penable-integration-tests test
mvn -B ${INTEGRATION_TEST_ARGS} -ntp -Pnative -Penable-integration-tests \
-Dit.test=!ITBigQueryWrite*RetryTest \
-Dsurefire.failIfNoSpecifiedTests=false \
-Dfailsafe.failIfNoSpecifiedTests=false \
test
RETURN_CODE=$?
;;
graalvm17)
# Run Unit and Integration Tests with Native Image
mvn -B ${INTEGRATION_TEST_ARGS} -ntp -Pnative -Penable-integration-tests test
mvn -B ${INTEGRATION_TEST_ARGS} -ntp -Pnative -Penable-integration-tests \
-Dit.test=!ITBigQueryWrite*RetryTest \
-Dsurefire.failIfNoSpecifiedTests=false \
-Dfailsafe.failIfNoSpecifiedTests=false \
test
RETURN_CODE=$?
;;
samples)
Expand Down Expand Up @@ -112,6 +123,22 @@ clirr)
mvn -B -Denforcer.skip=true clirr:check
RETURN_CODE=$?
;;
retry_quota)
mvn -B ${INTEGRATION_TEST_ARGS} \
-Dit.test=ITBigQueryWriteQuotaRetryTest \
-Dsurefire.failIfNoSpecifiedTests=false \
-Dfailsafe.failIfNoSpecifiedTests=false \
test
RETURN_CODE=$?
;;
retry_non_quota)
mvn -B ${INTEGRATION_TEST_ARGS} \
-Dit.test=ITBigQueryWriteNonQuotaRetryTest \
-Dsurefire.failIfNoSpecifiedTests=false \
-Dfailsafe.failIfNoSpecifiedTests=false \
test
RETURN_CODE=$?
;;
*)
;;
esac
Expand Down
2 changes: 1 addition & 1 deletion .kokoro/continuous/graalvm-native-17.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Configure the docker image for kokoro-trampoline.
env_vars: {
key: "TRAMPOLINE_IMAGE"
value: "gcr.io/cloud-devrel-kokoro-resources/graalvm17:22.3.2"
value: "gcr.io/cloud-devrel-kokoro-resources/graalvm17:22.3.3"
}

env_vars: {
Expand Down
2 changes: 1 addition & 1 deletion .kokoro/continuous/graalvm-native.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Configure the docker image for kokoro-trampoline.
env_vars: {
key: "TRAMPOLINE_IMAGE"
value: "gcr.io/cloud-devrel-kokoro-resources/graalvm:22.3.2"
value: "gcr.io/cloud-devrel-kokoro-resources/graalvm:22.3.3"
}

env_vars: {
Expand Down
1 change: 1 addition & 0 deletions .kokoro/nightly/integration.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,4 @@ env_vars: {
key: "SECRET_MANAGER_KEYS"
value: "java-it-service-account"
}

1 change: 1 addition & 0 deletions .kokoro/nightly/java11-integration.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,4 @@ env_vars: {
key: "SECRET_MANAGER_KEYS"
value: "java-it-service-account"
}

Original file line number Diff line number Diff line change
@@ -1,25 +1,29 @@
# Format: //devtools/kokoro/config/proto/build.proto

# Configure the docker image for kokoro-trampoline.
env_vars: {
key: "TRAMPOLINE_IMAGE"
value: "gcr.io/cloud-devrel-kokoro-resources/graalvm17:22.3.2"
}

env_vars: {
key: "JOB_TYPE"
value: "graalvm17"
value: "gcr.io/cloud-devrel-kokoro-resources/java8"
}

# TODO: remove this after we've migrated all tests and scripts
env_vars: {
key: "GCLOUD_PROJECT"
value: "gcloud-devel"
value: "bq-write-api-java-retry-test"
}

env_vars: {
key: "GOOGLE_CLOUD_PROJECT"
value: "gcloud-devel"
value: "bq-write-api-java-retry-test"
}

env_vars: {
key: "JOB_TYPE"
value: "retry_non_quota"
}

env_vars: {
key: "ENABLE_FLAKYBOT"
value: "true"
}

env_vars: {
Expand All @@ -30,4 +34,4 @@ env_vars: {
env_vars: {
key: "SECRET_MANAGER_KEYS"
value: "java-it-service-account"
}
}
Original file line number Diff line number Diff line change
@@ -1,25 +1,29 @@
# Format: //devtools/kokoro/config/proto/build.proto

# Configure the docker image for kokoro-trampoline.
env_vars: {
key: "TRAMPOLINE_IMAGE"
value: "gcr.io/cloud-devrel-kokoro-resources/graalvm:22.3.2"
}

env_vars: {
key: "JOB_TYPE"
value: "graalvm"
value: "gcr.io/cloud-devrel-kokoro-resources/java8"
}

# TODO: remove this after we've migrated all tests and scripts
env_vars: {
key: "GCLOUD_PROJECT"
value: "gcloud-devel"
value: "bq-writeapi-java-quota-retry"
}

env_vars: {
key: "GOOGLE_CLOUD_PROJECT"
value: "gcloud-devel"
value: "bq-writeapi-java-quota-retry"
}

env_vars: {
key: "JOB_TYPE"
value: "retry_quota"
}

env_vars: {
key: "ENABLE_FLAKYBOT"
value: "true"
}

env_vars: {
Expand Down
1 change: 1 addition & 0 deletions .kokoro/presubmit/integration.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,4 @@ env_vars: {
key: "SECRET_MANAGER_KEYS"
value: "java-it-service-account"
}

10 changes: 6 additions & 4 deletions .kokoro/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,7 @@ protobuf==3.20.3 \
# gcp-docuploader
# gcp-releasetool
# google-api-core
# googleapis-common-protos
pyasn1==0.5.0 \
--hash=sha256:87a2121042a1ac9358cabcaf1d07680ff97ee6404333bacca15f76aa8ad01a57 \
--hash=sha256:97b7290ca68e62a832558ec3976f15cbf911bf5d7c7039d8b861c2a0ece69fde
Expand Down Expand Up @@ -469,9 +470,9 @@ typing-extensions==4.7.1 \
--hash=sha256:440d5dd3af93b060174bf433bccd69b0babc3b15b1a8dca43789fd7f61514b36 \
--hash=sha256:b75ddc264f0ba5615db7ba217daeb99701ad295353c45f9e95963337ceeeffb2
# via -r requirements.in
urllib3==1.26.16 \
--hash=sha256:8d36afa7616d8ab714608411b4a3b13e58f463aee519024578e062e141dce20f \
--hash=sha256:8f135f6502756bde6b2a9b28989df5fbe87c9970cecaa69041edcce7f0589b14
urllib3==1.26.18 \
--hash=sha256:34b97092d7e0a3a8cf7cd10e386f401b3737364026c45e622aa02903dffe0f07 \
--hash=sha256:f8ecc1bba5667413457c529ab955bf8c67b45db799d159066261719e328580a0
# via
# google-auth
# requests
Expand All @@ -485,5 +486,6 @@ zipp==3.16.1 \
# via importlib-metadata

# WARNING: The following packages were not pinned, but pip requires them to be
# pinned when the requirements file includes hashes. Consider using the --allow-unsafe flag.
# pinned when the requirements file includes hashes and the requirement is not
# satisfied by a package already installed. Consider using the --allow-unsafe flag.
# setuptools
Loading
Loading