Fix failing test #20634

rodireich · 2022-12-19T05:35:57Z

What

Fix failing tests in source-oracle and source-db2 due to previous change in #19776

rodireich · 2022-12-19T05:36:59Z

/test connector=connectors/source-oracle

rodireich · 2022-12-19T05:41:47Z

/test connector=connectors/source-db2

🕑 connectors/source-db2 https://github.com/airbytehq/airbyte/actions/runs/3728778650
❌ connectors/source-db2 https://github.com/airbytehq/airbyte/actions/runs/3728778650
🐛 https://gradle.com/s/lml2j4zwoqu76

Build Failed

Test summary info:

Could not find result summary

rodireich · 2022-12-19T05:55:09Z

/test connector=connectors/source-oracle

🕑 connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3728831144
❌ connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3728831144
🐛 https://gradle.com/s/hehxb32qgabsk

Build Failed

Test summary info:

Could not find result summary

rodireich · 2022-12-19T16:49:43Z

/test connector=connectors/source-oracle

🕑 connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3733428724
❌ connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3733428724
🐛 https://gradle.com/s/fuffe5nbxun72

Build Failed

Test summary info:

Could not find result summary

rodireich · 2022-12-19T16:51:44Z

/test connector=connectors/source-db2

rodireich · 2022-12-19T21:47:20Z

/test connector=connectors/source-db2

🕑 connectors/source-db2 https://github.com/airbytehq/airbyte/actions/runs/3735444639
✅ connectors/source-db2 https://github.com/airbytehq/airbyte/actions/runs/3735444639
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1603    336    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 26.82s ==================

rodireich · 2022-12-19T21:56:49Z

/test connector=connectors/source-oracle

🕑 connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3735494900
❌ connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3735494900
🐛 https://gradle.com/s/j7woqcc36si6c

Build Failed

Test summary info:

Could not find result summary

rodireich · 2022-12-20T01:25:16Z

/test connector=connectors/source-oracle

🕑 connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3736629309

rodireich · 2022-12-20T01:27:12Z

/test connector=connectors/source-oracle

🕑 connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3736637551
❌ connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3736637551
🐛 https://gradle.com/s/gldrpvpgeoe7k

Build Failed

Test summary info:

Could not find result summary

rodireich · 2022-12-20T05:14:16Z

/test connector=connectors/source-oracle

🕑 connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3737732471
❌ connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3737732471
🐛 https://gradle.com/s/3hzv2komtzku4

Build Failed

Test summary info:

Could not find result summary

rodireich · 2022-12-20T07:29:03Z

/test connector=connectors/source-oracle

🕑 connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3738412930
✅ connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/3738412930
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1603    336    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 25.74s ==================

ryankfu · 2022-12-20T19:12:03Z

...rc/test-integration/java/io/airbyte/integrations/source/oracle/OracleSourceDatatypeTest.java

@@ -146,7 +147,8 @@ protected void initTests() {
            .sourceType("NUMBER")
            .airbyteType(JsonSchemaType.NUMBER)
            .addInsertValues("null", "1", "123.45", "power(10, -130)", "9.99999999999999999999 * power(10, 125)")
-            .addExpectedValues(null, "1", "123.45", String.valueOf(Math.pow(10, -130)), String.valueOf(9.99999999999999999999 * Math.pow(10, 125)))
+            .addExpectedValues(null, "1", "123.45", String.valueOf(Math.pow(10, -130)),
+                "999999999999999999999000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")


Is this not possible to have the last expected value be

String.valueOf(9.99999999999999999999 * Math.pow(10, 125)

This would be difficult to understand, but if the usage of Math.pow(10,125) is not an option can we add a comment to this test case explaining how come this string is the way it is?

The String.valueOf we used to have was printed as "1.0E126" - the scientific notation.
I couldn't find a way to compel BigDecimal to accept a value using pow that is not printed as a 1 with 126 0s, so had to resort to direct string.
I'll add a comment.

I might add in the comment the reason how come it's a plain string representation, like something along the lines of

"because normalization expects values in integer strings whereas `Math.pow(10, 125)` returns a scientific notation

Added. thank you

...rc/test-integration/java/io/airbyte/integrations/source/oracle/OracleSourceDatatypeTest.java

ryankfu · 2022-12-20T19:25:32Z

...java/io/airbyte/integrations/io/airbyte/integration_tests/sources/Db2SourceDatatypeTest.java

@@ -126,7 +127,7 @@ protected void initTests() {
            .airbyteType(JsonSchemaType.NUMBER)
            .fullSourceDataType("DECIMAL(31, 0)")
            .addInsertValues("null", "1", "DECIMAL((-1 + 10E+29), 31, 0)", "DECIMAL((1 - 10E+29), 31, 0)")
-            .addExpectedValues(null, "1", "1.0E30", "-1.0E30")
+            .addExpectedValues(null, "1", "%.0f".formatted(Double.valueOf("1.0E30")), "%.0f".formatted(Double.valueOf("-1.0E30")))


If I'm understanding this correctly, this will be showing integer values only, right? I haven't seen the usage of %.0f before, normally it would have one decimal precision value.

Also this seems to work but curious if this should be

Float.valueOf("1.0E30")

since the string formatting is indicating a Float f vs Double d

The weeds of floating point string format.
How I missed thee… :-)

f signifies a floating point number - a d modifier is a decimal integer I think, but it won't accept a double or float value.

the %.0f formats a floating point number to print with 0 decimal digits (0 precision) - which is the expected string.

And a Float.valueOf("1.0E30") is theoretically ok, but depending on actual hardware and implementation won't print as a clean 1 with zeroes but some weird floating point "1000000015047466200000000000000" and so on- it's the whole single/double precision implementation of respective float/double types.

Thanks for the explanation, nothing else to add and glad for the clarification

ryankfu

Overall looks good, minor nit and comments to make sure business context is clear

rodireich · 2022-12-20T23:16:19Z

/approve-and-merge reason="Low risk: Fixing test only code that's broken today to unblock the team"

octavia-approvington · 2022-12-20T23:17:01Z

Approve this

Fix failing test

05b61a1

rodireich linked an issue Dec 19, 2022 that may be closed by this pull request

Snowflake source encodes integers as floats #19776

Closed

octavia-squidington-iv added area/connectors Connector related issues connectors/source/oracle labels Dec 19, 2022

Fix failing test

92667a9

octavia-squidington-iv added the connectors/source/db2 label Dec 19, 2022

Fix failing test

0fc9606

Fix failing test

163b8f3

Fix failing test

7e2629a

Fix failing test

902ff86

Fix failing test

2c80758

Fix failing test

eb2bb40

Fix failing test

736d2a1

rodireich requested a review from VitaliiMaltsev December 20, 2022 16:27

rodireich marked this pull request as ready for review December 20, 2022 16:27

rodireich requested a review from a team as a code owner December 20, 2022 16:27

Merge branch 'master' into 19776-fix-failing-tests

cc2dd2f

akashkulk approved these changes Dec 20, 2022

View reviewed changes

ryankfu reviewed Dec 20, 2022

View reviewed changes

...rc/test-integration/java/io/airbyte/integrations/source/oracle/OracleSourceDatatypeTest.java Outdated Show resolved Hide resolved

comment

e8007ca

ryankfu reviewed Dec 20, 2022

View reviewed changes

ryankfu approved these changes Dec 20, 2022

View reviewed changes

comment

3847509

VitaliiMaltsev mentioned this pull request Dec 20, 2022

JDBC Sources: Consolidate putDate behavior, and potentially other methods #17068

Closed

Merge branch 'master' into 19776-fix-failing-tests

a1839e6

octavia-approvington approved these changes Dec 20, 2022

View reviewed changes

octavia-approvington merged commit 401aa44 into master Dec 20, 2022

octavia-approvington deleted the 19776-fix-failing-tests branch December 20, 2022 23:17

github-actions bot mentioned this pull request Dec 27, 2022

Bump helm chart version reference to 0.43.9 #20892

Merged

This was referenced Jan 6, 2023

Bump Airbyte version from 0.40.26 to 0.40.27 #21092

Closed

Bump Airbyte version from 0.40.26 to 0.40.27 #21121

Closed

This was referenced Jan 6, 2023

Bump Airbyte version from 0.40.26 to 0.40.27 #21130

Closed

Bump Airbyte version from 0.40.26 to 0.40.27 #21135

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix failing test #20634

Fix failing test #20634

rodireich commented Dec 19, 2022 •

edited

Loading

rodireich commented Dec 19, 2022

rodireich commented Dec 19, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 19, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 19, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 19, 2022

rodireich commented Dec 19, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 19, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 20, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 20, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 20, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 20, 2022 •

edited by github-actions bot

Loading

ryankfu Dec 20, 2022

rodireich Dec 20, 2022

ryankfu Dec 20, 2022

rodireich Dec 20, 2022

ryankfu Dec 20, 2022

rodireich Dec 20, 2022 •

edited

Loading

ryankfu Dec 20, 2022

ryankfu left a comment

rodireich commented Dec 20, 2022

octavia-approvington commented Dec 20, 2022

Fix failing test #20634

Fix failing test #20634

Conversation

rodireich commented Dec 19, 2022 • edited Loading

What

rodireich commented Dec 19, 2022

rodireich commented Dec 19, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Dec 19, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Dec 19, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Dec 19, 2022

rodireich commented Dec 19, 2022 • edited by github-actions bot Loading

Build Passed

rodireich commented Dec 19, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Dec 20, 2022 • edited by github-actions bot Loading

rodireich commented Dec 20, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Dec 20, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Dec 20, 2022 • edited by github-actions bot Loading

Build Passed

ryankfu Dec 20, 2022

Choose a reason for hiding this comment

rodireich Dec 20, 2022

Choose a reason for hiding this comment

ryankfu Dec 20, 2022

Choose a reason for hiding this comment

rodireich Dec 20, 2022

Choose a reason for hiding this comment

ryankfu Dec 20, 2022

Choose a reason for hiding this comment

rodireich Dec 20, 2022 • edited Loading

Choose a reason for hiding this comment

ryankfu Dec 20, 2022

Choose a reason for hiding this comment

ryankfu left a comment

Choose a reason for hiding this comment

rodireich commented Dec 20, 2022

octavia-approvington commented Dec 20, 2022

rodireich commented Dec 19, 2022 •

edited

Loading

rodireich commented Dec 19, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 19, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 19, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 19, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 19, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 20, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 20, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 20, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 20, 2022 •

edited by github-actions bot

Loading

rodireich Dec 20, 2022 •

edited

Loading