🐛 Source Airtable: fix cases with empty `result` while discover the schema #22245

bazarnov · 2023-02-01T22:46:34Z

What

Resolving: https://github.com/airbytehq/oncall/issues/1432

How

for certain fields types, the result object could be Null, which means we should fallback to string type

🚨 User Impact 🚨

No impact is expected. Just fixing the blocker.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

bazarnov · 2023-02-01T22:51:19Z

/test connector=connectors/source-airtable

🕑 connectors/source-airtable https://github.com/airbytehq/airbyte/actions/runs/4069438791
✅ connectors/source-airtable https://github.com/airbytehq/airbyte/actions/runs/4069438791
Python tests coverage:

Name                                Stmts   Miss  Cover
-------------------------------------------------------
source_airtable/__init__.py             2      0   100%
source_airtable/streams.py             92      1    99%
source_airtable/source.py              38      1    97%
source_airtable/schema_helpers.py      43      6    86%
source_airtable/auth.py                21     10    52%
-------------------------------------------------------
TOTAL                                 196     18    91%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              476    117    75%   53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     114     13    89%   31-32, 39, 42, 66-69, 97, 121, 203-205
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1690    341    80%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Incremental syncs are not supported on this connector.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
================== 44 passed, 4 skipped in 140.52s (0:02:20) ===================

pedroslopez

Thanks!

pedroslopez · 2023-02-01T23:15:11Z

airbyte-integrations/connectors/source-airtable/source_airtable/schema_helpers.py

@@ -89,17 +91,22 @@ def get_json_schema(table: Dict[str, Any]) -> Dict[str, str]:
            name: str = SchemaHelpers.clean_name(field.get("name"))
            original_type: str = field.get("type")
            options: Dict = field.get("options", {})
-            exec_type: str = options.get("result", {}).get("type") if options else None
-            field_type: str = exec_type if exec_type else original_type
+            options_result: Dict = options.get("result", {})


do we need an if options else None here?

Kept that to ensure options: null case is covered, just in case)

pedroslopez · 2023-02-02T06:37:41Z

Just got paged for this - publishing now.

pedroslopez · 2023-02-02T06:37:53Z

/publish connector=airbyte/source-airtable

🕑 Publishing the following connectors:
airbyte/source-airtable
https://github.com/airbytehq/airbyte/actions/runs/4071920637

Connector	Did it publish?	Were definitions generated?
airbyte/source-airtable	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

pedroslopez · 2023-02-02T06:40:18Z

/publish connector=connectors/source-airtable

🕑 Publishing the following connectors:
connectors/source-airtable
https://github.com/airbytehq/airbyte/actions/runs/4071933407

Connector	Did it publish?	Were definitions generated?
connectors/source-airtable	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2023-02-02T07:17:01Z

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage	24.52%	❌

fixed

f2e13d1

bazarnov requested a review from pedroslopez February 1, 2023 22:46

bazarnov self-assigned this Feb 1, 2023

octavia-squidington-iii added area/connectors Connector related issues connectors/source/airtable labels Feb 1, 2023

updated changelog

0651fab

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Feb 1, 2023

pedroslopez approved these changes Feb 1, 2023

View reviewed changes

auto-bump connector version

df27136

airbyteio temporarily deployed to more-secrets February 2, 2023 07:08 — with GitHub Actions Inactive

pedroslopez enabled auto-merge (squash) February 2, 2023 07:13

Merge branch 'master' into baz/source-airtable-fix-fields-results

f594d8a

pedroslopez temporarily deployed to more-secrets February 2, 2023 08:06 — with GitHub Actions Inactive

pedroslopez merged commit 244776c into master Feb 2, 2023

pedroslopez deleted the baz/source-airtable-fix-fields-results branch February 2, 2023 08:36

octavia-squidington-iii mentioned this pull request Feb 2, 2023

Bump Airbyte version from 0.40.32 to 0.40.33 #22335

Closed

github-actions bot mentioned this pull request Feb 7, 2023

Bump helm chart version reference to 0.43.28 #22489

Closed

This was referenced Feb 8, 2023

Bump Airbyte version from 0.40.32 to 0.40.33 #22578

Closed

Bump Airbyte version from 0.40.32 to 0.40.33 #22689

Closed

Bump Airbyte version from 0.40.32 to 0.40.33 #22839

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Airtable: fix cases with empty `result` while discover the schema #22245

🐛 Source Airtable: fix cases with empty `result` while discover the schema #22245

bazarnov commented Feb 1, 2023

bazarnov commented Feb 1, 2023 •

edited by github-actions bot

pedroslopez left a comment

pedroslopez Feb 1, 2023

bazarnov Feb 2, 2023

pedroslopez commented Feb 2, 2023

pedroslopez commented Feb 2, 2023 •

edited by github-actions bot

pedroslopez commented Feb 2, 2023 •

edited by github-actions bot

github-actions bot commented Feb 2, 2023

🐛 Source Airtable: fix cases with empty result while discover the schema #22245

🐛 Source Airtable: fix cases with empty result while discover the schema #22245

Conversation

bazarnov commented Feb 1, 2023

What

How

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

bazarnov commented Feb 1, 2023 • edited by github-actions bot

Build Passed

pedroslopez left a comment

Choose a reason for hiding this comment

pedroslopez Feb 1, 2023

Choose a reason for hiding this comment

bazarnov Feb 2, 2023

Choose a reason for hiding this comment

pedroslopez commented Feb 2, 2023

pedroslopez commented Feb 2, 2023 • edited by github-actions bot

pedroslopez commented Feb 2, 2023 • edited by github-actions bot

github-actions bot commented Feb 2, 2023

Airbyte Code Coverage

🐛 Source Airtable: fix cases with empty `result` while discover the schema #22245

🐛 Source Airtable: fix cases with empty `result` while discover the schema #22245

bazarnov commented Feb 1, 2023 •

edited by github-actions bot

pedroslopez commented Feb 2, 2023 •

edited by github-actions bot

pedroslopez commented Feb 2, 2023 •

edited by github-actions bot