Source file: remove nested properties #14535

davydov-d · 2022-07-08T11:46:04Z

What

File source generates invalid schema for JSON format - it contains an excessive empty properties layer

How

Remove redundant layer

Before

{
"streams": [
  {
    "stream": {
      "name": "us_p0_json_18",
      "json_schema": {
        "type": "object",
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "type": "object",
          "$schema": "http://json-schema.org/schema#",
          "required": [
            "data",
            "meta"
          ],
          "properties": {
            "data": {
              "type": "array",
              "items": {
                "type": "array",
                "items": {
                  "type": [
                    "integer",
                    "null",
                    "string"
                  ]
                }
              }
            },
            "meta": {
              "type": "object",
              "required": [
                "view"
              ],
              "properties": {
                "view": {
                  "type": "object",
                  "required": [
                    "approvals",
                    "assetType",
                    "attribution",
                    "attributionLink",
                    "averageRating",~
                   ...

After

{
"streams": [
  {
    "stream": {
      "name": "us_p0_json_18",
      "json_schema": {
        "type": "object",
        "$schema": "http://json-schema.org/draft-07/schema#",
        "required": [
            "data",
            "meta"
        ],
        "properties": {
            "data": {
              "type": "array",
              "items": {
                "type": "array",
                "items": {
                  "type": [
                    "integer",
                    "null",
                    "string"
                  ]
                }
              }
            },
            "meta": {
              "type": "object",
              "required": [
                "view"
              ],
              "properties": {
                "view": {
                  "type": "object",
                  "required": [
                    "approvals",
                    "assetType",
                    "attribution",
                    "attributionLink",
                    "averageRating",

airbyte-integrations/connectors/source-file/source_file/client.py

davydov-d · 2022-07-08T11:50:17Z

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2635889687
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2635889687
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/client.py       235     35    85%
source_file/source.py        50     35    30%
---------------------------------------------
TOTAL                       287     70    76%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/source.py        50     27    46%
source_file/client.py       235    140    40%
---------------------------------------------
TOTAL                       287    167    42%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 21 passed, 1 skipped in 25.97s ========================

davydov-d · 2022-07-12T10:50:47Z

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2655967640
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2655967640
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/client.py       236     36    85%
source_file/source.py        50     35    30%
---------------------------------------------
TOTAL                       288     71    75%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/source.py        50     27    46%
source_file/client.py       236    141    40%
---------------------------------------------
TOTAL                       288    168    42%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 21 passed, 1 skipped in 24.44s ========================

lazebnyi

LGTM

davydov-d · 2022-07-12T11:18:47Z

/publish connector=connectors/source-file

🕑 Publishing the following connectors:
connectors/source-file
https://github.com/airbytehq/airbyte/actions/runs/2656117492

Connector	Did it publish?	Were definitions generated?
connectors/source-file	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

davydov-d · 2022-07-12T13:19:16Z

/publish connector=connectors/source-file

🕑 Publishing the following connectors:
connectors/source-file
https://github.com/airbytehq/airbyte/actions/runs/2656793962

Connector	Did it publish?	Were definitions generated?
connectors/source-file	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…eration-for-json-files

davydov-d · 2022-07-12T19:43:45Z

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2658989030
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2658989030
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/client.py       248     37    85%
source_file/source.py        50     35    30%
---------------------------------------------
TOTAL                       300     72    76%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/source.py        50     27    46%
source_file/client.py       248    151    39%
---------------------------------------------
TOTAL                       300    178    41%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 21 passed, 1 skipped in 25.40s ========================

davydov-d · 2022-07-13T08:02:11Z

/publish connector=connectors/source-file

🕑 Publishing the following connectors:
connectors/source-file
https://github.com/airbytehq/airbyte/actions/runs/2661921271

Connector	Did it publish?	Were definitions generated?
connectors/source-file	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

davydov-d · 2022-07-13T08:08:44Z

/publish connector=connectors/source-file

🕑 Publishing the following connectors:
connectors/source-file
https://github.com/airbytehq/airbyte/actions/runs/2661961337

Connector	Did it publish?	Were definitions generated?
connectors/source-file	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

#13787 source file: remove nested properties

dd1175c

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jul 8, 2022

octavia-squidington-iii added the connectors/source/file label Jul 8, 2022

#13787: upd changelog

fbe4e0e

davydov-d linked an issue Jul 8, 2022 that may be closed by this pull request

Source File throwAssertionError: Mismatched number of tables during normalization #13787

Closed

davydov-d commented Jul 8, 2022

View reviewed changes

airbyte-integrations/connectors/source-file/source_file/client.py Show resolved Hide resolved

davydov-d requested review from Phlair, lazebnyi, midavadim, pedroslopez and drrest July 8, 2022 11:51

#13787 revert processing json array, add unit test

dd6a861

lazebnyi approved these changes Jul 12, 2022

View reviewed changes

davydov-d added 2 commits July 12, 2022 22:36

#13787 source file: upgrade pyarrow to allow builds on aarch64

175a3c4

Merge branch 'master' into ddavydov/#13787-source-file-fix-schema-gen…

222b254

…eration-for-json-files

#13787 source file: upd version

cf68015

auto-bump connector version

c09c939

davydov-d merged commit 2570f9c into master Jul 13, 2022

davydov-d deleted the ddavydov/#13787-source-file-fix-schema-generation-for-json-files branch July 13, 2022 08:29

octavia-squidington-iii temporarily deployed to more-secrets July 13, 2022 08:30 Inactive

octavia-squidington-iii mentioned this pull request Jul 13, 2022

Bump Airbyte version from 0.39.35-alpha to 0.39.36-alpha #14690

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source file: remove nested properties #14535

Source file: remove nested properties #14535

davydov-d commented Jul 8, 2022 •

edited

Loading

davydov-d commented Jul 8, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jul 12, 2022 •

edited by github-actions bot

Loading

lazebnyi left a comment

davydov-d commented Jul 12, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jul 12, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jul 12, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jul 13, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jul 13, 2022 •

edited by github-actions bot

Loading

Source file: remove nested properties #14535

Source file: remove nested properties #14535

Conversation

davydov-d commented Jul 8, 2022 • edited Loading

What

How

Before

After

davydov-d commented Jul 8, 2022 • edited by github-actions bot Loading

Build Passed

davydov-d commented Jul 12, 2022 • edited by github-actions bot Loading

Build Passed

lazebnyi left a comment

Choose a reason for hiding this comment

davydov-d commented Jul 12, 2022 • edited by github-actions bot Loading

davydov-d commented Jul 12, 2022 • edited by github-actions bot Loading

davydov-d commented Jul 12, 2022 • edited by github-actions bot Loading

Build Passed

davydov-d commented Jul 13, 2022 • edited by github-actions bot Loading

davydov-d commented Jul 13, 2022 • edited by github-actions bot Loading

davydov-d commented Jul 8, 2022 •

edited

Loading

davydov-d commented Jul 8, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jul 12, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jul 12, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jul 12, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jul 12, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jul 13, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jul 13, 2022 •

edited by github-actions bot

Loading