🐛 Source Smartsheets: fix scrambled columns bug #11911

davydov-d · 2022-04-12T07:27:32Z

What

[
  {'id': 1101932201830276, 'index': 0, 'title': 'id', 'type': 'TEXT_NUMBER'...},
  {'id': 5605531829200772, 'index': 1, 'title': 'first_name', 'type': 'TEXT_NUMBER'...}, 
  {'id': 3353732015515524, 'index': 2, 'title': 'last_name', 'type': 'TEXT_NUMBER'...}, 
  {'id': 7857331642886020, 'index': 3, 'title': 'email', 'type': 'TEXT_NUMBER'...}
]

Moving email column in the UI to the last position and performing read gives following results:

{"id": "99.0", "first_name": "Auria", "last_name": "Haryngton", "gender": "aharyngton2q@mapquest.com", "ip_address": "Non-binary", "dob": "246.28.159.95", "email": "2020-06-28"}

How

Do not take columns from Stream instance, use Sheet instance instead
Map cells and columns by columnId
Add unit test
Fix integration test
Speed up Dockerfile (perform COPY after pip install)

codecov · 2022-04-12T07:32:20Z

Codecov Report

❗ No coverage uploaded for pull request base (master@fa9c467). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #11911   +/-   ##
=========================================
  Coverage          ?   45.20%           
=========================================
  Files             ?        2           
  Lines             ?       73           
  Branches          ?        0           
=========================================
  Hits              ?       33           
  Misses            ?       40           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa9c467...ef45264. Read the comment docs.

davydov-d · 2022-04-12T07:34:33Z

/test connnector=connectors/source-smartsheets

Error: Unexpected inputs provided: ["connnector"]

davydov-d · 2022-04-12T07:35:11Z

/test connector=connectors/source-smartsheets

🕑 connectors/source-smartsheets https://github.com/airbytehq/airbyte/actions/runs/2153413712
✅ connectors/source-smartsheets https://github.com/airbytehq/airbyte/actions/runs/2153413712
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                             Stmts   Miss  Cover
----------------------------------------------------
source_smartsheets/__init__.py       2      0   100%
source_smartsheets/source.py        71     40    44%
----------------------------------------------------
TOTAL                               73     40    45%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 20 passed, 1 skipped in 8.85s =========================

airbyte-integrations/connectors/source-smartsheets/setup.py

airbyte-integrations/connectors/source-smartsheets/Dockerfile

airbyte-integrations/connectors/source-smartsheets/setup.py

…crambled

davydov-d · 2022-04-14T07:52:57Z

/test connector=connectors/source-smartsheets

🕑 connectors/source-smartsheets https://github.com/airbytehq/airbyte/actions/runs/2165921820
✅ connectors/source-smartsheets https://github.com/airbytehq/airbyte/actions/runs/2165921820
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                             Stmts   Miss  Cover
----------------------------------------------------
source_smartsheets/__init__.py       2      0   100%
source_smartsheets/source.py        71     40    44%
----------------------------------------------------
TOTAL                               73     40    45%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 20 passed, 1 skipped in 9.18s =========================

airbyte-integrations/connectors/source-smartsheets/unit_tests/test_source.py

davydov-d · 2022-04-18T14:00:08Z

/test connector=connectors/source-smartsheets

🕑 connectors/source-smartsheets https://github.com/airbytehq/airbyte/actions/runs/2184214471
✅ connectors/source-smartsheets https://github.com/airbytehq/airbyte/actions/runs/2184214471
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                             Stmts   Miss  Cover
----------------------------------------------------
source_smartsheets/__init__.py       2      0   100%
source_smartsheets/source.py        71     40    44%
----------------------------------------------------
TOTAL                               73     40    45%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 20 passed, 1 skipped in 9.56s =========================

midavadim

This update can be released without Airbyte review

davydov-d · 2022-04-18T16:16:13Z

/publish connector=connectors/source-smartsheets

🕑 connectors/source-smartsheets https://github.com/airbytehq/airbyte/actions/runs/2184811347
🚀 Successfully published connectors/source-smartsheets
🚀 Auto-bumped version for connectors/source-smartsheets
✅ connectors/source-smartsheets https://github.com/airbytehq/airbyte/actions/runs/2184811347

* #5520 fix scrambled columns bug * #5520 source smartsheets: add changelog item * #5520 move pytest to optional setup requirements * #5520 source smartsheets: add two unit tests * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

#5520 fix scrambled columns bug

24e25fb

github-actions bot added the area/connectors Connector related issues label Apr 12, 2022

#5520 source smartsheets: add changelog item

75830fe

github-actions bot added the area/documentation Improvements or additions to documentation label Apr 12, 2022

davydov-d requested review from annalvova05 and grubberr April 12, 2022 07:52

grubberr reviewed Apr 14, 2022

View reviewed changes

airbyte-integrations/connectors/source-smartsheets/setup.py Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-smartsheets/Dockerfile Show resolved Hide resolved

grubberr approved these changes Apr 14, 2022

View reviewed changes

airbyte-integrations/connectors/source-smartsheets/setup.py Outdated Show resolved Hide resolved

davydov-d added 2 commits April 14, 2022 10:46

Merge branch 'master' into ddavydov/5520-source-smartsheets-columns-s…

0f8813b

…crambled

#5520 move pytest to optional setup requirements

5a3165d

davydov-d requested a review from midavadim April 14, 2022 07:56

This was linked to issues Apr 14, 2022

Smart Sheet Conector / Columns Scrambled #5520

Closed

Smartsheets connector raw key:value pair data is scrambled. #8099

Closed

annalvova05 approved these changes Apr 14, 2022

View reviewed changes

midavadim reviewed Apr 18, 2022

View reviewed changes

airbyte-integrations/connectors/source-smartsheets/unit_tests/test_source.py Show resolved Hide resolved

#5520 source smartsheets: add two unit tests

93f5866

davydov-d requested a review from midavadim April 18, 2022 14:01

midavadim approved these changes Apr 18, 2022

View reviewed changes

auto-bump connector version

ef45264

octavia-squidington-iii temporarily deployed to more-secrets April 18, 2022 16:30 Inactive

davydov-d merged commit 5c5f38a into master Apr 18, 2022

davydov-d deleted the ddavydov/5520-source-smartsheets-columns-scrambled branch April 18, 2022 17:08

octavia-squidington-iii mentioned this pull request Apr 19, 2022

Bump Airbyte version from 0.36.0-alpha to 0.36.1-alpha #12148

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Smartsheets: fix scrambled columns bug #11911

🐛 Source Smartsheets: fix scrambled columns bug #11911

davydov-d commented Apr 12, 2022 •

edited

Loading

codecov bot commented Apr 12, 2022 •

edited

Loading

davydov-d commented Apr 12, 2022 •

edited by github-actions bot

Loading

davydov-d commented Apr 12, 2022 •

edited by github-actions bot

Loading

davydov-d commented Apr 14, 2022 •

edited by github-actions bot

Loading

davydov-d commented Apr 18, 2022 •

edited by github-actions bot

Loading

midavadim left a comment

davydov-d commented Apr 18, 2022 •

edited by github-actions bot

Loading

🐛 Source Smartsheets: fix scrambled columns bug #11911

🐛 Source Smartsheets: fix scrambled columns bug #11911

Conversation

davydov-d commented Apr 12, 2022 • edited Loading

What

How

codecov bot commented Apr 12, 2022 • edited Loading

Codecov Report

davydov-d commented Apr 12, 2022 • edited by github-actions bot Loading

davydov-d commented Apr 12, 2022 • edited by github-actions bot Loading

davydov-d commented Apr 14, 2022 • edited by github-actions bot Loading

davydov-d commented Apr 18, 2022 • edited by github-actions bot Loading

midavadim left a comment

Choose a reason for hiding this comment

davydov-d commented Apr 18, 2022 • edited by github-actions bot Loading

davydov-d commented Apr 12, 2022 •

edited

Loading

codecov bot commented Apr 12, 2022 •

edited

Loading

davydov-d commented Apr 12, 2022 •

edited by github-actions bot

Loading

davydov-d commented Apr 12, 2022 •

edited by github-actions bot

Loading

davydov-d commented Apr 14, 2022 •

edited by github-actions bot

Loading

davydov-d commented Apr 18, 2022 •

edited by github-actions bot

Loading

davydov-d commented Apr 18, 2022 •

edited by github-actions bot

Loading