🐛 Source Hubspot: use RFC3339 datetime #13159

grubberr · 2022-05-25T04:31:25Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

ReplicationWorker generates errors because date-time fields in "properties" object not RFC3339 compatible

2022-05-25 03:49:19 WARN i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):305 - Record schema validation failed. Errors: json schema validation failed when comparing the data to the json schema. 
Errors: $.properties.createdate: 2021-05-21 10:17:06 is an invalid date-time, $.properties.hs_lastmodifieddate: 2022-05-11 05:24:04 is an invalid date-time, $.properties.hubspot_owner_assigneddate: 2021-05-21 10:17:28 is an invalid date-time, $.properties.hs_createdate: 2021-05-21 10:17:28 is an invalid date-time, $.properties.closedate: 2021-05-31 10:17:06 is an invalid date-time, $.properties.hs_date_entered_appointmentscheduled: 2021-05-21 10:17:28 is an invalid date-time

Make datetime fields in "properties" object to comply RFC3339 standard

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-05-25T04:31:57Z

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2382150653
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2382150653
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                         Stmts   Miss  Cover
------------------------------------------------
source_hubspot/errors.py         6      0   100%
source_hubspot/__init__.py       2      0   100%
source_hubspot/streams.py      714     61    91%
source_hubspot/source.py        87     25    71%
------------------------------------------------
TOTAL                          809     86    89%

Build Passed

Test summary info:

All Passed

codecov · 2022-05-25T04:37:23Z

Codecov Report

❗ No coverage uploaded for pull request base (master@3bddf87). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head c2470eb differs from pull request most recent head c24a975. Consider uploading reports for the commit c24a975 to get more accurate results

@@            Coverage Diff            @@
##             master   #13159   +/-   ##
=========================================
  Coverage          ?   89.36%           
=========================================
  Files             ?        4           
  Lines             ?      809           
  Branches          ?        0           
=========================================
  Hits              ?      723           
  Misses            ?       86           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3bddf87...c24a975. Read the comment docs.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-05-25T05:03:15Z

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2382244782
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2382244782
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                         Stmts   Miss  Cover
------------------------------------------------
source_hubspot/errors.py         6      0   100%
source_hubspot/__init__.py       2      0   100%
source_hubspot/streams.py      714     61    91%
source_hubspot/source.py        87     25    71%
------------------------------------------------
TOTAL                          809     86    89%

Build Passed

Test summary info:

All Passed

lazebnyi

LGTM

grubberr · 2022-05-30T10:19:50Z

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2408370135
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2408370135
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                         Stmts   Miss  Cover
------------------------------------------------
source_hubspot/errors.py         6      0   100%
source_hubspot/__init__.py       2      0   100%
source_hubspot/streams.py      714     61    91%
source_hubspot/source.py        87     25    71%
------------------------------------------------
TOTAL                          809     86    89%

Build Passed

Test summary info:

All Passed

grubberr · 2022-05-30T12:31:02Z

/publish connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2409056708
🚀 Successfully published connectors/source-hubspot
🚀 Auto-bumped version for connectors/source-hubspot
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2409056708

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

to_datetime_string -> to_rfc3339_string

4373cb8

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot added the area/connectors Connector related issues label May 25, 2022

grubberr linked an issue May 25, 2022 that may be closed by this pull request

Fix Hubspot schema and add date-time format test in SAT #12333

Closed

grubberr self-assigned this May 25, 2022

grubberr added 2 commits May 25, 2022 07:58

change to to_rfc3339_string (2)

4c3170c

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

0.1.59 -> 0.1.60

604d067

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr changed the title ~~Source Hubspot: use rfc3339 datetime~~ Source Hubspot: use RFC3339 datetime May 25, 2022

github-actions bot added the area/documentation Improvements or additions to documentation label May 25, 2022

grubberr requested a review from lazebnyi May 25, 2022 05:15

grubberr changed the title ~~Source Hubspot: use RFC3339 datetime~~ 🐛 Source Hubspot: use RFC3339 datetime May 26, 2022

lazebnyi approved these changes May 30, 2022

View reviewed changes

Merge branch 'master' into grubberr/12333-source-hubspot

c2470eb

auto-bump connector version

c24a975

grubberr merged commit ad1fd2b into master May 30, 2022

grubberr deleted the grubberr/12333-source-hubspot branch May 30, 2022 12:54

octavia-squidington-iii temporarily deployed to more-secrets May 30, 2022 12:55 Inactive

octavia-squidington-iii mentioned this pull request May 31, 2022

Bump Airbyte version from 0.39.5-alpha to 0.39.6-alpha #13350

Merged

pedroslopez mentioned this pull request Jun 1, 2022

🐛 Source Amplitude: Failed to apply RFC3339 pattern on date-time string #13075

Closed

jscottpolevault pushed a commit to jscottpolevault/airbyte that referenced this pull request Jun 1, 2022

🐛 Source Hubspot: use RFC3339 datetime (airbytehq#13159)

3831cb8

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Hubspot: use RFC3339 datetime #13159

🐛 Source Hubspot: use RFC3339 datetime #13159

grubberr commented May 25, 2022 •

edited

grubberr commented May 25, 2022 •

edited by github-actions bot

codecov bot commented May 25, 2022 •

edited

grubberr commented May 25, 2022 •

edited by github-actions bot

lazebnyi left a comment

grubberr commented May 30, 2022 •

edited by github-actions bot

grubberr commented May 30, 2022 •

edited by github-actions bot

🐛 Source Hubspot: use RFC3339 datetime #13159

🐛 Source Hubspot: use RFC3339 datetime #13159

Conversation

grubberr commented May 25, 2022 • edited

What

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

grubberr commented May 25, 2022 • edited by github-actions bot

Build Passed

codecov bot commented May 25, 2022 • edited

Codecov Report

grubberr commented May 25, 2022 • edited by github-actions bot

Build Passed

lazebnyi left a comment

Choose a reason for hiding this comment

grubberr commented May 30, 2022 • edited by github-actions bot

Build Passed

grubberr commented May 30, 2022 • edited by github-actions bot

grubberr commented May 25, 2022 •

edited

grubberr commented May 25, 2022 •

edited by github-actions bot

codecov bot commented May 25, 2022 •

edited

grubberr commented May 25, 2022 •

edited by github-actions bot

grubberr commented May 30, 2022 •

edited by github-actions bot

grubberr commented May 30, 2022 •

edited by github-actions bot