Source Zendesk Chat: passing timeout in request to prevent a stuck connection #12883

RobertoBonnet · 2022-05-16T19:16:57Z

What

Resolving the problem that occurs when we are getting to much rows from Zendesk chat

#12591

How

Setting a timeout for requests.

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

CLAassistant · 2022-05-16T19:20:21Z

All committers have signed the CLA.

harshithmullapudi · 2022-05-17T07:07:48Z

@RobertoBonnet thanks for the contribution. Can you share the screenshot of sourceAcceptanceTests and also follow these https://docs.airbyte.com/connector-development/#updating-an-existing-connector

RobertoBonnet · 2022-05-17T13:11:40Z

@harshithmullapudi Is this?

RobertoBonnet · 2022-05-17T13:23:12Z

/test connector=connectors/source-zendesk-chat

marcosmarxm · 2022-05-17T13:37:54Z

@bruno-nirello maybe this PR can help you debugging the long running connection with Zendesk.

marcosmarxm · 2022-05-17T13:42:13Z

/test connector=connectors/source-zendesk-chat

🕑 connectors/source-zendesk-chat https://github.com/airbytehq/airbyte/actions/runs/2339027745
❌ connectors/source-zendesk-chat https://github.com/airbytehq/airbyte/actions/runs/2339027745
🐛 https://gradle.com/s/4npukcjblwgmy
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs0] - docker.errors.Cont...
FAILED test_core.py::TestConnection::test_check[inputs1] - docker.errors.Cont...
FAILED test_core.py::TestConnection::test_check[inputs2] - docker.errors.Cont...
FAILED test_core.py::TestConnection::test_check[inputs3] - docker.errors.Cont...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
FAILED test_core.py::TestDiscovery::test_discover[inputs1] - docker.errors.Co...
FAILED test_core.py::TestDiscovery::test_discover[inputs2] - docker.errors.Co...
ERROR test_core.py::TestSpec::test_config_match_spec[inputs0] - docker.errors...
ERROR test_core.py::TestSpec::test_match_expected[inputs0] - docker.errors.Co...
ERROR test_core.py::TestSpec::test_docker_env[inputs0] - docker.errors.Contai...
ERROR test_core.py::TestSpec::test_oneof_usage[inputs0] - docker.errors.Conta...
ERROR test_core.py::TestSpec::test_defined_refs_exist_in_json_spec_file[inputs0]
ERROR test_core.py::TestSpec::test_oauth_flow_parameters[inputs0] - docker.er...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs2]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs2]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs1-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs1-not]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs2-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs2-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs2]
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contain...
ERROR test_core.py::TestBasicRead::test_read[inputs1] - docker.errors.Contain...
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
=================== 7 failed, 4 passed, 26 errors in 17.13s ====================

marcosmarxm · 2022-05-17T13:42:56Z

@RobertoBonnet could add the link to Zendesk documentation about the timeout parameter? Did you have a connection with more than 100k records did you try it with dev image?

RobertoBonnet · 2022-05-17T13:56:06Z

@marcosmarxm I didn't find anything in the documentation regarding the timeout, but in my tests I saw that it was happening. Apparently there is no timeout on either the server or the client side.

Here's the print of what I processed with this image

RobertoBonnet · 2022-05-18T20:01:23Z

On a fresh install I am getting pendulum error. I decided to update to "pendulum >= 2,<3" and it worked. Should I do this in this PR?

RobertoBonnet · 2022-05-18T20:04:12Z

Look at theses logs. I tried with a simple table to make it easy to debug

original-requirements-logs-402.txt
upgraded-pendulum-requirements-logs-402.txt

harshithmullapudi · 2022-05-20T08:00:10Z

/test connector=connectors/source-zendesk-chat

🕑 connectors/source-zendesk-chat https://github.com/airbytehq/airbyte/actions/runs/2356949185
❌ connectors/source-zendesk-chat https://github.com/airbytehq/airbyte/actions/runs/2356949185
🐛 https://gradle.com/s/o3zun5nlpdmeu
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_oneof_usage[inputs0] - AssertionError: Th...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
=================== 1 failed, 38 passed, 1 skipped in 45.11s ===================

marcosmarxm · 2022-05-20T13:32:58Z

@RobertoBonnet looks the error is related to spec.json file do you mind correcting it?

RobertoBonnet · 2022-05-20T14:38:40Z

@marcosmarxm Is there any way to run similar test locally? I'll check it

marcosmarxm · 2022-05-20T14:56:16Z

you can run ./gradlew airbyte-integrations:connectors:source-zendesk-chat:integrationTest

harshithmullapudi · 2022-05-23T06:51:15Z

@RobertoBonnet any update here?

jablonskijakub · 2022-05-23T11:33:53Z

@RobertoBonnet have you tested it for ticket_metrics in Zendesk Support as well?

RobertoBonnet · 2022-05-23T13:39:59Z

@harshithmullapudi I was working with zendesk support. I'll check chat now.

@jablonskijakub Yes, but the problem with support is other. I have commented here #12591

RobertoBonnet · 2022-05-23T21:12:44Z

@marcosmarxm @harshithmullapudi Can you run the test again?

marcosmarxm · 2022-05-23T21:14:02Z

/test connector=connectors/source-zendesk-chat

🕑 connectors/source-zendesk-chat https://github.com/airbytehq/airbyte/actions/runs/2373978675
✅ connectors/source-zendesk-chat https://github.com/airbytehq/airbyte/actions/runs/2373978675
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  896    259    71%

harshithmullapudi · 2022-05-25T09:45:38Z

/publish connector=connectors/source-zendesk-chat auto-bump-version=false

🕑 connectors/source-zendesk-chat https://github.com/airbytehq/airbyte/actions/runs/2383574279
🚀 Successfully published connectors/source-zendesk-chat
✅ connectors/source-zendesk-chat https://github.com/airbytehq/airbyte/actions/runs/2383574279

…nnection (airbytehq#12883) * setting a timeout in client side * upgraded version * upgraded pendulum lib * fix spec like docs * chore: bump version in source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com>

setting a timeout in client side

a1b60e1

github-actions bot added the area/connectors Connector related issues label May 16, 2022

octavia-squidington-iii added the community label May 16, 2022

harshithmullapudi self-requested a review May 17, 2022 07:07

harshithmullapudi self-assigned this May 17, 2022

upgraded version

57eb7e5

marcosmarxm changed the title ~~Passing timeout in request to prevent a stuck connection~~ Source Zendesk Chat: passing timeout in request to prevent a stuck connection May 17, 2022

upgraded pendulum lib

98a8e40

fix spec like docs

9147d4a

chore: bump version in source definitions

43aaa02

github-actions bot added the area/documentation Improvements or additions to documentation label May 25, 2022

chore: update seed file

491ee71

harshithmullapudi merged commit 15950df into airbytehq:master May 25, 2022

octavia-squidington-iii mentioned this pull request May 25, 2022

Bump Airbyte version from 0.39.1-alpha to 0.39.2-alpha #13188

Merged

RobertoBonnet deleted the zendesk-chat-prevent-a-stuck-connection branch May 27, 2022 15:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Zendesk Chat: passing timeout in request to prevent a stuck connection #12883

Source Zendesk Chat: passing timeout in request to prevent a stuck connection #12883

RobertoBonnet commented May 16, 2022 •

edited

Loading

CLAassistant commented May 16, 2022 •

edited

Loading

harshithmullapudi commented May 17, 2022

RobertoBonnet commented May 17, 2022

RobertoBonnet commented May 17, 2022

marcosmarxm commented May 17, 2022

marcosmarxm commented May 17, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 17, 2022

RobertoBonnet commented May 17, 2022

RobertoBonnet commented May 18, 2022

RobertoBonnet commented May 18, 2022 •

edited

Loading

harshithmullapudi commented May 20, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 20, 2022

RobertoBonnet commented May 20, 2022

marcosmarxm commented May 20, 2022

harshithmullapudi commented May 23, 2022

jablonskijakub commented May 23, 2022

RobertoBonnet commented May 23, 2022

RobertoBonnet commented May 23, 2022

marcosmarxm commented May 23, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented May 25, 2022 •

edited by github-actions bot

Loading

Source Zendesk Chat: passing timeout in request to prevent a stuck connection #12883

Source Zendesk Chat: passing timeout in request to prevent a stuck connection #12883

Conversation

RobertoBonnet commented May 16, 2022 • edited Loading

What

How

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

CLAassistant commented May 16, 2022 • edited Loading

harshithmullapudi commented May 17, 2022

RobertoBonnet commented May 17, 2022

RobertoBonnet commented May 17, 2022

marcosmarxm commented May 17, 2022

marcosmarxm commented May 17, 2022 • edited by github-actions bot Loading

marcosmarxm commented May 17, 2022

RobertoBonnet commented May 17, 2022

RobertoBonnet commented May 18, 2022

RobertoBonnet commented May 18, 2022 • edited Loading

harshithmullapudi commented May 20, 2022 • edited by github-actions bot Loading

marcosmarxm commented May 20, 2022

RobertoBonnet commented May 20, 2022

marcosmarxm commented May 20, 2022

harshithmullapudi commented May 23, 2022

jablonskijakub commented May 23, 2022

RobertoBonnet commented May 23, 2022

RobertoBonnet commented May 23, 2022

marcosmarxm commented May 23, 2022 • edited by github-actions bot Loading

harshithmullapudi commented May 25, 2022 • edited by github-actions bot Loading

RobertoBonnet commented May 16, 2022 •

edited

Loading

CLAassistant commented May 16, 2022 •

edited

Loading

marcosmarxm commented May 17, 2022 •

edited by github-actions bot

Loading

RobertoBonnet commented May 18, 2022 •

edited

Loading

harshithmullapudi commented May 20, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 23, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented May 25, 2022 •

edited by github-actions bot

Loading