Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connector: Hubspot tickets stream produces an error that URI is too big #5835

Closed
vladimir-remar opened this issue Sep 3, 2021 · 8 comments · Fixed by #6954
Closed

Connector: Hubspot tickets stream produces an error that URI is too big #5835

vladimir-remar opened this issue Sep 3, 2021 · 8 comments · Fixed by #6954
Assignees
Labels

Comments

@vladimir-remar
Copy link
Contributor

Enviroment

  • Airbyte version: 0.29.13-alpha
  • OS Version / Instance: Ubuntu 20.04
  • Source Connector and version: Hubspot 0.1.11
  • Step where error happened: Sync job

Current Behavior

When I try to pass the tests by running python main_dev.py read --config secrets / config.json --catalog sample_files / configured_catalog.json , when reaching the stream "tickets" an error occurs, apparently the URI is too large to be sent in the request. I attach the error.

Expected Behavior

Finish all the sync jobs.

Logs

error.log

Steps to Reproduce

  1. Create a config.json file inside secrets directory with a valid credentials and start_date with a recent date like 1 one week o days ago, I put "start_date": "2021-08-26T10:00:00Z"
  2. Create a virtualenv and install connector dependencies
  3. run python main_dev.py read --config secrets/config.json --catalog sample_files/configured_catalog.json
@vladimir-remar vladimir-remar added the type/bug Something isn't working label Sep 3, 2021
@vladimir-remar vladimir-remar changed the title Connector: Hubspot stream tickets produce an error. Connector: Hubspot stream tickets produces an error. Sep 3, 2021
@rbryant1925
Copy link

Hey Airbyte Team, I am trying to use Airbyte to bring in several HubSpot tables but no matter how I configure the connector I get errors — the connector has tried to run three different times (3 attempts each, so 9 times total) and not one has succeeded. Here is the most recent failure logs. Am I missing something? Is there anyone who can give some guidance on this? Thank you in advance for your help!
P.S. I tried originally to normalize the data by using that option in the configuration and got normalization errors so I removed it. But theoretically, if normalization is possible that is what I would prefer.
(commented from Slack) https://airbytehq.slack.com/files/U02CUQW0GDD/F02D73VHFBQ/logs-347-2.txt

@sherifnada sherifnada added the area/connectors Connector related issues label Sep 10, 2021
@sherifnada sherifnada changed the title Connector: Hubspot stream tickets produces an error. Connector: Hubspot tickets stream produces an error that URI is too big Sep 10, 2021
@sherifnada
Copy link
Contributor

This one is a bit open ended - will assign this to the team today. For the next person to pick this up: what are our options to move forward on this? Some questions which may help solve:

  • Is there something the user can configure to fix this problem?
  • is there something we can do to detect when we are in this situation? If so, what recourse do we have? Can we batch/partition requests? (preferred)

We may need to get creative here...

@sherifnada
Copy link
Contributor

@vladimir-remar can you try the new version of the connector and let us know if it solves the issue?

@vladimir-remar
Copy link
Contributor Author

@sherifnada Sure! I will do it ASAP, thanks for the message.

@vladimir-remar
Copy link
Contributor Author

@sherifnada I modifed the configured_catalog.json just to sync the "tickets" stream
configured_catalog.json

{
  "streams": [{
      "stream": {
        "name": "tickets",
        "json_schema": {},
        "supported_sync_modes": ["full_refresh"]
      },
      "sync_mode": "full_refresh",
      "destination_sync_mode": "overwrite"
    }]
}

I ran this command
python main_dev.py read --config secrets/config.json --catalog sample_files/configured_catalog.json

LOG tickets.log
I did the pull from master and recreate the virtual enviroment.
If you want me to try something new let me a message here or one in slack.

@sherifnada sherifnada reopened this Oct 26, 2021
@yevhenii-ldv
Copy link
Contributor

Hi, @vladimir-remar
It looks like you still have the previous version of the connector, since the length of the URL line in your request becomes more than 29k characters, but at the moment this has been changed and the maximum length of the URL should be 16k characters.
Also, I just pushed in master branch a little hotfix, therefore, could you pull the changes from the master branch again and try to read the data again (please, preliminary create a new one or update the old virtual environment).

@vladimir-remar
Copy link
Contributor Author

@yevhenii-ldv now it works!!! Thanks.

@sherifnada
Copy link
Contributor

Nicely done @yevhenii-ldv !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants