🐛 Source Shopify: Failed to pull data #14065

marcosmarxm · 2022-06-23T12:29:27Z

This Github issue is synchronized with Zendesk:

Ticket ID: #1301
Priority: normal
Group: User Success Engineer
Assignee: Marcos Marx

Original ticket description:

Airbyte version: 0.39.23-alpha
OS Version / Instance: GCP Ubuntu 22.04 LTS
Deployment: Docker
Source Connector and version: Shopify 0.1.37
Destination Connector and version: Mysql 0.5.13

I am getting error on time sync. On case raw data and normalization too…

Here is normalization error:

2022-06-16 07:18:36 normalization > 07:18:31.133803 [error] [MainThread]:   ) as user_id,
2022-06-16 07:18:36 normalization > 07:18:31.134448 [error] [MainThread]:       cast(curr...' at line 84
2022-06-16 07:18:36 normalization > 07:18:31.135066 [error] [MainThread]:   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/airbyte_shopify/transactions.sql
2022-06-16 07:18:36 normalization > 07:18:31.135707 [info ] [MainThread]: 
2022-06-16 07:18:36 normalization > 07:18:31.136343 [error] [MainThread]: Database Error in model order_refunds_refund_line_items (models/generated/airbyte_incremental/airbyte_shopify/order_refunds_refund_line_items.sql)
2022-06-16 07:18:36 normalization > 07:18:31.137057 [error] [MainThread]:   1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'json) as line_item,
2022-06-16 07:18:36 normalization > 07:18:31.137647 [error] [MainThread]:       cast(total_tax as 
2022-06-16 07:18:36 normalization > 07:18:31.138289 [error] [MainThread]:       float
2022-06-16 07:18:36 normalization > 07:18:31.138908 [error] [MainThread]:   ) as total_tax,
2022-06-16 07:18:36 normalization > 07:18:31.139495 [error] [MainThread]:       cast...' at line 111
2022-06-16 07:18:36 normalization > 07:18:31.140133 [error] [MainThread]:   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/airbyte_shopify/order_refunds_refund_line_items.sql
2022-06-16 07:18:36 normalization > 07:18:31.140773 [info ] [MainThread]: 
2022-06-16 07:18:36 normalization > 07:18:31.141397 [error] [MainThread]: Database Error in model order_refunds_transactions (models/generated/airbyte_incremental/airbyte_shopify/order_refunds_transactions.sql)
2022-06-16 07:18:36 normalization > 07:18:31.142119 [error] [MainThread]:   1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'json) as receipt,
2022-06-16 07:18:36 normalization > 07:18:31.142736 [error] [MainThread]:       cast(user_id as 
2022-06-16 07:18:36 normalization > 07:18:31.143399 [error] [MainThread]:       signed
2022-06-16 07:18:36 normalization > 07:18:31.144009 [error] [MainThread]:   ) as user_id,
2022-06-16 07:18:36 normalization > 07:18:31.144604 [error] [MainThread]:       cast(curr...' at line 127
2022-06-16 07:18:36 normalization > 07:18:31.145228 [error] [MainThread]:   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/airbyte_shopify/order_refunds_transactions.sql

Raw data full log:
logs-74.txt (423.7 KB)

[Discourse post]

The text was updated successfully, but these errors were encountered:

marcosmarxm · 2022-06-23T12:30:59Z

Main error:

The error was:
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 127, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 118, in run
    for message in generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 123, in read
    raise e
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 114, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 159, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 248, in _read_full_refresh
    for record in records:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 373, in read_records
    response = self._send_request(request, request_kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 337, in _send_request
    return backoff_handler(user_backoff_handler)(request, request_kwargs)
  File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 110, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 110, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 299, in _send
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://<yourshopify>.myshopify.com/admin/api/2021-07/shopify_payments/balance/transactions.json?limit=250&order=id+asc&since_id=0

Solution here: https://community.shopify.com/c/shopify-apis-and-sdks/api-call-getting-404-error-back/td-p/1382293#:~:text=Generally%20a%20404%20error%20presents,outlined%20in%20our%20docs%20here.

marcosmarxm · 2022-06-23T12:31:20Z

Maybe bump Shopify connector to use latest version of error logging in CDK.

marcosmarxm · 2022-06-23T12:32:22Z

Comment made from Zendesk by Marcos Marx on 2022-06-23 at 12:32:

The error was:

Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 127, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 118, in run
    for message in generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 123, in read
    raise e
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 114, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 159, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 248, in _read_full_refresh
    for record in records:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 373, in read_records
    response = self._send_request(request, request_kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 337, in _send_request
    return backoff_handler(user_backoff_handler)(request, request_kwargs)
  File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 110, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 110, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 299, in _send
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://<yourshopify>.myshopify.com/admin/api/2021-07/shopify_payments/balance/transactions.json?limit=250&order=id+asc&since_id=0

I created issue #14065 to investigate and solve the problem.
Are you able to manual try to hit this endpoint and see the headers/response from Shopify? This will give us some clues about how to solve it.

marcosmarxm · 2022-06-23T18:21:55Z

Comment made from Zendesk by Marcos Marx on 2022-06-23 at 18:21:

You can see that here is missing meaning.

https://.myshopify.com/admin/api/2021-07/shopify_payments/balance/transactions.json?limit=250&order=id+asc&since_id=0

When I am trying reach manually, using my Shopify name, it is asking for user and pass.
I am not sure witch user/pass I need to use. As my regular Shopify one doesn’t work…

possible reason is log URL as is not replaced with meaning…

[Discourse post]

marcosmarxm · 2022-06-24T16:28:18Z

Comment made from Zendesk by Marcos Marx on 2022-06-24 at 16:28:

You must build the cURL call correctly with your api token, only requesting the endpoint will fail.

marcosmarxm added the zendesk label Jun 23, 2022

octavia-squidington-iii added the team/triage label Jun 23, 2022

marcosmarxm added area/connectors Connector related issues connectors/source/shopify team/extensibility community and removed team/triage labels Jun 23, 2022

marcosmarxm added team/connectors-python and removed team/extensibility labels Jul 5, 2022

natalyjazzviolin changed the title ~~Failed to pull data from Shopify~~ Source Shopify: Failed to pull data Aug 2, 2022

marcosmarxm mentioned this issue Aug 3, 2022

🐛 Source Shopify: unavailable balance transactions causes 404 error #15242

Closed

arsenlosenko self-assigned this Oct 6, 2022

bazarnov mentioned this issue Oct 10, 2022

🐛 Source Shopify: fix 404 for configured streams, fix missing cursor error for old records #17777

Merged

10 tasks

bazarnov linked a pull request Oct 10, 2022 that will close this issue

🐛 Source Shopify: fix 404 for configured streams, fix missing cursor error for old records #17777

Merged

10 tasks

bazarnov assigned bazarnov and unassigned arsenlosenko Oct 10, 2022

bazarnov changed the title ~~Source Shopify: Failed to pull data~~ 🐛 Source Shopify: Failed to pull data Oct 10, 2022

bazarnov closed this as completed in #17777 Oct 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Shopify: Failed to pull data #14065

🐛 Source Shopify: Failed to pull data #14065

marcosmarxm commented Jun 23, 2022 •

edited

marcosmarxm commented Jun 23, 2022

marcosmarxm commented Jun 23, 2022

marcosmarxm commented Jun 23, 2022

marcosmarxm commented Jun 23, 2022

marcosmarxm commented Jun 24, 2022

🐛 Source Shopify: Failed to pull data #14065

🐛 Source Shopify: Failed to pull data #14065

Comments

marcosmarxm commented Jun 23, 2022 • edited

marcosmarxm commented Jun 23, 2022

marcosmarxm commented Jun 23, 2022

marcosmarxm commented Jun 23, 2022

marcosmarxm commented Jun 23, 2022

marcosmarxm commented Jun 24, 2022

marcosmarxm commented Jun 23, 2022 •

edited