IncompleteRead errors for shopify sources

[jrhizor]

See https://airbytehq.slack.com/archives/C01AHCDHCKE/p1621177893413200?thread_ts=1620852215.386600&cid=C01AHCDHCKE

2021-05-16 12:59:13 INFO (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):110 - METRIC: {"type": "timer", "metric": "http_request_duration", "value": 99.18193340301514, "tags": {"endpoint": "metafield_products", "status": "failed"}}
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 - CRITICAL IncompleteRead(1788731 bytes read)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 - Traceback (most recent call last):

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/http/client.py", line 578, in _readall_chunked

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     value.append(self._safe_read(chunk_left))

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/http/client.py", line 626, in _safe_read

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     raise IncompleteRead(b''.join(s), amt)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 - http.client.IncompleteRead: IncompleteRead(16378 bytes read, 16382 more expected)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 - 

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 - During handling of the above exception, another exception occurred:

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 - 

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 - Traceback (most recent call last):

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/bin/tap-shopify", line 8, in <module>

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     sys.exit(main())

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/singer/utils.py", line 229, in wrapped

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     return fnc(*args, **kwargs)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/tap_shopify/__init__.py", line 181, in main

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     sync()

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/tap_shopify/__init__.py", line 143, in sync

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     for rec in stream.sync():

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/tap_shopify/streams/metafields.py", line 58, in sync

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     for metafield in self.get_objects():

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/tap_shopify/streams/metafields.py", line 40, in get_objects

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     for parent_object in selected_parent.get_objects():

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/tap_shopify/streams/base.py", line 172, in get_objects

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     objects = self.call_api(query_params)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     ret = target(*args, **kwargs)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     ret = target(*args, **kwargs)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     ret = target(*args, **kwargs)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/tap_shopify/streams/base.py", line 82, in wrapper

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     return fnc(*args, **kwargs)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/tap_shopify/streams/base.py", line 136, in call_api

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     return self.replication_object.find(**query_params)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/shopify/base.py", line 211, in find

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     collection = super(ShopifyResource, cls).find(id_=id_, from_=from_, **kwargs)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/activeresource.py", line 386, in find

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     return cls._find_every(from_=from_, **kwargs)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/activeresource.py", line 525, in _find_every

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     response = cls.connection.get(path, cls.headers)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/connection.py", line 329, in get

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     return self._open('GET', path, headers=headers)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/shopify/base.py", line 27, in _open

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     self.response = super(ShopifyConnection, self)._open(*args, **kwargs)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/connection.py", line 291, in _open

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     response = Response.from_httpresponse(http_response)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/connection.py", line 184, in from_httpresponse

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     return cls(response.code, response.read(),

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/http/client.py", line 468, in read

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     return self._readall_chunked()

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -   File "/usr/local/lib/python3.7/http/client.py", line 582, in _readall_chunked

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 -     raise IncompleteRead(b''.join(value))

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) DefaultAirbyteStreamFactory(internalLog):108 - http.client.IncompleteRead: IncompleteRead(1788731 bytes read)

2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 - Traceback (most recent call last):
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -   File "/usr/local/bin/base-python", line 8, in <module>
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -     sys.exit(main())
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -   File "/usr/local/lib/python3.7/site-packages/base_python/entrypoint.py", line 135, in main
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -     launch(source, sys.argv[1:])
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -   File "/usr/local/lib/python3.7/site-packages/base_python/entrypoint.py", line 120, in launch
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -     AirbyteEntrypoint(source).start(args)
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -   File "/usr/local/lib/python3.7/site-packages/base_python/entrypoint.py", line 111, in start
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -     for message in generator:
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -   File "/usr/local/lib/python3.7/site-packages/base_singer/singer_helpers.py", line 167, in read
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -     for line, text_wrapper in SingerHelper._read_lines(p):
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -   File "/usr/local/lib/python3.7/site-packages/base_singer/singer_helpers.py", line 201, in _read_lines
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 -     raise Exception(f"Underlying command {process.args} failed with exit code {process.returncode}")
2021-05-16 12:59:13 ERROR (/tmp/workspace/13/0) LineGobbler(voidCall):69 - Exception: Underlying command tap-shopify --config /tmp/tmpc_w88rpz/config.json --catalog singer_rendered_catalog.json failed with exit code 1
2021-05-16 12:59:14 INFO (/tmp/workspace/13/0) LineGobbler(voidCall):69 - 2021-05-16 12:59:14 �[32mINFO�[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):77 - {} - hasFailed: false.

I remember reading somewhere that the shopify api would sometimes run into these errors for large shops but I can't find the page.

We may just need to retry these errors like we did in airbytehq/tap-shopify#3

┆Issue is synchronized with this Asana task by Unito

[vitaliizazmic]

It is common bug. This error is caused by bug in Shopify API Python Library. There is open issue
Shopify/shopify_python_api#320
Issue with the same error is opened in Shopify Tap
singer-io/tap-shopify#36
And Shopify API Python Library fails because of error in Shopify pyactiveresource
Probably it's required to re-implement the Shopify connector to not use singer and instead use the CDK