Shopify flaky CI

[sherifnada]

https://status-api.airbyte.io/tests/summary/source-shopify-singer/

2021-02-17 12:36:30 DEBUG i.a.w.p.DockerProcessBuilderFactory(create):104 - *** - Preparing command: docker run --rm -i -v /tmp/airbyte_tests/test8767356921545817202:/data -v /tmp/airbyte_tests/output14148456242023316215:/local -w /data/job --network host airbyte/source-shopify-singer:dev discover --config tap_config.json
2021-02-17 12:36:31 INFO i.a.i.s.s.PythonTestSource(runExecutableInternal):126 - *** - testRoot = /tmp/standard_test/pytest14589037234217250053
2021-02-17 12:36:31 INFO i.a.c.i.LineGobbler(voidCall):69 - *** - parsing
2021-02-17 12:36:31 INFO i.a.c.i.LineGobbler(voidCall):69 - *** - executing command TestAction.TEARDOWN
2021-02-17 12:36:31 INFO i.a.c.i.LineGobbler(voidCall):69 - *** - writing output
2021-02-17 12:36:31 INFO i.a.c.i.LineGobbler(voidCall):69 - *** - complete

Failures (3):
  JUnit Jupiter:PythonTestSource:testIdenticalFullRefreshes()
    MethodSource [className = 'io.airbyte.integrations.standardtest.source.PythonTestSource', methodName = 'testIdenticalFullRefreshes', methodParameterTypes = '']
    => io.airbyte.workers.WorkerException: Tap process wasn't successful
       io.airbyte.workers.protocols.airbyte.DefaultAirbyteSource.close(DefaultAirbyteSource.java:112)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.runRead(StandardSourceTest.java:437)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.runRead(StandardSourceTest.java:420)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.testIdenticalFullRefreshes(StandardSourceTest.java:279)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       java.base/java.lang.reflect.Method.invoke(Method.java:564)
       org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
       org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
       [...]
  JUnit Jupiter:PythonTestSource:testEmptyStateIncrementalIdenticalToFullRefresh()
    MethodSource [className = 'io.airbyte.integrations.standardtest.source.PythonTestSource', methodName = 'testEmptyStateIncrementalIdenticalToFullRefresh', methodParameterTypes = '']
    => io.airbyte.workers.WorkerException: Tap process wasn't successful
       io.airbyte.workers.protocols.airbyte.DefaultAirbyteSource.close(DefaultAirbyteSource.java:112)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.runRead(StandardSourceTest.java:437)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.runRead(StandardSourceTest.java:420)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.testEmptyStateIncrementalIdenticalToFullRefresh(StandardSourceTest.java:357)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       java.base/java.lang.reflect.Method.invoke(Method.java:564)
       org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
       org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
       [...]
  JUnit Jupiter:PythonTestSource:testFullRefreshRead()
    MethodSource [className = 'io.airbyte.integrations.standardtest.source.PythonTestSource', methodName = 'testFullRefreshRead', methodParameterTypes = '']
    => io.airbyte.workers.WorkerException: Tap process wasn't successful
       io.airbyte.workers.protocols.airbyte.DefaultAirbyteSource.close(DefaultAirbyteSource.java:112)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.runRead(StandardSourceTest.java:437)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.runRead(StandardSourceTest.java:420)
       io.airbyte.integrations.standardtest.source.StandardSourceTest.testFullRefreshRead(StandardSourceTest.java:255)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       java.base/java.lang.reflect.Method.invoke(Method.java:564)
       org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
       org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
       [...]

Test run finished after 484669 ms
[         2 containers found      ]
[         0 containers skipped    ]
[         2 containers started    ]
[         0 containers aborted    ]
[         2 containers successful ]
[         0 containers failed     ]
[         7 tests found           ]
[         0 tests skipped         ]
[         7 tests started         ]
[         0 tests aborted         ]
[         4 tests successful      ]
[         3 tests failed          ]

There are failing tests. See https://docs.airbyte.io/contributing-to-airbyte/building-new-connector/standard-source-tests for more information about the standard source test suite.


> Task :airbyte-integrations:connectors:source-shopify-singer:standardSourceTestPython FAILED
FAILURE: Build failed with an exception.


Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
* What went wrong:
Use '--warning-mode all' to show the individual deprecation warnings.
Execution failed for task ':airbyte-integrations:connectors:source-shopify-singer:standardSourceTestPython'.
> Process 'command 'docker'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

[yevhenii-ldv]

Looking at the traceback, we can conclude that the backoff periodically stops working. Since CI tests are passed with a certain frequency, I suspect that in cases where the CI tests fail, then several users use the API key at the same time, and therefore the Shopify API is in constant backoff.

It is necessary to study more closely how the backoff works inside Singer Tap and may have to fork and rewrite the backoff for Tap Shopify

2021-02-17 12:36:27 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):108 - *** -   File "/usr/local/lib/python3.7/site-packages/pyactiveresource/connection.py", line 427, in _handle_error

2021-02-17 12:36:27 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):108 - *** -     raise ClientError(err)

2021-02-17 12:36:27 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):108 - *** - pyactiveresource.connection.ClientError: Response(code=429, body="b'***"errors":"Exceeded 2 calls 
per second for api client. Reduce request rates to resume uninterrupted service."***'", headers=***'Date': 'Wed, 17 Feb 2021 12:36:27 GMT', 'Content-Type': 'application/json; 
charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'close', 'X-Sorting-Hat-PodId': '157', 'X-Sorting-Hat-ShopId': '24824905794', 'Referrer-Policy': 'origin-when-cross-origin', '
X-Frame-Options': 'DENY', 'X-ShopId': '24824905794', 'X-ShardId': '157', 'X-Stats-UserId': '', 'X-Stats-ApiClientId': '4550305', 'X-Stats-ApiPermissionId': '295615463582', 'X-Shopify-
API-Terms': 'By accessing or using the Shopify API you agree to the Shopify API License and Terms of Use at https://www.shopify.com/legal/api-terms', 'Retry-After': '2.0', 'Strict-
Transport-Security': 'max-age=7889238', 'X-Request-Id': '99c4c9fc-3218-4baa-8257-f55ee6172a2a', 'X-Shopify-Stage': 'production', 'Content-Security-Policy': "default-src 'self' data: 
blob: 'unsafe-inline' 'unsafe-eval' https://* shopify-pos://*; block-all-mixed-content; child-src 'self' https://* shopify-pos://*; connect-src 'self' wss://* https://*; frame-ancestors
'none'; img-src 'self' data: blob: https:; script-src https://cdn.shopify.com https://cdn.shopifycdn.net https://checkout.shopifycs.com https://js-agent.newrelic.com 
https://bam.nr-data.net https://api.stripe.com https://mpsnare.iesnare.com https://appcenter.intuit.com https://www.paypal.com https://js.braintreegateway.com https://c.paypal.com https://maps.googleapis.com https://www.google-analytics.com https://v.shopify.com 
https://widget.intercom.io https://js.intercomcdn.com 'self' 'unsafe-inline' 'unsafe-eval'; 
upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=99c4c9fc-3218-4baa-8257-f55ee6172a2a", 
'X-Content-Type-Options': 'nosniff', 'X-Download-Options': 'noopen', 'X-Permitted-Cross-Domain-Policies': 'none', 'X-XSS-Protection': '1; mode=block; report=/xss-report
source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=99c4c9fc-3218-4baa-8257-f55ee6172a2a', 'X-Dc': 
'gcp-us-east1,gcp-us-east1', 'Set-Cookie': '_y=bf0292af-8136-4b0d-bb0b-11142a158df6; Expires=Thu, 17-Feb-22 12:36:27 GMT; Domain=jolicookie.myshopify.com; Path=/; SameSite=Lax', 
'CF-Cache-Status': 'DYNAMIC', 'cf-request-id': '085196f92b00002a9f52921000000001', 'Expect-CT': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"', 
'Server': 'cloudflare', 'CF-RAY': '622f8dd51e622a9f-IAD', 'alt-svc': 'h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400'***, msg="Too Many Requests")

[sherifnada]

Shopify tap is already forked here

[sherifnada]

guessing will take 5h