HTTP Request made importing Shopify library

[danpalmer]

Issue summary

When running import shopify, a blocking web request is made to Shopify's API.

This means that...

• App startup is dependent on Shopify's API being online.
• Services running in production can fail to start if Shopify's API is offline (as it was briefly today).
• Startup time is slower because of the API request.
• Offline development and testing is not possible.

Expected behavior

We would expected that the cost of importing a library is roughly the cost of loading that library's files into memory.

We would also expect that libraries only make network requests when explicitly asked to, so that developers are able to handle errors gracefully, monitor network performance, mock out in tests, and generally understand the implications of the code they write.

Actual behavior

This stacktrace shows the Shopify library making an API request at import time:

File "styleme/shopify/utils.py", line 8, in <module>
    import shopify
  File "__init__.py", line 2, in <module>
    from shopify.session import Session, ValidationException
  File "/opt/venv/lib/python3.5/site-packages/shopify/session.py", line 12, in <module>
    from shopify.api_version import ApiVersion, Release, Unstable
  File "/opt/venv/lib/python3.5/site-packages/shopify/api_version.py", line 86, in <module>
    ApiVersion.define_known_versions()
  File "/opt/venv/lib/python3.5/site-packages/shopify/api_version.py", line 28, in define_known_versions
    req = request.urlopen("https://app.shopify.com/services/apis.json")
  File "urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "urllib/request.py", line 472, in open
    response = meth(req, response)
  File "urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

We found this because our webserver crashed on startup with an HTTP 520 error from Shopify's API.

Also note that the issue traces to the Session import, which means it's not possible to circumvent this with careful importing.

Steps to reproduce the problem

1. import shopify

Specifications

• shopify_python_api version: 8.2.0
• pyactiveresource version: 2.2.2
• Shopify API version used (e.g. '2020-07'): N/A

[mllemango]

Hi @danpalmer, thanks for bringing this to our attention. We are currently looking into the best way to resolve this with your points in mind!

[jcalazan]

Just wanted to +1 this.

I think this is a pretty serious issue as it now requires that https://app.shopify.com/services/apis.json to always be up for Shopify apps using this client version (v8.2.0) to properly start.

We've gotten multiple HTTP 5xx errors today due to this as the https://app.shopify.com/services/apis.json site was unavailable (specifically 502 (bad gateway) and 524 errors (origin time-out)).

The last version we were using was v8.0.1 and didn't have this dependency.

[jcalazan]

It looks like this was only recently added on 8.2.0 (pull request #441).

My guess is the reason for the 5xx errors is due to that https://app.shopify.com/services/apis.json endpoint getting overwhelmed with requests at certain periods. I'm expecting the issue will get worse as more developers upgrade to the latest version.

It's not unusual for a single app to have potentially hundreds of background workers doing concurrent processing on the backend where each one could be making a request to this endpoint simply by importing the shopify module so I really think this could be a big issue.

[danpalmer]

Agreed. I'm keen to understand more about why #441 was made, it doesn't feel like a change that needs making from my current understanding.

At the moment I think the current issue is more serious for the ecosystem and production use of this library, so I'd propose reverting the change until another way of doing it can be found.

---

To elaborate on my feelings about #441, I think it might be solving the problem in the wrong way. My understanding is that Shopify wants to release new versions of the API and push those out to devs. By hard-coding supported versions, a new version of this library had to be released and devs had to update, which was understandably a maintenance burden.

Rather than solving this, I think #441 actually makes the problem worse - both making it unsafe to run in production because of the API request and making it unsafe to run long-term due to the way API removal will be surfaced.

By "automatically" getting the list of available API versions, when old versions are removed there is no indication in the versioning of the library. As devs specify the API version they wish to use, this would mean that their codebases do still need to be updated. This creates somewhat of a mismatch, making it harder to understand the versioning of the system as a whole, which necessarily includes the versions of services built on the Shopify API.

When an API version is removed from support, applications running in production will start seeing VersionNotFoundError exceptions raised in production.

I think a better approach could be to encode supported versions in the Python package (as it was before) and version the Python package according to the API versions. A Shopify API bump forces a Python library version bump, than then forces a developer's application version bump – there's an understandable link between each of these.

If getting developers to upgrade their packages is a problem, I think there are better solutions than crashing their applications. Perhaps a drip-feed email campaign to notify them of upcoming removals, perhaps even using API "brownouts" for very old APIs, if these aren't already being used (I'm new to the Shopify ecosystem, perhaps these are already done?).

GitHub has written about a similar process here, Facebook has a well defined API lifecycle and doesn't prevent old API libraries working outside of this, and Stripe just never seems to remove support (we were on their 2012 API for 6 years).

I may very well be missing a ton of context here as to why these decisions were made, and I may have misunderstood the failure modes. If so please do correct me!

[unfed]

Big +1. Great library in most respects, but this problem is even worse in some circumstances. Under a vanilla install of Python 3.8.5 on Ubuntu I get an

SSL: CERTIFICATE_VERIFY_FAILED

error just by importing the the shopify library. I suspect this has to do with the way Python is looking for root certs. This is a horrible workaround, but it works for me. Adding...

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

import shopify

bypasses the error, but at the expense of not verifying the certificate. I'm fine with that for now, but if anyone has a better solution please let me know.

[mllemango]

Hey everyone, thanks for all the comments. Dynamic versioning has been reverted and a new version 8.4.0 has been released.