Feature/api refactor

[Magnus167]

No description provided.

[lsimonsen]

Suggestion for refactoring of overwall structure`

• download
  • dq_auth.py
  • dq_api.py
  • jpmaqs.py

and include a stub dataquery folder, where we can redirect from with a deprecation warning.

The main interface for users should be the callse download.JPMaQSDownload which has a download attribute. This way, if we need to change/add another data distribution source, we can easily add it as a key-word argument in the jpmaqs.py file.

[lsimonsen]

Why are we trying to request multiple times?

[lsimonsen]

I wonder if we are better off using off the shelves libraries for the OAuth handling:

from oauthlib.oauth2 import BackendApplicationClient
from requests_oauthlib import OAuth2Session

OAUTH_BASE_URL: str = "https://api-developer.jpmorgan.com/research/dataquery-authe/api/v2/"
OAUTH_TOKEN_URL: str = "https://authe.jpmchase.com/as/token.oauth2"
OAUTH_DQ_RESOURCE_ID: str = "JPMC:URI:RS-06785-DataQueryExternalApi-PROD"

class DQOAuth(object):
    def __init__(self, client_id: str, client_secret: str):
         self.client_id: str = client_id
         self.client_secret = client_secret

         self.client = BackendApplicationClient(client_id=client_id, aud=OAUTH_DQ_RESOURCE_ID)
         self.session = OAuth2Session(client=client)

    def _renew_token(self):
         self.session.fetch_token(
             token_url=OAUTH_TOKEN_URL,
             client_id="<client_id>",
            client_secret="<client_secret>"
          )

   def authorized(self):
        return self.session.authorized

    def request(url: str, method: = "GET"):
        if not self.authorized():
             self._renew_token()


        with self.session.request(url=url, method=method) as r:
             assert r.ok
             js = r.json()
        return js

[Magnus167]

The logic behind the multiple requests was to retry getting the token 3 times if and when it a token renew request fails. I've reverted to the original method (currently in use).

[Magnus167]

I'll look into using the library methods directly. Seems cleaner to use as well.

[lsimonsen]

No need for the pass here.

[lsimonsen]

We could pass parameters to the logger:

logger.error(
    "Invalid credentials. Request failed for URL: %s with message: %s and response: %s",
    last_url, msg, js
)

[lsimonsen]

logger.error(
    "Invalid credentials. Request failed for URL: %s"
    " with message: %s and response: %s",
    last_url, msg, js
)

[lsimonsen]

Wouldn't we arbitrarily break after 5 retries now as default? Even if the next link has more left?

[Magnus167]

Yes; that's true.
I think setting server_count=10 and testing (counter+invalid_responses)<=server_count before breaking might be a better idea.
PS: the variables after refactor are
server_count→max_retries
counter→conxn_errors

[lsimonsen]

Shouldn't this be further up? in first error_tickers?

[lsimonsen]

I think this will fail if track_id is None:

msg = "Hello, World" + None

Throws a TypeError

[lsimonsen]

Ah got it we convert track id to "" if None.